Hand controller, system, and control method for a surgical robotic system

Abstract

Systems and methods using at least one hand controller for controlling a robotic assembly of a surgical robotic system are provided. The hand controller may be used at a surgeon's console of the surgical robotic system. The hand controller may include a contoured housing having a top surface, an inner side adjacent to the top surface, an outer side facing away from the first side, and a bottom surface facing away from the top surface. The hand controller may include a plurality of buttons including a first button positioned on the top surface and a second button positioned on one of the top surface, the inner side, or the outer side, a first touch input device positioned on the top surface, and a first paddle attached to either the inner side or the outer side.

Description

[Background technology]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of the filing dates under 35 U.S.C. §119(e) of U.S. Provisional Patent Application No. 63 / 412,372, filed September 30, 2022, and U.S. Provisional Patent Application No. 63 / 412,377, filed October 1, 2022, the entire contents of which are incorporated herein by reference.

[0002] A surgical robotic system allows a surgeon (also referred to herein as an operator or user) to perform actions using robotically controlled instruments to perform tasks and functions during a procedure. The surgeon may use a visualization system to view or observe actions, including viewing images from a camera that shows the patient and / or is attached to the robotically controlled instruments.

[0003] Controls are provided to enable a user to control features of the surgical robotic system, such as surgical tools, the arms of the surgical robotic system, and camera position, as well as navigate the features and settings available within the surgical robotic system. Controls used in surgical robotic systems may include one or more hand controllers operated by one or both hands of the surgeon.

[0004] Certain controls may suffer from several deficiencies. For example, certain control systems may not provide sufficient input to the surgical robotic system to control aspects, or may not provide user-friendly operation or a "natural" feel. These deficiencies may result in users having difficulty operating the surgical robotic system, reduced accuracy and ease of operation, or reduced user satisfaction. Summary of the Invention

[0005] The present disclosure is directed to one or more hand controllers for controlling a surgical robotic system including a robot assembly configured to be positioned in an internal body cavity of a subject. Each of the one or more hand controllers may include a contoured housing having a top surface, an inner side adjacent to the top surface, an outer side facing away from the inner side, and a bottom surface facing away from the top surface, a plurality of buttons including a first button positioned on the top surface and a second button positioned on one of the top surface, the inner side, or the outer side, a first touch input device positioned on the top surface, and a first paddle disposed on either the inner side or the outer side. The contoured housing is configured to be grasped on the top surface with a user's thumb on the inner side and at least a portion of a user's fingers, a user's palm, or both, and the first button is configured to be operated by a user's index finger.

[0006] In some embodiments, for each of the one or more hand controllers, the contact surface of the first paddle is disposed on an outer side of the housing. In some embodiments, each of the one or more hand controllers also includes a second paddle. The contact surface of the first paddle may be disposed on an outer side of the housing and the contact surface of the second paddle may be disposed on an inner side of the housing, or the contact surface of the first paddle may be disposed on an inner side of the housing and the contact surface of the second paddle may be disposed on an outer side of the housing. In some embodiments, the first paddle is engaged with the second paddle via one or more gears. Movement of the first paddle may cause reciprocating movement of the second paddle, and movement of the second paddle may cause reciprocating movement of the first paddle. In some embodiments, for each of the hand controllers, the first touch input device is a scroll wheel, a rocker button, a joystick, a pointing stick, a touchpad, or a trackball. In some embodiments, at least one of the one or more hand controllers also includes a second touch input device. In some embodiments, for each of the hand controllers, deflection of the first paddle is configured to generate a signal that the surgical robotic system uses as an input to control the robotic subassembly.

[0007] For some embodiments, for at least one of the hand controllers, at least one of the plurality of buttons is mapped to a function selected from a plurality of functions. The plurality of functions may include triggering activation of a clutch for the hand controller, where activation of a clutch associated with the hand controller allows or enables movement of the respective hand controller without causing movement of the robotic assembly. The plurality of functions may also include resetting a posture of the robotic arms and robotic assembly relative to the virtual chest to a default posture while maintaining the position and orientation of the instrument tip relative to each of the robotic arms. The plurality of functions may also include activating or exiting an instrument control mode of the surgical robotic system, where, when in instrument mode, movement of one of the hand controllers causes a corresponding movement in a corresponding robotic arm of the robotic assembly. The functions may also include activating or exiting a scan mode of the surgical robotic system, wherein when in the scan mode, movement of one of the hand controllers causes a corresponding change in the orientation of a camera assembly of the robotic assembly, while the robotic arm of the robotic assembly remains stationary if the robotic assembly includes one robotic arm, or all of the robotic arms of the robotic assembly remain stationary if the robotic assembly includes more than one robotic arm. The functions may also include activating or exiting a view mode of the surgical robotic system, wherein when in the view mode, movement of one of the hand controllers causes a corresponding change in the orientation and position of the camera assembly, while maintaining the position of the instrument tips of all of the robotic arms of the robotic assembly.The functions may also include activating or terminating a swivel mode of the surgical robotic system, where activating the swivel mode causes a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to center the camera assembly view at a location midway between the instrument tips of the robotic arms, which is the average instrument tip position, and where, when the swivel mode is activated, movement of one of the hand controllers causes a corresponding movement of the instrument tip of the corresponding robotic arm relative to the camera assembly view, causing a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to maintain the camera assembly view centered at the average instrument tip position. The functions may also include activating or terminating a menu feature of the surgical robotic system, where the menu feature causes a menu to be displayed in a graphical user interface of the surgical robotic system and input from a touch input device or button of one or more hand controllers used to select an item from the menu. The functions may also include navigating through and / or selecting options from menus displayed in the graphical user interface of the surgical robotic system. The functions may also include activating or deactivating an elbow biasing feature, which allows for adjustment of the magnitude and direction of the elevation of the elbow of a robotic arm while maintaining the position of the instrument tip of the robotic arm relative to each robotic arm of the robotic assembly. The functions may also include controlling the adjustment of the elevation of the elbow of a robotic arm of a robotic assembly for one robotic arm of the robotic assembly or for each robotic arm of the robotic assembly, and changing or selecting the zoom of the view from a camera assembly of the robotic assembly. In some embodiments, the functions may be grouped into one or more groups. In some embodiments, a surgical robotic system need not include all of the functions. For example, a swivel mode may be removed from the surgical robotic system, and / or the functionality of the swivel mode may be integrated into the camera mode.

[0008] In some embodiments, for at least one of the hand controllers, at least two of the plurality of buttons are mapped to functions selected from a plurality of functions. In some embodiments, for at least one of the hand controllers, at least three of the plurality of buttons are mapped to functions selected from a plurality of functions. In some embodiments, for at least one of the one or more hand controllers, a first touch input device has one or more functions selected from a plurality of functions. In some embodiments, for each of the one or more hand controllers, a first touch input device has one or more functions selected from a plurality of functions. In some embodiments, at least one of the one or more hand controllers also includes a second touch input device, and for at least one of the one or more hand controllers, the first touch input device or the second touch input device has one or more functions selected from a plurality of functions. In some embodiments, at least one of the one or more hand controllers also includes a second touch input device, and for at least one of the one or more hand controllers, the first touch input device and the second touch input device each have one or more functions selected from a plurality of functions. In some embodiments, activation of the first button for at least one of the one or more hand controllers generates a signal that triggers activation of a clutch feature of the surgical robotic system for the hand controller, where activation of the clutch feature allows or enables movement of the respective hand controller without causing movement of the robotic assembly. In some embodiments, for each hand controller, the outer side of the housing is substantially concave. In some embodiments, each of the one or more hand controllers also includes a thumb pad located on either the inner side or the outer side. In some embodiments, at least one of the one or more hand controllers includes a sensor configured to sense contact of an operator's hand with the hand controller or to sense proximity of an operator's hand to the hand controller.In some embodiments, input from the sensors is used to activate or execute an instrument control mode of the surgical robotic system, and when in instrument mode, movement of one of the hand controllers causes a corresponding movement in a corresponding robotic arm of the robotic assembly.

[0009] In some embodiments, the present disclosure is directed to one or more hand controllers for controlling a surgical robotic system including a robot assembly configured to be disposed in an internal body cavity of a subject. Each of the one or more hand controllers may include a contoured housing having an upper surface, an inner side adjacent to the upper surface, an outer side facing away from the inner side, and a lower surface facing away from the upper surface. Each of the one or more hand controllers may also include a plurality of buttons including a first button positioned on the upper surface and a second button positioned on one of the upper surface, the inner side, or the outer side. Each of the one or more hand controllers may also include a first touch input device positioned on the upper surface and a first paddle positioned on the outer side. Each of the one or more hand controllers may also include a second paddle positioned on the inner side. The contoured housing may be configured to be grasped on the upper surface with a user's thumb on the inner side and at least a portion of the user's fingers, the user's palm, or both, and the first button may be configured to be operated by the user's index finger.

[0010] In another aspect, the present disclosure is directed to a hand controller system for controlling a surgical robotic system including a robotic assembly configured to be disposed within an internal body cavity of a subject. The hand controller system may include a first hand controller configured to be removably coupled or connected to an operator console via a first hand controller support, as described above. The hand controller system may also include a second hand controller configured to be removably coupled or connected to an operator console via a second hand controller support, as described above. An inner surface of the first hand controller may face an inner surface of the second hand controller when the first and second hand controllers are in a neutral position during use.

[0011] In another aspect, the present disclosure is directed to a hand controller system for controlling a surgical robotic system including a robotic assembly configured to be disposed in an internal body cavity of a subject. The hand controller system may include a first hand controller configured to be removably coupled or connected to an operator console via a first hand controller support. The first hand controller may include (i) a contoured housing having a top surface, a first side surface adjacent to the top surface, a second side surface facing away from the first side, and a bottom surface facing away from the top surface; (ii) a first button and a first scroll wheel disposed on the top surface; (iii) a first paddle having a contact surface disposed on the first side or the second side; and (iv) a second button disposed on the first side or the second side. The hand controller system may also include a second hand controller configured to be removably coupled or connected to the operator console via a second hand controller support. The second hand controller may include (i) a contoured housing having a top surface, a first side surface adjacent to the top surface, a second side surface facing away from the first side, and a bottom surface facing away from the top surface, (ii) a first button and a first scroll wheel disposed on the top surface, (iii) a first paddle having a contact surface disposed on the first side or the second side, and (iv) a second button disposed on the first side or the second side. The inner surface of the first hand controller may face the inner surface of the second hand controller when the first and second hand controllers are in a neutral position during use.

[0012] In some embodiments, the present disclosure is directed to a surgeon's console including the hand controller system described above and a plurality of foot pedals. The plurality of foot pedals may include one or more of a view mode foot pedal, a pivot mode foot pedal, a drive mode foot pedal, and a translate mode foot pedal. The view mode foot pedal may be configured, when activated, to activate a view mode of the surgical robotic system, such that movement of one of the hand controllers when the view mode is activated causes a corresponding change in orientation and position of the camera assembly and a rotation of the virtual chest of the robotic assembly to maintain the positions and orientations of the instrument tips of all robotic arms of the robotic assembly while maintaining the angular deviation between a line from the center of the virtual chest to the position of the midpoint between the instrument tips of the robotic arms of the robotic assembly, which is the average instrument tip position, and a normal to the virtual chest within an allowable angular deviation range, while maintaining the positions and orientations of the instrument tips of all robotic arms of the robotic assembly. The swivel mode foot pedal may be configured to activate a swivel mode of the surgical robotic system when activated, wherein activating the swivel mode causes a change in orientation of the camera assembly, or a change in orientation of the camera assembly and a change in orientation of the virtual chest of the robot assembly, to center the camera assembly's view on a location of the midpoint between the instrument tips of the robot arms of the robot assembly, which is the average instrument tip position, and wherein movement of one of the hand controllers when the swivel mode is activated causes a corresponding movement of the instrument tip of the corresponding robot arm relative to the camera assembly's view, causing a change in orientation of the camera assembly, or a change in orientation of the camera assembly and a change in orientation of the virtual chest, to keep the camera assembly's view centered at the average instrument tip position and maintain an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.The drive mode foot pedal may be configured, when activated, to activate a drive mode of the surgical robotic system, and when the drive mode is activated, movement of one of the hand controllers may cause a corresponding movement of the instrument tip of the corresponding robotic arm by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the view of the camera assembly, rotating the camera assembly to center the view of the camera assembly on the average tip position, translating, rotating, or both, the virtual chest of the robot assembly to maintain a distance between the center of the virtual chest of the robot assembly and the average instrument tip position within an acceptable distance range, and maintaining an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range. The translation mode foot pedal may be configured, when activated, to activate a translation mode of the surgical robotic system, such that when the translation mode is activated, movement of one of the hand controllers causes a corresponding movement of the instrument tip of the corresponding robotic arm relative to the view of the camera assembly, causes a rotation of the camera assembly to center the view of the camera assembly on the average tip position, and causes a translation of the virtual chest to maintain the distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0013] In some embodiments, the present disclosure is directed to a method for controlling a robot assembly of a surgical robotic system including a visual display and right and left hand controllers configured to sense hand movements of an operator. The robotic assembly may include a camera assembly and a robotic arm assembly including a first robotic arm and a second robotic arm, each having an associated instrument tip, position, and orientation of the robotic arm and camera assembly that define a virtual chest of the robotic assembly. The method may include receiving a first control mode selection input while the robotic assembly is positioned within an internal cavity of a subject, and changing a current control mode of the surgical robotic system to the first control mode in response to the first control mode selection input, the surgical robotic system having a plurality of control modes including one or more of a scan mode, a view mode, a drive mode, a turn mode, and a translate mode, and the first control mode is the scan mode, the view mode, the drive mode, the turn mode, or the translate mode. The method may also include a specific action responsive to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller while the current control mode is a first control mode. If the first control mode is a scan mode, the method may also include holding the robotic arm assembly stationary while rotating the camera assembly a corresponding first rotation relative to a displayed camera view, a view of the camera assembly displayed on a video display of the surgical robotic system.If the first control mode is a travel mode, the method also includes moving a corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on a location of the midpoint between the instrument tips of the robot arm of the robot assembly, which is the average tip position, or while translating the robot assembly to translate a chest point of the virtual chest and while rotating the robot assembly to rotate the virtual chest, or both, to adjust the distance between the center of the virtual chest and the average tip position within an acceptable distance range. and maintaining an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range, and if the first control mode is a view mode, moving the camera assembly and rotating the virtual chest of the robot assembly by a corresponding first movement including a corresponding first translation of the virtual chest and a corresponding first rotation of the camera assembly relative to the displayed camera view, while holding the position and orientation of each instrument to maintain an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range. If the first control mode is a swivel mode, the method may also include rotating the camera assembly while moving the corresponding instrument tip by a corresponding first movement, including a corresponding first rotation, relative to the displayed camera view to center the camera assembly view on the average instrument tip position, causing a change in orientation of the camera assembly, or a change in orientation of the camera assembly and a change in orientation of the virtual chest, to maintain the camera assembly view centered on the average instrument tip position and maintaining an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.If the first control mode is a translation mode, the method may also include rotating the camera assembly, or rotating the camera assembly and translating the virtual chest while moving the corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view to center the camera assembly view on the average tip position and maintain a distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0014] In some embodiments, the current control mode may be an instrument control mode in which movement of the right hand controller or the left hand controller causes corresponding movement of an instrument tip at the distal end of the corresponding robotic arm relative to the displayed camera view without moving or changing the orientation of the virtual chest of the robotic assembly. In some embodiments, the first control mode selection input may be received via a foot pedal of the surgical robotic system. In some embodiments, the first control mode selection input may be received via a first hand controller or a second hand controller. In some embodiments, the first control mode selection input may be received via a button or a touch input device on the right hand controller or the left hand controller. In some embodiments, the first control mode may be a view mode. In some embodiments, the first control mode may be a scan mode. In some embodiments, the first control mode may be a drive mode. In some embodiments, the first control mode may be a translate mode. In some embodiments, the first control mode may be a swivel mode.

[0015] In some embodiments, the method may further include receiving a second control mode selection input and, in response to the second control mode selection input, changing a current control mode of the surgical robotic system to a second control mode different from the first control mode, wherein the control modes of the surgical robotic system include two or more of a scanning mode, a view mode, a drive mode, a translation mode, and a turn mode, and the method is responsive to a second movement of the right hand controller or the left hand controller while the current control mode is the second control mode, the second movement including a second translation and a second rotation. If the second control mode is the scan mode, the method includes holding the robotic arm assembly stationary while rotating the camera assembly a corresponding second rotation relative to the displayed camera view. If the second control mode is a running mode, the method includes moving a corresponding instrument tip by a corresponding second translation, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on the average tip position, or while translating the robot assembly to translate a chest point of the virtual chest and rotating the robot assembly to rotate the virtual chest, or both, to maintain a distance between the center of the virtual chest and the average tip position within an acceptable distance range, and to maintain an angular deviation between a line from the chest point to the average tip position and a normal to the virtual chest within an acceptable angular deviation range. If the second control mode is a view mode, the method includes moving the camera assembly and rotating the virtual chest of the robot assembly by a corresponding second movement, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while maintaining the position and orientation of each instrument to maintain an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.If the second control mode is a swivel mode, the method may include rotating the camera assemblies while moving the corresponding instrument tips by corresponding second movements including corresponding second rotations relative to the displayed camera view to center the camera assembly view on the average instrument tip position, causing a change in orientation of the camera assemblies, or a change in orientation of the camera assemblies and a change in orientation of the virtual chest, to keep the camera assembly view centered on the average instrument tip position and maintaining an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range. If the second control mode is a translation mode, the method includes rotating the camera assemblies, or rotating the camera assemblies and translating the virtual chest while moving the corresponding instrument tips by corresponding second movements including corresponding second translations and corresponding second rotations relative to the displayed camera view to center the camera assembly view on the average tip position and maintaining a distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0016] In some embodiments, the virtual chest is defined by a chest plane extending between a first pivot point of the most proximal joint of the first robotic arm, a second pivot point of the most proximal joint of the second robotic arm, and a camera imaging center point of the camera assembly. In some embodiments, the method may further include receiving a first function selection input corresponding to an input function located on the left hand controller, the right hand controller, or both, and performing a first function in response to receiving the first function selection input. In some embodiments, performing the first function includes displaying a menu on a portion of the video display while holding constant the positions and orientations of the camera assembly and the robotic arm assembly, resetting the orientation and position of the camera assembly to align the camera view with the average instrument tip position and the virtual chest, or activating a clutch for the first hand controller or the second hand controller to allow movement of the respective hand controller without causing movement of the robotic assembly.

[0017] In some embodiments, the first function includes displaying a menu on a portion of the image display while holding the positions and orientations of the camera assembly and the robotic arm assembly constant, and the method further includes receiving one or more second function selection inputs corresponding to one or more input functions located on the left hand controller, the right hand controller, or both, and in response to receiving the one or more second function selection inputs, graphically indicating options on the portion of the image display for one or more menu items, one of which is selected based on the one or more second function inputs, and displaying one or more sub-menus on a portion of the image display corresponding to the one or more second function inputs.

[0018] In some embodiments, the method also includes receiving at least one third feature selection input selecting a choice on the image display corresponding to an input function located on the left hand controller, the right hand controller, or both. If the at least one third feature selection input selects a zoom level, in response to receiving the at least one third feature selection input, the method may include changing the zoom level of the displayed camera view and storing information about the new zoom level for zoom-level-dependent control mode modification. If the at least one third feature selection input includes selecting an elbow force level or selecting an elbow force menu and selecting an elbow force level from the elbow force menu, in response to receiving the third feature selection input, the method may include modifying an elbow force of the first robotic arm, an elbow force of the second robotic arm, or both.

[0019] In some embodiments, the present disclosure is directed to a non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic surgical system, perform one of the methods described above.

[0020] In some embodiments, the present disclosure is directed to a surgical robotic system for performing a surgical procedure within an internal cavity of a subject, the surgical robotic system including: right and left hand controllers operative to operate the surgical robotic system; a camera assembly; a robotic arm assembly configured to be inserted into the internal cavity during use, the robotic arm assembly including a first robotic arm including or coupled to a first instrument tip disposed at a distal end of the first robotic arm and a second robotic arm including or coupled to a second instrument tip disposed at a distal end of the second robotic arm; an image display for outputting images from the camera assembly; and at least one computing module or control unit. The control unit may be configured to receive from the right and left hand controllers and, in response thereto, generate control signals based on a current control mode of the surgical robotic system. The control unit may additionally be configured to receive a control mode selection input and, in response thereto, change the current control mode of the surgical robotic system to a selected one of a plurality of control modes of the surgical robotic system. The control unit may be additionally configured to receive a function selection input from the first hand controller and / or the left hand controller and generate a control signal to perform a corresponding function of the plurality of functions.

[0021] In some embodiments, the plurality of control modes include one or more of a scan mode, a view mode, a drive mode, a swivel mode, and a translate mode. While the current control mode is the scan mode, in response to a first movement including a first rotation of the right hand controller or the left hand controller, the robotic arm assembly may be held stationary while rotating at a corresponding first rotation relative to a view of the camera assembly displayed on the video display, the displayed camera view. While the current control mode is a drive mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, the corresponding instrument tip may move by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the displayed camera view while the camera assembly rotates to center the view of the camera assembly on a location of the midpoint between the instrument tips of the robot arm of the robot assembly, which is the average tip position, or while the robot assembly translates to translate a chest point of the virtual chest and rotates the robot assembly to rotate the virtual chest, or both, to maintain the distance between the center of the virtual chest and the average tip position within an acceptable distance range and to maintain the angular deviation between the line from the chest point to the average tip position and the normal to the virtual chest within an acceptable angular deviation range. While the current control mode is the driving mode, in response to a first movement, including a first translation and a first rotation, of the right hand controller or the left hand controller, the position and orientation of each instrument can be maintained while the camera assembly moves by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, and the virtual chest of the robot assembly rotates to maintain the angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.While the current control mode is a pivot mode, in response to a first movement including a first rotation of the right hand controller or the left hand controller, the corresponding instrument tip may move by a corresponding first movement including a corresponding first translation and a first rotation relative to the displayed camera view, causing the camera assembly to rotate to center the camera assembly's view on the average instrument tip position, causing a change in the camera assembly orientation, or a change in the camera assembly orientation and a change in the virtual chest orientation, to maintain the camera assembly's view centered on the average instrument tip position and maintaining the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range. While the current control mode is a translation mode, in response to a first movement, including a first translation and a first rotation, of the right hand controller or the left hand controller, the corresponding instrument tip may move by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while the camera assembly rotates, or while the camera assembly rotates and the virtual chest translates, to center the camera assembly view on the average tip position and maintain the distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0022] In some embodiments, the multiple modes also include an instrument control mode in which movement of the right hand controller or the left hand controller causes corresponding movement of an instrument tip at the distal end of the corresponding robotic arm relative to the displayed camera view without moving or changing the orientation of the virtual chest of the robotic assembly. In some embodiments, the multiple modes include two or more of a scan mode, a view mode, a drive mode, a swivel mode, and a translate mode. In some embodiments, the multiple modes include three or more of a scan mode, a view mode, a drive mode, a swivel mode, and a translate mode. In some embodiments, the multiple modes include a scan mode. In some embodiments, the multiple modes include a view mode. In some embodiments, the multiple modes include a drive mode. In some embodiments, the multiple modes include a swivel mode. In some embodiments, the multiple modes include a translate mode. In some embodiments, the surgical robotic system includes at least one pedal and a control mode selection input received via the at least one pedal. In some embodiments, the surgical robotic system includes at least one pedal, and the control mode selection input is received via the at least one pedal or via the right hand controller or the left hand controller. In some embodiments, a first control mode selection input is received via a button or touch input device on the right hand controller or the left hand controller, or via at least one pedal. In some embodiments, the virtual chest is defined by a chest plane extending between a first pivot point of the most proximal joint of the left robotic arm, a second pivot point of the most proximal joint of the right robotic arm, and a camera imaging center point of the camera assembly.

[0023] In some embodiments, the functions include displaying a menu on a portion of the video display while holding the position and orientation of the camera assembly and the robotic arm assembly constant. In some embodiments, the functions include displaying a submenu on a portion of the video display. In some embodiments, the functions include selecting an option from a menu or submenu displayed on a portion of the video display. In some embodiments, the functions include resetting the orientation and position of the camera assembly to align the camera view with the average instrument tip position and the virtual chest. In some embodiments, the functions include resetting the pose of the robot assembly relative to the robotic arms and the virtual chest to a default pose while maintaining the position and orientation of the instrument tip relative to each of the robotic arms. In some embodiments, the functions include activating a clutch for the first hand controller or the second hand controller to allow movement of the respective first hand controller or the second hand controller without causing movement of the robotic assembly. In some embodiments, the functions include activating or exiting an instrument control mode of the surgical robotic system, wherein, when in the instrument mode, movement of one of the hand controllers causes a corresponding movement of a corresponding robotic arm of the robotic assembly. In some embodiments, the plurality of functions includes activating or deactivating an elbow biasing feature, the elbow biasing feature allowing adjustment of the magnitude and direction of elevation of the elbow of a robotic arm while maintaining a position of the instrument tip of the robotic arm relative to each robotic arm of the robotic assembly, and controlling adjustment of the elevation of the elbow of a robotic arm of a robotic assembly for one robotic arm of the robotic assembly or for each robotic arm of the robotic assembly. In some embodiments, the plurality of functions includes changing or selecting a zoom of a view from a camera assembly of the robotic assembly.

[0024] These and other features and advantages of the present invention will be more fully understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like reference characters refer to like elements throughout the various views, illustrating the principles of the invention and showing relative dimensions, although not to scale, in which: [Brief explanation of the drawings]

[0025] [Figure 1] 1 illustrates a schematic representation of a surgical robotic system, according to some embodiments. [Figure 2A] FIG. 1 illustrates a perspective view of a patient cart including a robotic support system coupled to a robotic subsystem of a surgical robotic system, according to some embodiments. [Figure 2B] FIG. 1 is a perspective view of an exemplary operator console of the surgical robotic system of the present disclosure, according to some embodiments. [Figure 3A] 1A and 1B schematically illustrate a side view of a surgical robotic system for performing a surgical procedure within an internal cavity of a subject, according to some embodiments. [Figure 3B] 3B schematically illustrates a top view of a surgical robotic system for performing a surgical procedure within an internal cavity of the object of FIG. 3A, according to some embodiments. [Figure 4A] FIG. 1 illustrates a perspective view of a single robotic arm subsystem, according to some embodiments. [Figure 4B] FIG. 4B is a side perspective view of a single robotic arm of the single robotic arm subsystem of FIG. 4A, according to some embodiments. [Figure 5] FIG. 10 illustrates a front perspective view of a camera assembly and a robotic arm assembly, according to some embodiments. [Figure 6A] FIG. 1 is a perspective view of a left hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 6B] FIG. 1 is a perspective view of a right hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 7A]FIG. 1 is a perspective view of a left hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 7B] FIG. 1 is a perspective view of a right hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 8A] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8B] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8C] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8D] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8E] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8F] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 8G] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system demonstrating operational modes and functionality, according to some embodiments. [Figure 9A] FIG. 1 is a perspective view of a left hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 9B] FIG. 1 is a perspective view of a right hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 10] FIG. 1 is a perspective view of a hand controller for use in an operator console of a surgical robotic system, according to some embodiments. [Figure 11A] FIG. 10 schematically illustrates a cross-sectional top view of a hand controller having paddles in a first configuration and employing linear sensors to determine the configuration of the paddles, according to some embodiments. [Figure 11B] FIG. 11B schematically illustrates a cross-sectional top view of the hand controller of FIG. 11A with the paddle in a second, more closed configuration, according to some embodiments. [Figure 12A] FIG. 10 schematically illustrates a cross-sectional top view of a hand controller having paddles in a first configuration employing linear sensors positioned more distally at the ends of the hand controller to determine the configuration of the paddles, according to some embodiments. [Figure 12B] FIG. 12B schematically illustrates a cross-sectional top view of the hand controller of FIG. 12A with the paddles in a second, more closed configuration, according to some embodiments. [Figure 13A] FIG. 10 schematically illustrates a cross-sectional top view of a hand controller having paddles in a first configuration and employing a rotary sensor to determine the configuration of the first and second paddles, according to some embodiments. [Figure 13B] FIG. 13B schematically illustrates a cross-sectional top view of the hand controller of FIG. 13A with the paddles in a second, more closed configuration, according to some embodiments. [Figure 14A] FIG. 10 schematically illustrates a cross-sectional top view of a hand controller having paddles in a first configuration and extending proximally and having a rotary sensor disposed at a distal end of the hand controller for determining the configuration of the first and second paddles, according to some embodiments. [Figure 14B] FIG. 14B schematically illustrates a cross-sectional top view of the hand controller of FIG. 14A with the paddles in a second, more open configuration, according to some embodiments. [Figure 15A] FIG. 1 illustrates a perspective view of a hand controller, according to some embodiments. [Figure 15B] FIG. 12B is an inner side view of the hand controller of FIG. 12A. [Figure 15C] FIG. 12B is a top view of the hand controller of FIG. 12A. [Figure 15D] FIG. 12B is an exterior side view of the hand controller of FIG. 12A. [Figure 16A] FIG. 1 illustrates a perspective view of a hand controller, according to some embodiments. [Figure 16B] 16B is a perspective view of a right or left hand controller, each having a structure like that of FIG. 16A, with different functions mapped to buttons and touch input devices, according to some embodiments. [Figure 16C] 16B is a perspective view of a right or left hand controller, each having a structure like that of FIG. 16A, with different functions mapped to buttons and touch input devices, according to some embodiments. [Figure 17] 1 illustrates a schematic diagram of a graphical user interface (GUI) displayed to an operator, according to some embodiments. [Figure 18A] FIG. 1 is a perspective view of a left hand controller for use in connection with an operator console of a surgical robotic system, according to some embodiments. [Figure 18B] FIG. 1 is a perspective view of a right hand controller for use in connection with an operator console of a surgical robotic system, according to some embodiments. [Figure 19A] FIG. 1 is a perspective view of a left hand controller for use in connection with an operator console of a surgical robotic system, according to some embodiments. [Figure 19B] FIG. 1 is a perspective view of a right hand controller for use in connection with an operator console of a surgical robotic system, according to some embodiments. [Figure 20] 10A and 10B are schematic illustrations of a graphical user interface including a camera view portion displaying views from a camera assembly and menus; [Figure 21]1 is a feature map table illustrating functions accessed by the hand controller, foot pedals, and menus. [Figure 22] 1 is a flowchart illustrating steps performed by a surgical robotic system for controlling a robotic assembly using a hand controller, according to some embodiments. [Figure 23] 10A-10C schematically illustrate the trocar plane and various points and directions associated with the virtual chest and camera assembly of the robotic assembly and the trocar, according to some embodiments. [Figure 24] 10A-10C schematically illustrate positioning and orienting the chest plane to keep the distance from chest points to the average instrument tip position within an acceptable range and to keep the angular deviation of the average instrument tip position from the chest normal / chest direction within an acceptable range, according to some embodiments. [Figure 25] FIG. 1 is a perspective view of an operator console of a surgical robotic system featuring a foot pedal array, according to some embodiments. [Figure 26] FIG. 26 is a perspective view of a foot pedal array of the operator console illustrated in FIG. 25, according to some embodiments. [Figure 27] FIG. 10 is a perspective view of another operator console of a surgical robotic system featuring a foot pedal array including a different arrangement of the fifth and sixth foot pedals, according to some embodiments. [Figure 28] FIG. 12 is a perspective view of a foot pedal array for an operator console of a surgical robotic system including both column and row sensors, according to some embodiments. [Figure 29] FIG. 28 is a side view of the foot pedal portion of FIG. 27 illustrating an operator's foot breaking the sensor beam, according to some embodiments. [Figure 30] 1 illustrates a schematic graphical user interface (GUI) for an operator, including a central area for displaying images based on image input from a camera assembly. DETAILED DESCRIPTION OF THE INVENTION

[0026] While various embodiments of the present invention have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions may occur to those skilled in the art without departing from the invention. It will be understood that various alternatives to the embodiments of the invention described herein may be employed.

[0027] As used in this specification and claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. It will be further understood that the terms "comprises" and / or "comprising," or "include" and / or "including," as used herein, indicate the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. The term "and / or," as used herein, includes any and all combinations of one or more of the associated listed items.

[0028] Unless specifically stated or apparent from the context, the term "about" as used herein is understood to mean within normal tolerances in the art, for example, within two standard deviations of the mean. "About" may be understood to mean within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. Unless otherwise apparent from the context, all numerical values ​​provided herein are modified by the term "about."

[0029] While exemplary embodiments are described as using multiple units to perform exemplary processes, it is understood that exemplary processes may also be performed by one or more modules. Furthermore, the term controller / control unit is understood to refer to a hardware device, including a memory and a processor, specifically programmed to perform processes according to some embodiments described herein. In some embodiments, the memory is configured to store modules, and the processor is specifically configured to execute the modules to perform one or more processes described further below. In some embodiments, multiple different controllers or control units, or multiple different types of controllers or control units, may be employed to perform one or more processes. In some embodiments, different controllers or control units may be implemented in different portions of a surgical robotic system.

[0030] Embodiments provide a hand controller for receiving input from an operator to control a robotic assembly of a surgical robotic system, an operator console including such a hand controller, and a surgical robotic system including such a hand controller and operator console, a method for controlling a robotic assembly of a surgical robotic system, a surgical robotic system configured to implement the control method, and a computer-readable medium including instructions for implementing the control method.

[0031] As disclosed herein, some advantages of embodiments employing hand controllers include increased input options for the user, increased ease of access to modes and controls of the surgical robotic device, providing improved ergonomics and comfort for the user, and increasing the user's degree of control over the surgical robotic system. Further advantages of some embodiments of the present technology may include increased speed and efficiency in operator control of the system as a result of easier access to modes and controls. This may reduce the overall time of the surgical procedure, providing health benefits to the patient.

[0032] Some embodiments disclosed herein are implemented on, employ, or incorporate into a surgical robotic system including a camera assembly having at least three degrees of freedom of articulation and two or more robotic arms, each having at least six degrees of freedom of articulation and an additional degree of freedom corresponding to the movement of an associated end effector (e.g., grasper, manipulator, etc.). In some embodiments, the camera assembly, when mounted within a subject (e.g., a patient), can move or rotate approximately 180 degrees in a pitch or yaw direction so that the camera assembly can look rearward toward the insertion site. In this manner, the camera assembly and robotic arms can maneuver by looking to either side of the front (e.g., away from the insertion site) in an upward or downward direction, as well as in a backward direction, to look rearward toward the insertion site. The robotic arms and camera assembly can also move in a roll, pitch, and yaw direction.

[0033] Some embodiments described herein provide methods and systems that employ multiple different control modes for controlling a robotic assembly of a surgical robotic system when the robotic assembly is positioned within an internal body cavity of a subject. In some embodiments, the robotic assembly includes at least two robotic arms, which may be described herein as a "robotic arm assembly" or "arm assembly." In some embodiments, the robotic assembly also includes a camera assembly, which may be referred to herein as a "surgical camera assembly" or "robotic camera assembly."

[0034] Each control mode may use sensed movements of one or more hand controllers, and also input from one or more foot pedals, to control the robotic arm assembly and / or camera assembly. The control mode may be changed from a current control mode to a different selected control mode based on operator input (e.g., provided via one or more hand controllers and / or pedals of a surgical robotic system). In different control modes, the same movement of a hand controller may result in different movements of the robotic assembly.

[0035] Some embodiments employ multiple control modes, including an instrument control mode, which may also be referred to herein as an "instrument mode," one or more of a view control mode, which may also be referred to herein as a "view mode," "camera control mode," "camera mode," "framing control mode," "framing mode," or "point of view mode," a scan mode, which may also be referred to herein as a "scanning mode" or "survey mode," and a drive control mode, which may also be referred to herein as a "drive mode" or "auto-track mode." The swivel mode, drive mode, and translate mode may all be referred to herein as "track modes."

[0036] Some embodiments employ additional features for controlling the robotic assembly. For example, some embodiments allow for individual control of elbow bias or elbow elevation for the right and left robotic arms. Some embodiments employ a graphical user interface that identifies the current control mode of the surgical robotic system. Some embodiments employ a menu feature in which a menu is displayed on the graphical user interface, and one or more of the hand controllers can be used to traverse and select menu options.

[0037] The control modes and methods described herein can be particularly advantageous in surgical robotic systems that are more maneuverable than conventional systems. For example, many conventional surgical robotic systems with two robotic arms and fewer degrees of freedom per arm may not be able to change the position or orientation of the virtual chest of the robotic assembly while keeping the instrument tips of the end effectors of the robotic arms stationary. As another example, the cameras in many conventional surgical robotic systems may only have degrees of freedom associated with the movement of the support relative to the camera extending through a trocar, and may not have independent degrees of freedom of movement relative to the support.

[0038] This large number of degrees of freedom in some surgical robotic systems described herein, compared to some conventional surgical robotic systems, allows for movement of the robotic arm assembly and orientation of the robotic arm assembly that is not possible with some conventional surgical robotic arms, and allows for movement of the camera of the robotic camera assembly that is not possible with cameras for some conventional robotic surgical systems.

[0039] Some embodiments described herein provide methods and systems employing multiple different control modes for controlling a surgical robotic system including a robotic assembly configured to be positioned within a subject's internal body cavity. The robotic assembly may include at least two robotic arms, which may be described herein as a robotic arm assembly or "arm assembly." The robotic assembly may also include a camera assembly, which may be referred to herein as a "surgical camera assembly" or "robotic camera assembly." Each control mode uses sensed movements of one or more hand controllers to control the robotic arm assembly and / or camera assembly. The control mode may be changed from a current control mode to a different selected control mode based on operator input. In different control modes, the same movement of the hand controllers results in different movements of the robotic assembly.

[0040] In some methods and systems, the multiple different control modes, which may be described as multiple control modes, include at least one view control mode, also referred to herein as a camera control mode, in which the position and / or orientation of at least a portion of the camera assembly, at least a portion of the robotic arm assembly, and at least a portion of the camera assembly are changed in response to movement of at least one of the hand controllers while maintaining a stationary position of the instrument tip of the end effector disposed at the distal end of the robotic arm. In some systems and methods, the multiple different control modes include at least one travel arm control mode. In the travel mode, which is a tracking mode, the chest may translate, the arm and chest may translate together, the chest orientation may be changed, or any combination of the above, to center the camera view on the instrument tip. In the camera / view mode, the surgical robotic system may rotate the camera assembly, translate the chest, or pivot the chest to change the view direction and viewpoint orientation in response to movement of the hand controllers while the instrument tip of the robotic arm is in a stationary position. In translation tracking mode, the chest or chest and arms may be translated together while the camera view is centered on the instrument tip. In orbit tracking mode, the orientation of the camera view may be changed to center the camera view on the instrument tip.

[0041] Before providing additional specific descriptions of the hand controller and operator input controls to the operator console with respect to FIGS. 6-21, methods for control of a surgical robotic system with respect to FIGS. 22-24, and foot pedal array with respect to FIGS. 25-30, a surgical robotic system with which some embodiments may be employed will be described below with respect to FIGS. 1-5.

[0042] Surgical Robot System Some embodiments may be employed in surgical robotic systems. A system for robotic surgery may include a robotic subsystem. The robotic subsystem includes at least a portion, which may be referred to herein as a robot assembly, that can be inserted into a patient via a trocar through a single incision point or site. The portion inserted into the patient via the trocar is small enough to be deployed in vivo at the surgical site and is sufficiently maneuverable when inserted into the body so that it can be moved within the body to perform various surgical procedures at multiple different points or sites. Herein, the portion inserted into the body that performs a functional task may be referred to as a surgical robotic unit, surgical robotic module, or robotic assembly. A surgical robotic unit or surgical robotic module may include multiple different subunits or parts that are separately insertable into a trocar. A surgical robotic unit, surgical robotic module, or robotic assembly includes multiple separate robotic arms that can be deployed into a patient along different or separate axes. These multiple separate robotic arms may be collectively referred to herein as a robotic arm assembly. Additionally, a surgical camera assembly may also be deployed along a separate axis. The surgical robotic unit, surgical robotic module, or robotic assembly may also include a surgical camera assembly. Thus, a surgical robotic unit, surgical robotic module, or robotic assembly employs multiple distinct components, such as a pair of robotic arms and a surgical or robotic camera assembly, each of which is deployable along a different axis and separately operable, steerable, and movable. A robotic arm and camera assembly that are positionable along separate operable axes is referred to herein as a split-arm (SA) architecture. The SA architecture is designed to simplify and increase the efficiency of robotic surgical instrument insertion through a single trocar at a single insertion site, while also assisting in the deployment of the surgical instruments into a surgical-ready state and their subsequent removal through the trocar. As an example, surgical instruments may be inserted through a trocar to access a patient's abdominal cavity and perform surgery in vivo.In some embodiments, a variety of surgical instruments may be used or employed, including, but not limited to, robotic surgical instruments and other surgical instruments known in the art.

[0043] The systems, devices, and methods disclosed herein may be incorporated into and / or used in conjunction with, for example, the robotic surgical devices and related systems disclosed in U.S. Patent No. 10,285,765 and PCT Patent Application No. PCT / US2020 / 39203, the camera assemblies and systems disclosed in U.S. Publication No. 2019 / 0076199, and / or the systems and methods for exchanging surgical tools in an implantable surgical robotic system disclosed in PCT Patent Application No. PCT / US2021 / 058820, the entire contents and teachings of which are incorporated herein by reference. A surgical robotic unit forming part of the present invention may, in some embodiments, form part of a surgical robotic system including a surgeon workstation including appropriate sensors and displays, and a robotic support system (RSS) for supporting and interacting with the robotic subsystem of the present invention. The robotic subsystem, in some embodiments, includes a motor unit and a surgical robotic unit including one or more robotic arms and one or more camera assemblies. The robotic arm and camera assembly may form part of a single support axis robotic system, a split-arm (SA) architecture robotic system, or other arrangements. The robotic support system can provide multiple degrees of freedom so that the robotic unit can be maneuvered to a single location or multiple different locations within the patient's body. In one embodiment, the robotic support system can be attached directly to the operating table or to the floor or ceiling within the operating room. In another embodiment, attachment is achieved by various fastening means, including, but not limited to, clamps, screws, or combinations thereof. In other embodiments, the structure may be freestanding. The robotic support system can attach a motor assembly that couples to the surgical robotic unit, including the robotic arm and camera assembly. The motor assembly can include gears, motors, drivetrains, electronics, etc., for powering the components of the surgical robotic unit.

[0044] The robotic arm and camera assembly are capable of multiple degrees of freedom of movement. According to some embodiments, when the robotic arm and camera assembly are inserted into a patient through a trocar, they are capable of movement in at least the axial, yaw, pitch, and roll directions. The robotic arm is designed to incorporate and employ a multi-degree-of-freedom of movement robotic arm having an end effector attached to its distal end that corresponds to the area or joint of a user's wrist. In other embodiments, the working end (e.g., end effector end) of the robotic arm is designed to incorporate, use, or employ other robotic surgical instruments, such as, for example, the surgical instruments described in U.S. Publication No. 2018 / 0221102, the contents of which are incorporated herein by reference.

[0045] 1 is a schematic illustration of an exemplary surgical robotic system 10 in which aspects of the present disclosure may be employed according to some embodiments of the present disclosure. The surgical robotic system 10 includes an operator console 11 and a robotic subsystem 20, according to some embodiments.

[0046] The operator console 11 includes a display device or unit 12, an image computing unit 14, which may be a three-dimensional (3D) computing unit, a hand controller 17 having a sensing and tracking unit 16, and a computing unit 18. Additionally, the operator console 11 may include a foot pedal array 19 including a plurality of pedals.

[0047] The display unit 12 may be any selected type of display for displaying information, images, or video generated by the image computation unit 14, the computation unit 18, and / or the robotic subsystem 20. The display unit 12 may include or form part of, for example, a head-mounted display (HMD), an augmented reality (AR) display (e.g., an AR display or AR glasses combined with a screen or display), a screen or display, a two-dimensional (2D) screen or display, a three-dimensional (3D) screen or display, or the like. The display unit 12 may also include an optional sensing and tracking unit 16A. In some embodiments, the display unit 12 may include an image display for outputting images from a camera assembly 44 (see FIG. 17 ) of the robotic subsystem 20.

[0048] In some embodiments, if the display unit 12 includes an HMD device, an AR device that senses head position, or another device with an associated sensing and tracking unit 16A, the HMD device or head tracking device generates tracking and position data 34A that is received and processed by the image computation unit 14. In some embodiments, the HMD, AR device, or other head tracking device can provide an operator (e.g., a surgeon, nurse, or other suitable medical professional) with a display at least partially coupled to or attached to the operator's head, a lens that enables a focused view of the display, and a sensing and tracking unit 16A for providing position and orientation tracking of the operator's head. The sensing and tracking unit 16A can include, for example, an accelerometer, a gyroscope, a magnetometer, a motion processor, infrared tracking, eye tracking, computer vision, emitting and sensing an alternating magnetic field, and any other method of tracking at least one of position and orientation, or any combination thereof. In some embodiments, the HMD or AR device can provide image data from a camera assembly 44 to the operator's right and left eyes. In some embodiments, to maintain the operator's virtual reality experience, the sensing and tracking unit 16A tracks the position and orientation of the operator's head and generates tracking and position data 34A, which can then be relayed to the image calculation unit 14 and / or the calculation unit 18 directly or via the image calculation unit 14.

[0049] The hand controller 17 is configured to sense the movement of the operator's hand and / or arm to operate the surgical robotic system 10. The hand controller 17 may include a sensing and tracking unit 16, circuitry, and / or other hardware. The sensing and tracking unit 16 may include one or more sensors or detectors that sense the movement of the operator's hand. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are disposed within a pair of hand controllers that are grasped by or engaged with the operator's hands. In some embodiments, the one or more sensors or detectors that sense the movement of the operator's hand are coupled to the operator's hand and / or arm. For example, sensors of the sensing and tracking unit 16 may be coupled to regions of the hand and / or arm, such as the fingers, wrist region, elbow region, and / or shoulder region. If an HMD is not used, in some embodiments, additional sensors may also be coupled to the operator's head and / or neck region. If the operator uses an HMD, eye, head, and / or neck sensors and associated tracking technology may be incorporated into or used within the HMD device and, as described above, may therefore form part of optional sensor and tracking unit 16A. In some embodiments, sensing and tracking unit 16A may be external and coupled to hand controller 17 via electrical components and / or mounted hardware. In some embodiments, optional sensor and tracking unit 16A may sense and track movement of one or more of the operator's head, the operator's eyes, or at least a portion of the operator's neck based at least in part on imaging of the operator, in addition to, or instead of, sensors attached to the operator's body.

[0050] In some embodiments, the sensing and tracking unit 16 may use sensors coupled to the operator's torso or any other body part. In some embodiments, the sensing and tracking unit 16 may use, in addition to sensors, an inertial momentum unit (IMU) having, for example, an accelerometer, a gyroscope, a magnetometer, and a motion processor. The addition of a magnetometer may reduce sensor drift around the vertical axis. In some embodiments, the sensing and tracking unit 16 also includes sensors placed within surgical equipment such as gloves, surgical scrubs, or a surgical gown. The sensors may be reusable or disposable. In some embodiments, the sensors may be located external to the operator, such as in a fixed location in a room such as an operating room. The external sensors 37 may generate external data 36 that may be processed by the computing unit 18 and thus used by the surgical robotic system 10.

[0051] The sensors generate position and / or orientation data indicative of the position and / or orientation of the operator's hands and / or arms. The sensing and tracking units 16 and / or 16A may be utilized to control the movement (e.g., change in position and / or orientation) of the camera assembly 44 and the robotic arm 42 of the robotic subsystem 20. The tracking and position data 34 generated by the sensing and tracking units 16 may be communicated to the computing unit 18 for processing by at least one processor 22.

[0052] The computing unit 18 may determine or calculate the position and / or orientation of the operator's hands or arms, and in some embodiments, the operator's head, from the tracking and position data 34 and 34A and communicate the tracking and position data 34 and 34A to the robotic subsystem 20. The tracking and position data 34, 34A may be processed by the processor 22 and stored, for example, in the storage unit 24. The tracking and position data 34 and 34A may also be used by the control unit 26, which may responsively generate control signals to control the movement of the robotic arm 42 and / or the camera assembly 44. For example, the control unit 26 may change the position and / or orientation of at least a portion of the camera assembly 44, at least a portion of the robotic arm 42, or both. In some embodiments, the control unit 26 may also adjust the pan and tilt of the camera assembly 44 to follow the movement of the operator's head.

[0053] The robotic subsystem 20 may include a robotic support system (RSS) 46 having a motor unit 40 and a trocar 50 or trocar mount, a robotic arm 42, and a camera assembly 44. The robotic arm 42 and camera assembly 44 may form part of a single support axis robotic unit as disclosed and described in U.S. Pat. No. 10,285,765, or may form part of a split-arm (SA) architecture robotic system as disclosed and described in PCT Patent Application No. PCT / US2020 / 039203, both of which are incorporated herein by reference in their entireties.

[0054] The robotic subsystem 20 may employ multiple distinct robotic arms deployable along different or separate axes. In some embodiments, the camera assembly 44, which may employ multiple distinct camera elements, may also be deployed along a common, separate axis. Thus, the surgical robotic system 10 may employ multiple distinct components, such as a pair of distinct robotic arms and camera assemblies 44 deployable along different axes. In some embodiments, the robotic arm 42 and camera assembly 44 are independently steerable, steerable, and movable. The robotic subsystem 20, including the robotic arm 42 and camera assembly 44, is free to operate along separate steerable axes and is referred to herein as an SA architecture. The SA architecture is designed to simplify and increase the efficiency of insertion of robotic surgical instruments through a single trocar at a single insertion point or site, while also assisting in the deployment of the surgical instruments into a surgical-ready state and the subsequent removal of the surgical instruments through the trocar 50, as further described below.

[0055] The RSS 46 may include a motor unit 40 and a trocar 50 or trocar mount. The RSS 46 may further include a support member supporting the motor unit 40 coupled to its distal end. The motor unit 40 may in turn be coupled to each of the camera assembly 44 and the robotic arm 42. The support member may be configured and controlled to move one or more components of the robotic subsystem 20 linearly or in any other selected direction or orientation. In some embodiments, the RSS 46 may be freestanding. In some embodiments, the RSS 46 may include a motor unit 40 coupled at one end to the robotic subsystem 20 and at an opposite end to an adjustable support member or element.

[0056] The motor unit 40 may receive control signals generated by the control unit 26. The motor unit 40 may include gears, one or more motors, a drivetrain, electronics, etc. for powering and driving the robotic arm 42 and the camera assembly 44, individually or together. The motor unit 40 may also provide mechanical power, electrical power, mechanical communications, and electrical communications to the robotic arm 42, the camera assembly 44, and / or the RSS 46 and other components of the robotic subsystem 20. The motor unit 40 may be controlled by the computing unit 18. Thus, the motor unit 40 may generate signals to control one or more motors that may control and drive the robotic arm 42, including, for example, the position and orientation of each articulating joint of each robotic arm, and the camera assembly 44. The motor unit 40 may further provide translational or linear degrees of freedom that are primarily utilized to insert and remove each component of the robotic subsystem 20 through the trocar 50. The motor unit 40 may also be employed to adjust the insertion depth of each robotic arm 42 as it is inserted into the patient 100 through the trocar 50.

[0057] The trocar 50 is a medical device that, in some embodiments, may consist of a claw (which may be a metal or plastic, sharp or non-bladed tip), a cannula (essentially a hollow tube), and a seal. The trocar may be used to place at least a portion of the robotic subsystem 20 within an internal cavity of a subject (e.g., a patient) and may withdraw gases and / or fluids from the body cavity. The robotic subsystem 20 may be inserted through the trocar to access the patient's body cavity and perform surgery in vivo. In some embodiments, the robotic subsystem 20 of the present invention may be supported, at least in part, by a trocar 50 or trocar mount with multiple degrees of freedom so that the robotic arm 42 and camera assembly 44 may be maneuvered within the patient to a single position or multiple different positions. In some embodiments, the robotic arm 42 and camera assembly 44 may be supported by a trocar 50 or trocar mount with multiple degrees of freedom so that the robotic arm 42 and camera assembly 44 may be maneuvered within the patient to a single position or multiple different positions.

[0058] In some embodiments, the RSS 46 may further include an optional controller for processing input data from one or more of the system components (e.g., the display 12, the sensing and tracking unit 16, the robotic arm 42, the camera assembly 44, etc.) and for generating control signals in response thereto. The motor unit 40 may also, in some embodiments, include a memory element for storing data.

[0059] In some embodiments, and in some modes of operation, the robotic arm 42 may be controlled to follow scaled-down movements or motions of an operator's arm and / or hand as sensed by associated sensors. The robotic arm 42 includes a first robotic arm including a first end effector at a distal end of the first robotic arm and a second robotic arm including a second end effector disposed at a distal end of the second robotic arm. In some embodiments, the robotic arm 42 may have portions or regions associated with shoulder, elbow, and wrist joints, as well as movements associated with the operator's fingers. For example, a robotic elbow joint may track the position and orientation of a human elbow, and a robotic wrist joint may track the position and orientation of a human wrist. The robotic arm 42 may also have a terminal region associated with it, which in some embodiments may terminate in an end effector that tracks the movement of one or more of the operator's fingers, such as the index finger, when the user pinches the index finger and thumb together. In some embodiments, the robotic arm 42 may follow the movement of the operator's arm in some control modes, while the virtual chest of the robotic assembly may remain stationary (e.g., in an instrument control mode). In some embodiments, the position and orientation of the operator's torso is subtracted from the position and orientation of the operator's arms and / or hands. This subtraction allows the operator to move the torso without the robotic arm moving. Further disclosure of control of the movement of the individual arms of the robotic arm assembly is provided in International Patent Application Publication Nos. 2022 / 094000 A1 and 2021 / 231402 A1, each of which is incorporated herein by reference in its entirety.

[0060] The camera assembly 44 is configured to provide the operator with image data 48, such as a live video feed of the procedure or surgical site, as well as to allow the operator to operate and control the cameras forming part of the camera assembly 44. In some embodiments, the camera assembly 44 may include one or more cameras (e.g., a pair of cameras) whose optical axes are axially spaced a selected distance, known as the inter-camera distance, to provide a stereoscopic view or image of the surgical site. In some embodiments, the operator can control camera movement through hand movement, via a sensor coupled to the operator's hand or via a hand controller grasped or held by the operator's hand, thus allowing the operator to obtain a desired view of the surgical site in an intuitive and natural manner. In some embodiments, the operator can additionally control camera movement through movement of the operator's head. The camera assembly 44 is movable in multiple directions, including, for example, yaw, pitch, and roll, relative to the direction of view. In some embodiments, the stereoscopic camera components may be configured to provide a natural and comfortable user experience. In some embodiments, the axial distance between the cameras can be modified to adjust the depth of the surgical site as perceived by the operator.

[0061] Image or video data 48 generated by camera assembly 44 may be displayed on display unit 12. In embodiments, if display unit 12 includes an HMD, the display may include an embedded sensing and tracking unit 16A that acquires raw orientation data in the yaw, pitch, and roll directions of the HMD, as well as position data in Cartesian space (x, y, z) of the HMD. In some embodiments, position and orientation data for the operator's head may be provided via a separate head tracking unit. In some embodiments, sensing and tracking unit 16A may be used to provide supplemental position and orientation tracking data for the display instead of, or in addition to, the HMD's embedded tracking system. In some embodiments, operator head tracking is not used or employed. In some embodiments, an image of the operator may be used by sensing and tracking unit 16A to track at least a portion of the operator's head.

[0062] 2A illustrates an exemplary robot assembly 20, also referred to herein as a robotic subsystem, of a surgical robotic system 10 integrated into or mounted on a mobile patient cart, according to some embodiments. In some embodiments, the robot assembly 20 includes an RSS 46, which in turn includes a motor unit 40, a robotic arm 42 having an end effector 45, a camera assembly 44 having one or more cameras 47, and may also include a trocar 50 or a trocar mount.

[0063] 2B illustrates an example of an operator console 11 of the presently disclosed surgical robotic system 10, according to some embodiments. The operator console 11 includes a display unit 12, a hand controller 17, and one or more additional controllers, such as a foot pedal array 19, for controlling the robotic arm 42, the camera assembly 44, and other aspects of the system.

[0064] 2B also illustrates the left and right hand controller subsystems 23A and 23B of the operator console. The left hand controller subsystem 23A includes and supports the left hand controller 17A, and the right hand controller subsystem 23B includes and supports the right hand controller 17B. In some embodiments, the left hand controller subsystem 23A may be removably connected or engaged with the left hand controller 17A, and the right hand controller subsystem 23B may be removably connected or engaged with the right hand controller 17A. In some embodiments, the connections may be both physical and electronic, such that the left and right hand controller subsystems 23A and 23B may receive signals from the left and right hand controllers 17A and 17B, respectively, including signals conveying input received from user selections on buttons or touch input devices of the left or right hand controller 17A or 17B.

[0065] Each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may include components that enable a range of motion for the respective left hand controller 17A and right hand controller 17B, such that the left hand controller 17A and the right hand controller 17B may translate or displace in three dimensions and may additionally move in roll, pitch, and yaw directions. Furthermore, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may register the movement of the respective left hand controller 17A and right hand controller 17B in each of the aforementioned directions and may send signals providing such movement information to a processor (not shown) of the surgical robotic system.

[0066] In some embodiments, each of the left hand controller subsystem 23A and the right hand controller subsystem 23B may be configured to receive and connect to or engage a different hand controller (not shown). For example, hand controllers having different configurations of buttons and touch input devices may be provided. Furthermore, hand controllers having different shapes may be provided. The hand controller may be selected to be compatible with a particular surgical robotic system or a particular surgical robotic procedure, or may be selected based on operator preferences for buttons and input devices or with respect to the shape of the hand controller to provide greater comfort and ease for the operator.

[0067] FIG. 3A schematically illustrates a side view of a surgical robotic system 10 performing surgery within an internal cavity 104 of a subject 100, according to some embodiments and for some surgical procedures. FIG. 3B schematically illustrates a perspective top view of a surgical robotic system 10 performing surgery within an internal cavity 104 of a subject 100. The subject 100 (e.g., a patient) is positioned on a surgical table 102 (e.g., surgical table 102). In some embodiments and for some surgical procedures, an incision is made in the patient 100 to gain access to the internal cavity 104. A trocar 50 is then inserted into the patient 100 at a selected location to provide access to the internal cavity 104 or surgical site. The RSS 46 can then be manipulated into position on the patient 100 and the trocar 50. In some embodiments, the RSS 46 includes a trocar mount that attaches to the trocar 50. The robot assembly 20 may be coupled to the motor unit 40, and at least a portion of the robot assembly may be inserted into the trocar 50 and, therefore, into the internal cavity 104 of the patient 100. For example, the camera assembly 44 and the robotic arm assembly 42 may be inserted individually and sequentially into the patient 100 through the trocar 50. While the camera assembly and the robotic arm assembly may include some portions that remain outside the subject's body during use, references to inserting the robotic arm assembly 42 and / or the camera assembly 44 into the internal cavity of the subject and disposing the robotic arm assembly 42 and / or the camera assembly 44 within the internal cavity of the subject refer to the portions of the robotic arm assembly 42 and the camera assembly 44 that are intended to be within the internal cavity of the subject during use. The sequential insertion method has the advantage of supporting smaller trocars, thus allowing for smaller incisions to be made in the patient 100, thus reducing trauma experienced by the patient 100. In some embodiments, the camera assembly 44 and the robotic arm assembly 42 may be inserted in any order or in a specific order. In some embodiments, the camera assembly 44 may be followed by a first robotic arm of the robotic arm assembly 42, followed by a second robotic arm of the robotic arm assembly 42, all of which may be inserted into the trocar 50 and thus into the internal cavity 104.Once inserted into the patient 100 , the RSS 46 can move the robotic arm assembly 42 and camera assembly 44 to the surgical site, controlled manually or automatically by the operator console 11 .

[0068] Disclosure controls relating to managing the movement of individual arms of a robotic arm assembly are provided in International Patent Application Publication Nos. 2022 / 094000A1 and 2021 / 231402A1, each of which is incorporated herein by reference in its entirety.

[0069] 4A is a perspective view of the robotic arm subassembly 21, according to some embodiments. The robotic arm subassembly 21 includes a robotic arm 42A, an end effector 45 having an instrument tip 120 (e.g., monopolar scissors, a needle driver / holder, a bipolar grasper, or any other suitable tool), and a shaft 122 that supports the robotic arm 42A. A distal end of the shaft 122 is coupled to the robotic arm 42A, and a proximal end of the shaft 122 is coupled to the housing 124 of the motor 40 (as shown in FIG. 2A). At least a portion of the shaft 122 can be external to the internal cavity 104 (as shown in FIGS. 3A and 3B). At least a portion of the shaft 122 can be inserted into the internal cavity 104 (as shown in FIGS. 3A and 3B).

[0070] 4B is a side view of the robotic arm assembly 42, which in some implementations includes a virtual shoulder 126, a virtual elbow 128 with a position sensor 132 (e.g., a capacitive proximity sensor), a virtual wrist 130, and an end effector 45. In some embodiments, the virtual shoulder 126, virtual elbow 128, and virtual wrist 130 can include a series of hinges and revolute joints to provide seven positionable degrees of freedom for each arm, along with an additional grasping degree of freedom for the end effector 45.

[0071] 5 illustrates a perspective front view of a portion of a robotic assembly 20 configured for insertion into a patient's internal body cavity. The robotic assembly 20 includes a first robotic arm 42A and a second robotic arm 42B. The two robotic arms 42A and 42B may, in some embodiments, define a virtual chest 140 of the robotic assembly 20. In some embodiments, the virtual chest 140 may be defined by a chest plane extending between a first pivot point 142A of the most proximal joint of the first robotic arm 42A (e.g., shoulder joint 126), a second pivot point 142B of the most proximal joint of the second robotic arm 42B, and a camera imaging center point 144 of the camera 47. A pivot center 146 of the virtual chest 140 is at the center of the virtual chest.

[0072] In some embodiments, sensors on one or both of the first robotic arm 42A and the second robotic arm 42B may be used by the system to determine a change in position in three-dimensional space of at least a portion of the robotic arms. In some embodiments, sensors on one or both of the first robotic arm and the second robotic arm may be used by the system to determine a position in three-dimensional space of at least a portion of one robotic arm relative to a position in three-dimensional space of at least a portion of the other robotic arm.

[0073] In some embodiments, the camera assembly 44 is configured to acquire images that enable the system to determine relative positions in three-dimensional space. For example, the camera assembly may include multiple cameras, at least two of which are laterally displaced from one another relative to the imaging axis, and the system may be configured to determine distances to features within an internal body cavity. Further disclosure regarding surgical robotic systems including camera assemblies and associated systems for determining distances to features may be found in International Patent Application Publication No. WO 2021 / 159409, entitled "System and Method for Determining Depth Perception In Vivo in a Surgical Robotic System," published August 12, 2021, and incorporated herein by reference in its entirety. Information regarding distances to features and information regarding optical properties of the cameras may be used by the system to determine relative positions in three-dimensional space.

[0074] Hand controller for surgical robotic system The hand controllers for surgical robotic systems described herein can be used in conjunction with any of the surgical robotic systems described above, or any other suitable surgical robotic system. Additionally, some embodiments of the hand controllers described herein can be used in conjunction with semi-robotic endoscopic surgical systems that are only partially robotic.

[0075] As mentioned above, it may be desirable for a controller for a surgical robotic system to feature sufficient inputs to provide control of the system, ergonomic design, and a "natural" feel during use.

[0076] The hand controller, hand controller system, and surgeon or operator console employing the hand controller may be understood with reference to the embodiments illustrated in Figures 6-16, 18, and 19, as described below.

[0077] In some embodiments described herein, reference is made to a left hand controller and corresponding left robotic arm, which may be a first robotic arm, and a right hand controller and corresponding right robotic arm, which may be a second robotic arm. In some embodiments, the robotic arm considered to be the left robotic arm and the robotic arm considered to be the right robotic arm may vary due to the configuration of the robotic arms, and the camera assembly is adjusted so that the second robotic arm corresponds to the left robotic arm relative to the view provided by the camera assembly, and the first robotic arm corresponds to the right robotic arm relative to the view provided by the camera assembly. In some embodiments, the surgical robotic system, during use, identifies a robotic arm corresponding to the left hand controller and a robotic arm corresponding to the right hand controller. In some embodiments, at least one hand controller includes one or more operator input devices for providing one or more inputs for additional control of the robotic assembly. In some embodiments, one or more operator input devices receive one or more operator inputs for at least one of engaging a scanning mode, resetting the orientation and position of the camera assembly to align the camera assembly's field of view with the instrument tip and chest, displaying a menu, traversing a menu or highlighting an option or item for selection, selecting an item or option, selecting and adjusting elbow position, engaging a clutch associated with an individual hand controller, etc. In some embodiments, additional functionality may be accessed via the menu, such as selecting a level of grip force (e.g., high / low), selecting an insertion mode, extraction mode, or exchange mode, adjusting focus, lighting, or gain, camera cleaning, motion scaling, rotating the camera to allow for downward viewing, etc.

[0078] According to some embodiments, Figure 6A illustrates a left hand controller 201 and Figure 6B illustrates a right hand controller 202. The left hand controller 201 and the right hand controller 202 each include a contoured housing 210, 211, respectively. Each contoured housing 210, 211 includes a top surface 212a, 213a, an inner side surface 212b, 213b adjacent the top surface, an outer side surface (not visible in these views) facing away from the inner side surfaces 212b, 213b, and a bottom surface (not visible in these views) facing away from the top surfaces 212a, 213a.

[0079] In some embodiments, each hand controller 201, 202 includes a mounting assembly 215, 216, respectively. The mounting assemblies 215, 216 can be used to directly or indirectly mount the respective hand controller 201, 202 to a surgeon console of a surgical robotic system. In some embodiments, the mounting assembly 215 defines a hole 217, which can be a countersink, configured to receive a screw or bolt to connect the left hand controller 201 to the surgeon console.

[0080] In some embodiments of the present disclosure, such as the embodiment illustrated in FIGS. 6A and 6B, the hand controller includes two paddles, three buttons, and one touch input device. As described herein, embodiments may feature other combinations of touch input devices, buttons, and levers, or subsets thereof. The embodiment illustrated as left hand controller 201 features a first paddle 221 and a second paddle 222. Similarly, right hand controller 202 includes a first paddle 223 and a second paddle 224. In some embodiments, first paddle 221 is engaged with second paddle 222 via one or more gears (not shown) such that a user's depression of first paddle 221 causes reciprocating movement within second paddle 222, and vice versa. Further discussion regarding gear engagement between the first and second paddles is provided below with respect to FIGS. 11A and 11B. In another embodiment, the first paddle 221 and the second paddle 222 may be configured to operate independently. In an embodiment employing reciprocating movement of the first and second paddles, the hand controller may employ only one signal indicative of the deflection of the first and second paddles. In an embodiment in which the first and second paddles operate independently, the hand controller may employ a first signal indicative of the deflection of the first paddle and a second signal indicative of the deflection of the second paddle.

[0081] In some embodiments, the first paddle 221, 223 and the second paddle 222, 224 may be contoured to receive a user's thumb and / or finger. In some embodiments, the first paddle 221, 223 extends from or beyond the outer side of the respective contoured housing 210, 211, and the second paddle 222, 224 extends from or beyond the inner side 212b, 213b of the respective contoured housing. Deflection or depression of the first paddle 221, 223 and the second paddle 222, 224 for each hand controller 210, 211 is configured to generate a signal that the surgical robotic system uses as an input to control a tool or instrument tip at the distal end of a robotic arm of the surgical robotic system (e.g., to open / close a grasper / jaw opening at the instrument tip). For example, depressing the first and second paddles may change the angle of the jaws of the graspers at the distal end of the respective robotic arms. In some embodiments, the end effectors, tools, or instruments are used to separate tissue, drive needle drivers, grasp items (e.g., mesh, sutures, needles), or pick up items within a body cavity when they are dropped, and deliver energy (e.g., to cut or coagulate) via an electrosurgical unit (ESU).

[0082] In some embodiments, the housing of the hand controller may be contoured. For example, in FIGS. 6A and 6B , the contoured housing 210, 211 includes a rounded shape. In some embodiments, the housing may be shaped to have a contour that matches the contour of at least a portion of the thumb of a user's hand. In some embodiments, the contoured housing 210, 211, the first paddle 221, 223, and the second paddle 222, 224 may each be shaped to comfortably and ergonomically receive a respective user's hand. In some embodiments, the hand controller housing, the hand controller lever, the hand controller button, and / or one or more touch input devices may have shapes and / or positions on the hand controller to accommodate different palm sizes and finger lengths.

[0083] The left hand controller 201 also includes a first button 231, a second button 232, and a third button 233. Similarly, the right hand controller 202 also includes a first button 234, a second button 235, and a third button 236. As described herein, each button may provide one or more inputs that may be mapped to various different functions of a surgical robotic device to control a surgical robotic system, including a camera assembly and a robotic arm assembly. In one embodiment, inputs received via the first button 231 of the left hand controller 201 and the first button 234 of the right hand controller 202 may control a clutch feature. For example, activating the first button 231, 234 activates the clutch, allowing an operator to move the respective left hand controller 201 or right hand controller 202 without causing any movement of the respective robotic assemblies (e.g., the first robotic arm, the second robotic arm, and the camera assembly) of the surgical robotic system. When the clutch is activated for a hand controller, movement of the respective right or left hand controller is not translated into movement of the robot assembly. In some embodiments, an operator activating a hand controller input (e.g., tapping or pressing a button) activates the clutch, and when the operator activates it again (e.g., tapping or pressing a button), the clutch is turned off or exits clutch mode. In some embodiments, an operator activating a hand controller input (e.g., tapping or pressing a button and holding the button) activates the clutch, and the clutch remains active as long as the input is active, and exits the clutch when the operator is no longer activating the hand controller input (e.g., releasing the button).Activating the clutch or entering the clutch mode of the hand controllers allows the operator to reposition each hand controller (e.g., repositioning the left controller 201 within the range of motion of the left hand controller 201 and / or repositioning the right hand controller 202 within the range of motion of the right hand controller 202) without causing movement of the robot assembly itself.

[0084] A second button 232 on the left hand controller 201 can provide input to control the swivel function of the surgical robotic device. An operator who activates (e.g., presses and holds) the second button 232 on the left hand controller 201 can activate a swivel function or swivel mode that reorients the chest of the robotic assembly and centers the camera on the midpoint between the instrument tips. The swivel function can be activated with a short tap or held down to continuously track the instrument tips as they move, according to some embodiments.

[0085] The second button 235 on the right hand controller 202 may provide inputs for entering and exiting a menu mode, in which a menu is displayed on the surgical robot system's graphical user interface. The operator may activate the menu mode by pressing the second button 235 once and deactivate the menu function by pressing the second button 235 again. When the menu mode is activated, the operator may be able to select choices within the menu by navigating the menu using the left and / or right hand controllers. For example, the first touch input device 242 on the right hand controller 202 may, in some embodiments, be used to navigate the menu and select menu items. During the menu mode, robot movement may be suspended in response to movement of the left or right hand controller 201 or 202.

[0086] In some embodiments, a third button 233 on the left hand controller and a third button 236 on the right hand controller can provide inputs to enable or disable an instrument control mode of the surgical robotic system. Movement of at least one of the one or more hand controllers while in instrument mode causes a corresponding movement in a corresponding robotic arm of the robotic assembly. Instrument control mode is described in more detail below.

[0087] The left-hand controller 201 further includes a touch input device 241. Similarly, the right-hand controller 202 further includes a touch input device 242. In one embodiment, the touch input devices 241, 242 may be scroll wheels, as shown in Figures 6A and 6B. Other touch input devices that may be employed include, but are not limited to, rocker buttons, joysticks, pointing sticks, touchpads, trackballs, trackpoint nubs, etc.

[0088] The touch input devices 241, 242 may be capable of receiving input through several different forms of activation by an operator. For example, if the touch input devices 241, 242 are scroll wheels, the operator may be able to press or click the first touch input device 241, 242, scroll the first touch input device 241, 242 backward or forward, or both.

[0089] In some embodiments, scrolling forward on the first touch input device 241 of the left hand controller 241 may activate a zoom-in function provided by the camera assembly of the surgical robotic system to enlarge the view displayed to the operator, and scrolling backward on the first touch input device 241 may provide a zoom-out function provided by the camera assembly of the surgical robotic system to reduce the view displayed to the operator, or vice versa. In embodiments, the zoom function may be mechanical or digital. In some embodiments, the zoom function may be partially mechanical and partially digital (e.g., mechanical zoom over one zoom range and mechanical zoom plus digital zoom over another zoom range).

[0090] In some embodiments, clicking or pressing the first touch input device 241 may enable a scan mode of the surgical robotic system. When in scan mode, movement of at least one of the left hand controller 201 or the right hand controller 202 causes a corresponding change in the orientation of the camera assembly of the robotic assembly without changing the position or orientation of any of the robotic arms of the surgical robotic system. In another embodiment, pressing and holding the first touch input device 241 may activate the scan mode, and releasing the first touch input device 241 may exit the scan mode of the surgical robotic system. In some embodiments, exiting scan mode returns the camera to the orientation it had when entering scan mode. In some embodiments, a feature may be provided for locking the orientation (e.g., to change the "horizon" line) when exiting scan mode.

[0091] In some embodiments, when in menu mode and the left elbow menu item is selected, the first touch input device 241 of the left hand controller 201 can be used to select the direction and degree of left elbow bias. As used herein, elbow bias refers to the degree to which the virtual elbow of the robotic arm is above or below the neutral or default position.

[0092] In some embodiments, when in menu mode, an operator may be able to select options within a menu by navigating the menu using the left-hand controller and / or the right-hand controller. For example, when in menu mode, the touch input device 242 (e.g., a scroll wheel) on the right-hand controller provides a series of inputs for traversing the displayed menu and selecting items within the displayed menu. For example, by scrolling forward on the touch input device 242, the operator may move up the menu, and by scrolling backward on the touch input device 242, the user may move down the menu, or vice versa. In one embodiment, by clicking the first touch input device 242, the operator may make a selection within the menu.

[0093] In some embodiments, the touch input device 242 of the right hand controller 202 may be used to control right elbow actuation when the right elbow actuation menu item is selected.

[0094] In some embodiments, the functions of the various buttons and touch input devices described above with respect to the left-hand controller may instead be assigned to the right-hand controller, and vice versa.

[0095] 6A also shows a schematic diagram 203 of a first foot pedal 251 and a second foot pedal 252 for receiving operator input. As shown in FIG. 6A, in some embodiments, the first foot pedal 251 enables a camera control mode of the surgical robotic system, and the second foot pedal 252 enables a drive control mode of the surgical robotic system.

[0096] In some embodiments, when the camera control mode is activated, for example using the foot pedal 251, movement of the left hand controller 201 and / or right hand controller 202 by the operator can provide input that is interpreted by the system to control the movement and orientation of the camera assembly of the surgical robotic system while keeping the position of the instrument tip of the robotic arm of the robotic assembly constant.

[0097] In some embodiments, when the cruise-control mode is activated, for example, using the foot pedal 252, the left hand controller 201 and the right hand controller 202 can be used to move the robotic arm assemblies of the surgical robotic system in a manner such that the distal tips of the robotic arms direct or guide the movement of the chest of the robotic assembly through an internal body cavity. In the cruise-control mode, the position and orientation of the camera assembly are automatically adjusted to keep the view of the camera assembly pointed at the tips (e.g., a point between the tip of the distal end of the first robotic arm and the tip of the distal end of the second robotic arm). This can be described as the camera assembly being pinned to the chest of the robotic assembly and automatically following the tips. Further details regarding the cruise-control mode are provided below.

[0098] In different embodiments, different functions may be mapped to different buttons and different touch input devices on the hand controller. In different embodiments, different or other functions may correspond to buttons and touch input devices on the hand controller that have the same physical structure.

[0099] For example, in some embodiments, compared to FIGS. 6A and 6B, buttons 231 and 234 may provide input to enable or disable the instrument control mode of the surgical robotic system. In some embodiments, hand controllers 201 and 202 may not include an enable / disable button. Instead, the operator can position their head near the display so that it is within range of the surgeon alert sensor, and the operator can squeeze paddles 221-224 to enable / disable. In some embodiments, compared to FIGS. 6A and 6B, a second button 232 may enable a camera control mode. When the camera control mode is activated, the operator's movement of left hand controller 201 and / or right hand controller 202 may provide input that is interpreted by the system to control the movement and orientation of the camera assembly of the surgical robotic system while keeping the position of the instrument tip of the robotic arm of the robotic assembly constant. In some embodiments, a first pedal 251 may enable a swivel mode. In some embodiments, the functionality of the swivel mode may be integrated into the camera mode, and the system need not have a swivel mode. In some embodiments, compared to FIGS. 6A and 6B, the first pedal 252 may enable a translational mode.

[0100] 7A and 7B illustrate another embodiment according to the present disclosure, featuring a left hand controller 301 and a right hand controller 302. The left hand controller 301 includes a contoured housing 310, and the right hand controller 302 includes a contoured housing 311. Each contoured housing 310, 311 includes a top surface 312a, 313a, an inner side surface 312b, 313b adjacent the top surface, an outer side surface (not visible in these views) facing away from the inner side surfaces 312b, 313b, and a bottom surface (not visible in these views) facing away from the top surfaces 312a, 313a (see outer side and outer side views of a similar hand controller 701 in FIGS. 15-15D).

[0101] In some embodiments, the left hand controller 301 also includes a mounting assembly 315 that includes a proximal hand controller attachment 317, and the right hand controller 302 also includes a mounting assembly 316 that includes a proximal hand controller attachment 318. For each hand controller 301, 302, a respective mounting assembly 315, 316 can be used to attach the hand controller 301, 302 to a surgeon console of a surgical robotic system.

[0102] Each of the left hand controller 301 and the right hand controller 302 also includes a first button 331, 334, a second button 332, 335, and a third button 333, 336, respectively. In some embodiments, each of the first hand controller 301 and the second hand controller 301 also includes a touch input device 341, 342, respectively, which may be a scroll wheel. In each hand controller 301, 302, the first button 321, 334, the second button 332, 335, and the touch input device 341, 342 are located on or at the top surface 312a, 313a of the housing. In some embodiments, the first button 321, 334, the second button 332, 335, and the touch input device 341, 342 are located on or at a portion of the top surface 312a, 313a that protrudes from the top surface. In each hand controller 301, 302, a third button 333, 336 is located on or at the inner side 312b, 313b of the housing. For each hand controller 301, 302, a lever (not visible in this view) extends from the respective outer side (not visible in this view). In some embodiments, a different mechanism may be used for grip input on the hand controller. For example, in some embodiments, the hand controller may include at least one "pistol trigger" type button that can be pulled back to close and released to open, instead of or in addition to a lever.

[0103] The contoured housings 310, 311 can be configured to comfortably and ergonomically fit with the corresponding hand of an operator. An operator can engage each hand controller 301, 302 by placing the thumb of each hand on the inner side 312b, 313b, positioning the index or middle finger of each hand on or above the protruding portion of the top surface 313a, 313a on which the first button 321, 334, the second button 332, 335, and the touch input devices 341, 342 are located, and by positioning at least the middle or ring finger of each hand on or above the outer side or lever. Examples of functions that can be controlled by the first button 331, 334, the second button 332, 335, the third button 333, 336, and the touch input devices 341, 342 will be further described below with reference to Figures 8A-8G.

[0104] While various exemplary embodiments described herein assign specific functions to specific buttons and specific touch input devices, those skilled in the art will understand, in light of this disclosure, that which functions are assigned to which buttons and touch input devices may vary in different embodiments. Furthermore, those skilled in the art will understand, in light of this disclosure, that additional functions not explicitly described herein may be assigned to some buttons and some touch input devices in some embodiments. Those skilled in the art will also understand, in light of this disclosure, that some embodiments may not assign some of the functions described herein, or any of the functions described herein, to any of the buttons and / or touch input devices of a hand controller. In some embodiments, one or more functions may be assigned to a foot pedal of a surgical robotic system that includes one or more hand controllers described herein.

[0105] By way of example, a set of functions that may be controlled by the left hand controller 301 and the right hand controller 302 are described herein for some embodiments of the present technology. In some embodiments, a more frequently used function, such as a clutch, may be assigned to a button or touch input device that is easier for the operator to reach, which may be the first button 331 for some operators. Which button or touch input device is easier for the operator to reach may depend on the particular design of the hand controller and may depend on the length of the operator's digits (e.g., fingers or thumb) and the size of the operator's palm. In some embodiments, activating (e.g., pressing or pressing and holding) the first button 331 of the left hand controller 301 may generate an input signal for control of the clutch function of the left hand controller 301. Pressing the second button 332 of the left hand controller may generate an input signal for control of a reset feature of the surgical robotic system, which may also be described herein as a realignment feature of the surgical robotic system. In some embodiments, a reset or realign feature changes the robotic arm and chest to a neutral or default configuration while maintaining the location of the robotic arm's end effector or instrument tip. Pressing the third button 333 on the left hand controller 301 may, in some embodiments, generate a signal to enable / disable an instrument control mode of the surgical robotic system. Scrolling the touch input device 341 may generate a signal used to control a zoom function of the surgical robotic system. Pressing or pressing and holding the touch input device 341 may generate a signal used to activate a scan function or scan mode. Scrolling the touch input device 341 may generate a signal used to adjust the left elbow bias when the elbow bias function is activated using a menu.

[0106] On the right hand controller 302, pressing or pressing and holding the first button 334 may generate a signal used to control the right hand controller's clutch function of the surgical robotic system. Pressing the second button 335 may generate a signal used to turn on and off menus of the surgical robotic system. Pressing or pressing and holding the third button 336 on the right hand controller may generate a signal to enable or disable an instrument control mode of the surgical robotic system. Scrolling the right hand controller's touch input device 342 may generate a signal used to traverse a menu or highlight a portion of a menu when a menu is displayed or when a menu mode is active. Pressing the touch input device 342 may generate a signal used to select a highlighted portion on a menu or a menu or feature when a menu is displayed. Scrolling the touch input device 342 may generate a signal used to adjust the right elbow bias when the elbow bias function is activated using a menu.

[0107] Some functions and operations of the hand controllers are described in relation to Figures 8A-8G. The left hand controller 301 and the right hand controller 302 illustrated in Figures 8A-8G may be substantially similar to the right hand controller 301 and the right hand controller 302 illustrated in Figures 7A-7B. Where similar features are described, the same reference numbers are used.

[0108] FIG. 8A shows third buttons 333, 336 on the left hand controller 301, right hand controller 302, and left and right hand controllers 301, 302, respectively. In one embodiment, the third buttons 333, 336 can each generate a signal to enable / disable an instrument control mode of the surgical robotic system. When the instrument control mode is disabled, input from the surgeon console via the hand controllers will not result in any movement or other activity by the robotic assembly. Disabling does not affect the camera view, but prevents control of any robot movement (including camera, scan, instrument, and move / track modes). Unlike a clutch, for which an "enable" input must occur to re-enable, a clutch only needs to be released to terminate the clutch function according to some embodiments.

[0109] In some embodiments, the instrument control mode is the default control mode of the surgical robotic system. In the instrument control mode, movement of each hand controller corresponds to movement of the end effector of the respective robotic arm, but the chest or base of the robotic arm is not translated in space.

[0110] In the instrument control mode, the hand controllers are also used to operate certain tool functions of the robotic arms of the surgical robotic assembly, such as opening and closing grippers or operating electronic functions of the surgical robotic system. The instrument control mode is also used to operate instruments attached to the arms of the surgical robotic device, for example, to manipulate tissue to cauterize, suture, or perform other functions. In the instrument control mode, the instrument tip of the left robotic arm is controlled at least in part by the left hand controller 301, and the instrument tip of the right robotic arm is controlled at least in part by the right hand controller 302. In some embodiments, the pedal assembly may include an additional pedal for controlling certain aspects of the instrument tip, such as electrocautery functionality. In some embodiments, in the instrument control mode, the orientation and position of the camera assembly remain fixed while the instrument tip is manipulated. In one embodiment, the left hand controller 301 may be used to control the movement of an instrument on the left arm assembly, and the right hand controller 302 may be used to control an instrument on the right arm assembly. For example, the instruments may include graspers, scissors, cautery, needles, and other end effectors. In one embodiment, the instrument control mode may be activated by clicking an enable button (e.g., third button 333 or third button 336). In one embodiment, the instrument control mode may be the default mode, such that the system returns to the instrument control mode when the user exits any other control mode while the system is enabled. The left hand controller 301 and the right hand controller 302 may move independently. For example, one or both of the left hand controller 301 and the right hand controller 302 may be moved right, left, forward, backward, up, down, or any combination of these directions.Additionally, the left hand controller 301 and the right hand controller 302 may rotate in a clockwise or counterclockwise direction about the longitudinal axis 345, 346 of each hand controller, may tilt / pitch forward or backward about the pitch axis of each hand controller, may rotate or tilt about a yaw axis perpendicular to the longitudinal axis 345, 346 and the pitch axis 347, 348 of each hand, or any combination of the above.

[0111] FIG. 8B illustrates an embodiment in which the left hand controller 301, and in particular the touch input device 341, is used to select scan mode. In an embodiment, a user may activate scan mode by pressing and holding the touch input device 341. In scan mode, the user can change the orientation (e.g., lateral yaw and up / down pitch) of the surgical robotic system's camera assembly while the arm and instrument remain fixed so that the user can, for example, examine the abdomen, check the position of an elbow, or locate a surgical tool. In scan mode, movement of the left hand controller 301 can be used to control the yaw, pitch, and roll of the camera assembly. In some embodiments, the operator exits scan mode by releasing the touch input device 341. In some embodiments, the surgical robotic system will exit scan mode after the user clicks the touch input device 341 again. In some embodiments, when scan mode is terminated, the orientation of the camera assembly returns to the orientation it had when scan mode was activated.

[0112] FIG. 8C illustrates a hand controller clutch function enabled by selecting either the first button 331 or the second button 335 of the hand controller, according to some embodiments. In one embodiment, the operator may activate (e.g., press or hold) the first button 331, 334 of the respective hand controller to activate the clutch for that hand controller. In some embodiments, when the clutch is activated, the robotic surgical system remains in its current control mode but ignores input from movement of the respective hand controller for which the clutch is enabled, allowing the user to reposition the respective left hand controller 301 or right hand controller 302, or within the respective controller's range of motion. When the clutch is no longer activated or enabled, the surgical robotic system still remains in the same current control mode, but movement of the hand controller causes normal movement of the robot assembly and camera assembly corresponding to the current control mode. In some embodiments, the hand controller or foot pedal may receive input to a universal clutch that collectively functions as a clutch for both hand controllers.

[0113] FIG. 8D illustrates activating the camera control mode using a foot pedal on the operator console to select the camera control mode. After the camera control mode is selected, the left hand controller 301 can be used to change the orientation (e.g., pitch and yaw) of the robotic assembly and the position of the camera assembly of the surgical robotic system. In camera control mode, the user can control the yaw, pitch, and roll of the camera assembly, translating or displacing the camera assembly up, down, forward, backward, right, left, or any combination of the above. In camera control mode, the chest of the robotic arm assembly and at least a portion of each robotic arm can move, while the instrument tip of each robotic arm remains stationary. Each of these movements of the camera assembly can be controlled by a corresponding movement of the left hand controller 301, as indicated by the movement indicator. In some embodiments, pressing the camera control mode pedal again causes the surgical robotic system to exit camera control mode.

[0114] In some embodiments, after exiting camera control mode, the framing selected in camera control mode is maintained, meaning the camera assembly remains in its current position when returning to instrument control mode, and the view from the camera assembly remains the same when returning to instrument control mode. Scan mode and camera control mode may differ at least in that the camera control mode view is maintained in subsequent instrument control modes, whereas in scan mode or scan function, the current view from the camera assembly is not maintained after the system returns to instrument control mode. Furthermore, scan view does not allow for translation of the camera assembly or movement of the robotic arm assembly chest. Scan mode allows the operator to view other parts of the internal cavity without moving the robotic arm before using the scan function to return to a preferred or previous camera view. Camera control mode allows the operator to reframe the camera view or change the viewpoint relative to the instrument tip before returning to instrument control mode or changing to travel mode. In some embodiments, when in camera control mode, the robotic surgical system may not recognize input from the right hand controller 302 when the left hand controller is being used for camera control, or the robotic surgical system may not recognize input from the left hand controller 301 when the right hand controller 302 is being used for camera control. In other embodiments, input from both hand controllers may be used when in camera control mode.

[0115] FIG. 8E illustrates the hand controller in ride control mode. In some embodiments, ride control mode can be selected by pressing the foot pedal. In some embodiments, the foot pedal can be pressed once to enable ride control mode and again to disable ride control mode. Alternatively, in some embodiments, ride control mode can be enabled while the pedal is pressed and disabled when the pedal is released. Some embodiments can provide both functions by recognizing when a user briefly presses and releases the pedal and enabling the function, or by recognizing when a user presses and releases the pedal for a long time and enabling the function until the pedal is released.

[0116] When the navigation-control mode is enabled, movement of the left hand controller 301 and the right hand controller 302 is translated into corresponding movement of the end effectors of the robotic arms of the robotic assembly. Similar to the instrument control mode, the end effectors of the robotic arms and tools can be manipulated in the navigation-control mode. In the navigation-control mode, the camera assembly's view tracks the midpoint between the instrument tip of the right robotic arm and the instrument tip of the left robotic arm. Unlike the instrument control mode, movement of the hand controllers can also cause displacement of the robotic assembly's chest. The surgical robot system establishes a cone of movement for the camera assembly's displayed view, and if the instrument tip or end effector exceeds the cone of movement, the robotic system's chest automatically moves. Further description of this aspect of the navigation mode is provided below in the navigation mode section. In some embodiments, the navigation-control mode can be used to navigate the robotic assembly to another location within a patient's internal body cavity or to maintain visualization while a surgical task is performed. As described above, in the navigation mode, the camera assembly automatically tracks the midpoint between the instrument tips during instrument tip movement. Thus, drive mode can be useful for task navigation and visualization because it allows the user to maintain a consistent view of the instrument tips. This visualization can be beneficial in procedures such as suturing around a mesh or creating a flap, for example. As indicated by the movement indicators 355, in drive control mode, both the left hand controller 301 and the right hand controller 302 can move independently or together and can move up, down, right, left, forward, backward, pitch, yaw, and roll. In some embodiments, movement of the hand controllers is translated into corresponding movement of the respective instrument tips of the robotic assembly, which automatically follows the instrument tips without requiring additional input from the user.

[0117] FIG. 8F illustrates the second button 335 and touch input device 342 of the right-hand controller 302 used to control menu functions of the surgical robotic system. In one embodiment, a user can select the second button 335 to initiate a menu function, including displaying a menu on the surgical robotic system's graphical user interface, scroll the touch input device 342 to move up or down the menu or highlight a menu item, and then click the touch input device 342 to make a selection from the menu. In some embodiments, activating the menu deactivates the instrument control mode, or deactivates the instrument control mode, navigation control mode, and camera control mode all. In some embodiments, the second button 335 can be pressed or clicked once to activate the menu and again to deactivate the menu. In some other embodiments, the second button 335 can be pressed or clicked once to activate a menu option and released to deactivate the menu function.

[0118] 8G illustrates the touch input device 341 on the left hand controller 301, the second button 335 on the right hand controller 302, and the touch input device 342 on the right hand controller 302, which are used to control the elbow bias function of the surgical robotic system. In one embodiment, an operator activates a menu function to display a menu, selects an elbow bias mode or function, and then scrolls up or down on the touch input device 341 and presses the touch input device 341 to select a bias for the left elbow of the surgical robotic system. Similarly, in one embodiment, a user may scroll up or down on the touch input device 342 and press the touch input device 342 to select a bias for the right elbow of the surgical robotic system. In some embodiments, the elbow bias options include one or more levels above the nominal or default elbow bias and one or more levels below the normal or default elbow bias.

[0119] 9A and 9B illustrate another embodiment of a left hand controller 401 and a right hand controller 402. Each of the hand controllers 401, 402 includes a contoured housing 410, 411, and a first button 431, 434 and a second button 432, 435 on a top surface of the housing. Each hand controller 401, 402 includes a first paddle or gripping device 421, 422 and a second paddle or gripping device 423, 424. Each hand controller 401, 402 also includes a first touch input device 441, 442 and a second touch input device 443, 444 (e.g., a scroll wheel).

[0120] In some embodiments, the first button 431 on the left hand controller 401 is used to activate or close a menu, and the first button 434 on the right hand controller 402 is used to open a right elbow actuation menu. In some embodiments, the second button 432 on the left hand controller 401 is also used to activate or close a menu, and the second button 435 on the right hand controller 402 is used to open a right elbow actuation menu. In some embodiments, activating the arm actuation button on the left hand controller or the arm actuation button on the right hand controller opens an arm actuation menu for both arms. In some embodiments, the first touch input device 441 on the first hand controller 401 is used to traverse a menu or to highlight a menu item to be selected when a menu is displayed, and to select a left elbow actuation when a left elbow menu is displayed. In some embodiments, the first touch input device 442 on the first hand controller 402 is used to traverse a menu or to highlight a menu item to be selected when a menu is displayed, and to select a right elbow actuation when a right elbow menu is displayed. In some embodiments, the second touch input device 443 of the left hand controller 401 is used to enable or disable instrument control mode when pressed or clicked, and is used to set the configuration of the robotic arm and chest of the robot assembly while keeping the instrument stationary when the second touch input device 443 is rolled forward. In some embodiments, the second touch input device 444 of the second hand controller 402 is used to enable or disable instrument control mode when pressed or clicked.

[0121] In some embodiments, different functions may be assigned to some or all of the buttons and touch input devices on the left hand controller 401 and the right hand controller 402. For example, in some embodiments, the second button 432 on the left hand controller 401 is used to open and close a menu. In some embodiments, the second button 435 on the right hand controller 402 is used to open and close a menu. In some embodiments, the second button 432 is used to set the configuration of the robotic arm and chest of the robot assembly while keeping the instruments stationary when the second button 432 is rolled forward. In some embodiments, the first button 434 on the right hand controller is used to select the right elbow bias when the elbow bias menu is displayed, and the second button 435 on the right hand controller 402 is used to select the left elbow bias when the elbow bias menu is displayed. In some embodiments, the first touch input device 442 of the right hand controller 402 is used to traverse menus or highlight menu items to be selected when a menu is displayed, and the second touch input device 443 of the left hand controller 401 is used to activate or deactivate an instrument control mode when pressed, to activate the hand controller clutch when slid or rolled back, and to activate a scan mode when slid or rolled forward. In some embodiments, rolling the first touch control input 442 of the right hand controller 402 may be used to highlight an elbow bias level for selection. In some embodiments, pressing the second touch control input 444 of the right hand controller 402 may be used to activate or deactivate an instrument control mode, and sliding or rolling the second touch control input 444 may activate the clutch. In some embodiments, pressing a particular pedal while pressing or rolling the first touch control input 442 of the right hand controller 402 may change the corresponding function of pressing or rolling the first touch control input 442.For example, in some embodiments, pressing and holding the first touch control input 442 while holding a particular pedal may activate a camera scan mode or feature. In some embodiments, clicking the first touch control input 442 while holding a particular pedal may reset the attitude of the robot assembly. In some embodiments, rolling the first touch control input 442 forward or backward while holding a particular pedal may increase or decrease the zoom of the image from the camera assembly. In some embodiments, the foot pedals 203 may include a third pedal for a clutch function. The hand controllers 401, 402 may not include a clutch function.

[0122] 10 includes a hand controller 501 in accordance with another embodiment of the present technology. The hand controller 501 includes a contoured housing 510, a first paddle 521, a second paddle 522, and a button 531. In some embodiments, the first paddle 521 and the second paddle 522 may be connected via a gear connection (not shown) such that movement of the first paddle 521 causes reciprocating movement of the second paddle 522, and movement of the second paddle 522 causes reciprocating movement of the first paddle 521.

[0123] In some embodiments, the first paddle 521 includes a finger pad 525. In some embodiments, the finger pad 525 forms at least a portion of the contact surface of the first paddle 521. In some embodiments, a magnet 562 is located on the first paddle 521 opposite the finger pad 525 and proximate the contoured housing 510. In some embodiments, a printed circuit board (PCB) 563 is housed and located within the contoured housing 510. In embodiments, the PCB 563 may include or connect to one or more proximity and / or optical sensors. In some embodiments, the PCB 563 may be configured to determine the distance between the magnet 562 and the PCB 563 or a proximity sensor, or to generate a signal corresponding to the distance between the magnet 562 and the PCB 563 or a proximity sensor. In some embodiments, the magnet 562 may instead be located on the second paddle. In some embodiments, a magnet may be located on the first paddle and another magnet may be located on the second paddle.

[0124] In some embodiments, at least one of the one or more hand controllers includes a sensor configured to sense contact of an operator's hand with the hand controller or to sense proximity of an operator's hand to the hand controller. In some embodiments, input from the sensor is used to activate or terminate an instrument control mode of the surgical robotic system. In some embodiments, at least one sensor may be located on a lever of the hand controller. In some embodiments, at least one sensor may be located on a portion of the housing that is not the lever. In some embodiments, the hand controller may have multiple sensors for sensing activation of the hand controller. In some embodiments, the sensor is optical. In some embodiments, the sensor is a capacitance sensor. In some embodiments, another type of sensor may be used.

[0125] In some embodiments, the surgical robotic system may sense activation by the hand controller or activation by the surgical robotic system using a sensing system that does not include sensors on the hand controller (e.g., optical tracking of the operator's hand, optical tracking of the operator's head, tracking of the operator's gaze, etc.).

[0126] In some embodiments, the second paddle 522 includes a finger pad 526. In some embodiments, the finger pad 526 forms at least a portion of the contact surface of the second paddle 522. In some embodiments, the second paddle 522 includes an optical window 561 and an optical sensor positioned to receive light incident on the optical window 561 (not shown). In some embodiments, the optical sensor generates a signal indicating that something (e.g., a portion of the operator's hand) is positioned above the optical window 561 and blocking at least some of the incident light from reaching the optical window, or that nothing (e.g., not even a portion of the operator's hand) is positioned above the optical window 561 so as to block at least some of the incident light from reaching the optical window 561. Such a signal can be used to detect whether the operator's hand is positioned on the hand controller and to disable at least some functions or the hand controller or to disengage a control mode of the hand controller when the operator's hand is not positioned on the hand controller. In some embodiments, the first hand controller may also or alternatively include a sensor or the like. In some embodiments, a capacitance sensor is used to enable activation. Because the surgeon may need to remove his or her hand when discarding sutures, in some embodiments a capacitance sensor may not be used to disengage or disable the function. In some embodiments, fingertip capacitance sensing may be used in conjunction with another form of "supervisory" detection (e.g., forehead eye tracking or optical sensors) to enable activation, but then if the user does not have their eyes / head in the correct position, only the eye / forehead tracking will trigger activation.

[0127] 11A and 11B schematically illustrate a hand controller 601 with different mechanisms for measuring paddle deflection, according to some embodiments. FIG. 11A illustrates the hand controller 601 with a first paddle 625A and a second paddle 626B in a first position / orientation 602, and FIG. 11B illustrates the hand controller 601 with the first paddle 625A and the second paddle 626B in a second depressed position / configuration 603. The first paddle 625A is connected to the housing or body 610 of the hand controller 601 via a first gear 680A at the proximal end of the first paddle and via a shaft that engages with the first gear 680A. The second paddle 625B is connected to the housing or body 610 of the hand controller 601 via a second gear 680B at the proximal end of the second paddle and via a shaft that engages with the second gear 680B. Additionally, the first gear 680A and the second gear 680B engage with one another such that movement of the first paddle 625A causes reciprocating movement of the second paddle 625B, and vice versa. The hand controller 601 also features a linear sensor 663. The first paddle 625A is connected to the linear sensor 663 via a first linkage 670A, and the second paddle 625B is connected to the linear sensor 663 via a second linkage 670B. The linear sensor 663 is configured to determine where the first linkage 670A and the second linkage 670B engage with the linear sensor 663, thereby enabling the linear sensor 663 to identify the configuration of the first paddle 625A and the second paddle 625B (e.g., to determine how open or closed the first paddle 625A and the second paddle 625B are relative to one another). In various embodiments, linear sensor 670B may be a PCB having a linear sensor or another sensor configured to determine location, such as an optical sensor or a proximity sensor.

[0128] 12A and 12B schematically illustrate a hand controller 601' similar to the hand controller 601 of Figures 11A and 11B including first and second paddles 625A and 625B in an open configuration (Figure 12A) and a more closed configuration (Figure 12B). In the hand controller 601', the linear sensor 663' is positioned further toward the distal end of the hand controller 601' such that a first linkage 670A' connecting the linear sensor 663' forms an obtuse angle with the first paddle 625, and a second linkage 670B' connecting the linear sensor 663' forms an obtuse angle with the second paddle 625B.

[0129] 13A and 13B schematically illustrate a hand controller 601″ similar to the hand controller 601 of FIGS. 11A and 11B including first and second paddles 625A and 625B in an open configuration (FIG. 13A) and a more closed configuration (FIG. 13B). In the hand controller 601″, a rotation sensor 664 is engaged with the first gear 680A and measures the angular position of the first paddle 625A relative to the configuration of the first and second paddles 625A and 625B (e.g., how open or closed the first and second paddles 625A and 625B are relative to each other). In some embodiments, the rotation sensor may be engaged with the second gear 680B instead of the first gear 680A.

[0130] 14A and 14B schematically illustrate a hand controller 601''' similar to the hand controller 601'' of FIGS. 13A and 13B including first and second paddles 625A' and 625B' in an open configuration (FIG. 13A) and a more closed configuration (FIG. 13B). In the hand controller 601''', the first and second paddles 625A' and 625B' extend proximally, and each of the first and second paddles 625A' and 625B' has a respective gear 680'A and 680B' associated with a pivot point distal to the respective paddle's finger pad 626A, 626B. Additionally, each finger pad 626A, 626B has a rotational or pivotal connection with the rest of its associated paddle 625A', 625B', respectively.

[0131] 15A-15D illustrate various views of another embodiment of a hand controller 701 of the present disclosure. The hand controller 701 includes a housing 710 having a top surface 711a, an inner side 711b, an outer side 711c, and a bottom surface (not visible in these views). The hand controller 701 includes a thumb pad 712 on the inner side 711b, a paddle 721 having finger cups 722, and a finger pad 719 disposed on or extending from the outer side 711c. The hand controller 701 also includes a first button 730, a second button 731, and a touch input device 740 disposed on the top surface 711a. The hand controller 701 also includes a third button 731 disposed on the inward-facing surface 711b. The hand controller 701 is a right-hand controller. In use, the operator's thumb is typically positioned on or around the thumb pad 712, the user's index finger is positioned near the first button 730, the second button 731, or the touch input device 740, the user's middle finger is positioned on or above the finger cup 722 of the paddle 721, and the user's remaining fingers are positioned on or near the finger pad 719.

[0132] Figures 16A and 16B illustrate another embodiment of the left hand controller 801 and the right hand controller 802. Figure 16C illustrates another embodiment of the foot pedals.

[0133] Each of the hand controllers 801, 802 includes a housing 810, 811, a first paddle 821, 823 including finger pads 825, 827, a second paddle 822, 824 including finger pads 826, 828, a first button 831, 834, a second button 832, 835, a first touch input device 841, 842, and a second touch input device 843, 844 (e.g., a trackwheel, a mini-joystick, a touchpad, or any other suitable multi-function device).

[0134] In some embodiments, a first button 831 on the left hand controller 801 is used to activate or close a menu, and a first button 834 on the right hand controller 802 is used to open a right elbow actuation menu. In some embodiments, a second button 832 on the left hand controller 801 is also used to activate or close a menu, and a second button 835 on the right hand controller 802 is used to open a right elbow actuation menu. In some embodiments, activating the arm actuation button on the left hand controller or the arm actuation button on the right hand controller opens an arm actuation menu for both arms. In some embodiments, a first touch input device 841 on the first hand controller 801 is used to traverse a menu or to highlight a menu item to be selected when a menu is displayed, and to select a left elbow actuation when a left elbow menu is displayed. In some embodiments, a first touch input device 842 on the first hand controller 802 is used to traverse a menu or to highlight a menu item to be selected when a menu is displayed, and to select a right elbow actuation when a right elbow menu is displayed. In some embodiments, the second touch input device 843 of the left hand controller 801 is used to enable or disable instrument control mode when pressed or clicked, and is used to set the configuration of the robotic arm and chest of the robot assembly while keeping the instrument stationary when the second touch input device 843 is rolled forward. In some embodiments, the second touch input device 844 of the second hand controller 802 is used to enable or disable instrument control mode when pressed or clicked.

[0135] In some embodiments, different functions may be assigned to some or all of the buttons and touch input devices on the left hand controller 801 and the right hand controller 802. For example, in some embodiments, the second button 832 on the left hand controller 801 is used to open and close a menu. In some embodiments, the second button 835 on the right hand controller 802 is used to open and close a menu. In some embodiments, the second button 832 is used to set the configuration of the robotic arm and chest of the robot assembly while keeping the instruments stationary when the second button 832 is rolled forward. In some embodiments, the first button 834 on the right hand controller is used to select the right elbow bias when the elbow bias menu is displayed, and the second button 835 on the right hand controller 802 is used to select the left elbow bias when the elbow bias menu is displayed. In some embodiments, the first touch input device 842 of the right hand controller 802 is used to traverse menus or highlight menu items to be selected when a menu is displayed, and the second touch input device 843 of the left hand controller 801 is used to activate or deactivate an instrument control mode when pressed, to activate the hand controller clutch when slid or rolled back, and to activate a scan mode when slid or rolled forward. In some embodiments, rolling the first touch control input 842 of the right hand controller 802 may be used to highlight an elbow bias level for selection. In some embodiments, pressing the second touch control input 844 of the right hand controller 802 may be used to activate or deactivate an instrument control mode, and sliding or rolling the second touch control input 844 may activate the clutch. In some embodiments, pressing a particular pedal while pressing or rolling the first touch control input 842 of the right hand controller 802 may change the corresponding function of pressing or rolling the first touch control input 842.For example, in some embodiments, pressing and holding the first touch control input 842 while holding a particular pedal may activate a camera scan mode or feature. In some embodiments, clicking the first touch control input 842 while holding a particular pedal may reset the attitude of the robot assembly. In some embodiments, rolling the first touch control input 842 forward or backward while holding a particular pedal may increase or decrease the zoom of the image from the camera assembly. In some embodiments, the first foot pedal 851 may be used to activate a rotation function of the drive mode to rotate the robot arm assembly and camera assembly. The second foot pedal 852 may be used to activate a translation function of the drive mode to translate the robot arm assembly and camera assembly. The third foot pedal 853 may be used to activate a clutch function. In some embodiments, the first, second, and third foot pedals 851-853 can be used in conjunction with other embodiments of the hand controller (e.g., the hand controllers 301 and 302 illustrated in FIGS. 7A and 7B, the hand controllers 401 and 402 illustrated in FIGS. 9A and 9B, or other hand controllers taught herein). If the foot pedals include a clutch function, the hand controllers 801 and 802 do not need to include a clutch function.

[0136] FIG. 17 schematically illustrates a graphical user interface 900 including a camera view portion 910 displaying a view from a camera assembly, and a first information portion 920 and a second information portion 930, each identifying the current control mode of the surgical robotic system.

[0137] 18A and 18B illustrate another embodiment according to the present disclosure, featuring a left hand controller 1001 and a right hand controller 1002. The left hand controller 1001 includes a contoured housing 1010, and the right hand controller 1002 includes a contoured housing 1011. Each contoured housing 1010, 1011 includes a top surface 1012a, 1013a, an inner side surface 1012b, 1013b adjacent the top surface, an outer side surface (not visible in these views) facing away from the inner side surfaces 1012b, 1013b, and a bottom surface (not visible in these views) facing away from the top surfaces 1012a, 1013a (see outer side and outer side views of a similar hand controller 701 in FIGS. 15-15D).

[0138] Each hand controller 1001, 1002 includes a mounting assembly 1015, 1016, respectively. The mounting assemblies 1015, 1016 can be used to directly or indirectly mount each of the respective hand controllers 1001, 1002 to a surgeon's console of a surgical robotic system. The mounting assembly 1015 includes an opening 1017, and the mounting assembly 1016 defines an opening 1018. The openings 1017, 1018 may be countersunk openings configured to receive screws or bolts for connecting the respective hand controllers 1001, 1002 to the surgeon's console. The mounting assembly 1015 includes a button 1004, and the mounting assembly 1016 includes a button 1005. The buttons 1004, 1005 provide an input for toggling between insertion and removal of one or more of the robotic arm assemblies 42A, 42B and the camera assembly 44. For example, button 1004 may be used to insert or remove first robotic arm 42A, and button 1005 may be used to insert or remove second robotic arm 42B. In some embodiments, buttons 1004, 1005 do not actually control the insertion or removal of camera assembly 44, but allow the operator to enter an insertion or removal mode. The actual process of camera assembly 44 may be controlled by other user elements.

[0139] The left hand controller 1001 and the right hand controller 1002 each also include a first button 1031, 1034, a second button 1032, 1035, and a touch input device 1041, 1042 (e.g., a joystick or scroll wheel), respectively. In each hand controller 1001, 1002, the first button 1021, 1034, the second button 1032, 1035, and the touch input device 1041, 1042 are located on or at the top surface 1012a, 1013a of the housing 1010, 1011. In some embodiments, the first button 1021, 1034, the second button 1032, 1035, and the touch input device 1041, 1042 are located on or at a portion of the top surface 1012a, 1013a that protrudes from the top surface. For each hand controller 1001, 1002, a lever (not visible in this view) extends from its respective outer side (not visible in this view). In some embodiments, a different mechanism may be used for grip input on the hand controller. For example, in some embodiments, the hand controller may include at least one "pistol trigger" type button that can be pulled back to close and released to open, instead of or in addition to a lever.

[0140] The left hand controller 1001 includes a first paddle 1021 and a second paddle 1022. Similarly, the right hand controller 1002 includes a first paddle 1023 and a second paddle 1024. In some embodiments, the first paddles 1021, 1023 are engaged with the second paddles 1022, 1024 of each hand controller 1001, 1002 via one or more gears (not shown) such that a user's depression of the first paddle 1021, 1023 causes reciprocating movement in the second paddles 1022, 1024, and vice versa. Further discussion regarding gear engagement between the first and second paddles is provided above with respect to FIGS. 11A and 11B. In another embodiment, the first paddles 1021, 1023 and the second paddles 1022, 1024 of each hand controller may be configured to operate independently. In embodiments employing reciprocating movement of first and second paddles, the hand controllers 1001, 1002 may employ some form of signal or other indicator indicative of the deflection of the first paddles 1021, 1023 and the second paddles 1022, 1024. In embodiments in which the first and second paddles operate independently, the hand controllers 1001, 1002 may employ a first signal or other indicator indicative of the deflection of the first paddles 1021, 1023 and a second signal or other indicator indicative of the deflection of the second paddles 1022, 1024.

[0141] In some embodiments, the first paddle 1021, 1023 and the second paddle 1022, 1024 may be contoured to receive a user's thumb and / or finger. In some embodiments, the first paddle 1021, 1023 extends from or beyond an outer side of the respective contoured housing 1010, 1011, and the second paddle 1022, 1024 extends from or beyond an inner side 1012b, 1013b of the respective contoured housing. Deflection or depression of the first paddle 1021, 1023 and second paddle 1022, 1024 for each hand controller 1010, 1011 is configured to toggle a signal that the surgical robotic system uses as an input to control a tool or instrument tip at the distal end of the robotic arm of the surgical robotic system (e.g., to open / close a grasper / jaw opening at the instrument tip). For example, depressing the first paddle 1021, 1023 and the second paddle 1022, 1024 may change the angle of the jaws of the graspers at the distal end of the respective robotic arms. In some embodiments, the end effectors, tools, or instruments are used to retract tissue, drive needle drivers, grasp items (e.g., mesh, sutures, needles), or pick up such items within a body cavity when they are dropped, and deliver energy (e.g., to cut or coagulate) via an electrosurgical unit (ESU).

[0142] In some embodiments, each of the first paddles 1021, 1023 and second paddles 1022, 1024 may have a loop for receiving a user's thumb and / or finger, as further described with respect to Figures 19A and 19B. In some embodiments, parameters (e.g., length, angle, finger ergonomics, etc.) of each of the first paddles 1021, 1023 and second paddles 1022, 1024 may be adjusted.

[0143] The contoured housings 1010, 1011 can be configured to comfortably and ergonomically fit with the corresponding hands of an operator. An operator can engage each hand controller 1001, 1002 by placing the thumb of each hand on the second paddle 1022, 1024, positioning the index or middle finger of each hand on or above the protruding portion of the top surface 1013 a, 1013 a on which the first button 1021, 1034, second button 1032, 1035, and touch input device 1041, 1042 are located, and by positioning at least the middle or ring finger of each hand on or above the first paddle 1021, 1024.

[0144] While various exemplary embodiments described herein assign specific functions to specific buttons and specific touch input devices, those skilled in the art will understand, in light of this disclosure, that which functions are assigned to which buttons and touch input devices may vary in different embodiments. Furthermore, those skilled in the art will understand, in light of this disclosure, that additional functions not explicitly described herein may be assigned to some buttons and some touch input devices in some embodiments. In some embodiments, one or more functions may be assigned to a foot pedal of a surgical robotic system that includes one or more hand controllers described herein.

[0145] By way of example, a set of functions that may be controlled by the left hand controller 1001 and the right hand controller 1002 for some embodiments of the present technology will now be described.

[0146] On the left hand controller 1001, pressing or pressing and holding the first button 1004 may trigger a signal used to enable insertion or removal of the left robotic arm assembly and / or camera assembly of the surgical robotic system. Pressing or pressing and holding the first button 1031 may trigger a signal used to control a clutch function of the left hand controller of the surgical robotic system. Pressing or pressing and holding the second button 1032 may trigger a signal to enable or disable a camera control mode of the surgical robotic system. Scrolling the touch input device 1041 forward may activate a zoom-in function to enlarge the view provided by the camera assembly of the surgical robotic system and displayed to the operator, and scrolling backward with the first touch input device 1041 may provide a zoom-out function to reduce the view provided by the camera assembly of the surgical robotic device and displayed to the operator, or vice versa. Scrolling the touch input device 1041 may trigger a signal used to select left elbow bias when the elbow bias function is activated using a menu (as illustrated in FIG. 20 ).

[0147] On the left hand controller 1002, pressing or pressing and holding the first button 1005 may trigger a signal used to enable insertion or removal of the right robotic arm assembly and / or camera assembly of the surgical robotic system. Pressing or pressing and holding the first button 1034 may trigger a signal used to control a clutch function of the right hand controller of the surgical robotic system. Clicking or pressing the second button 1035 may enable a scan mode of the surgical robotic system. When in scan mode, movement of at least one of the left hand controller 1001 or the right hand controller 1002 causes a corresponding change in the orientation of the camera assembly of the robotic assembly without changing the position or orientation of any of the robotic arms of the surgical robotic system. In another embodiment, pressing and holding the second button 1035 may activate the scan mode, and releasing the second button 1035 may exit the scan mode of the surgical robotic system. In some embodiments, exiting scan mode returns the camera to the orientation it was in when entering scan mode. In some embodiments, a function may be provided for locking the orientation (e.g., for changing the "horizontal" line) when exiting scan mode. Scrolling the touch input device 1042 may trigger a signal used to traverse a menu or highlight a portion of a menu when a menu is displayed or when menu mode is active (as illustrated in FIG. 20). Pressing the touch input device 1042 may trigger a signal used to select a highlighted portion on a menu or a menu or feature when a menu is displayed. Scrolling the touch input device 1042 may generate a signal used to adjust the right elbow bias when the elbow bias function is activated using a menu. Scrolling forward on the touch input device 1042 may move up the menu, and scrolling backward on the touch input device 1042 may move down the menu, or vice versa. Clicking the first touch input device 1042 may make a selection within a menu.

[0148] 19A and 19B illustrate another embodiment according to the present disclosure, featuring a left hand controller 1001′ and a right hand controller 1002′. Compared to the hand controllers 1001, 1002 of FIGS. 18A and 18B, some buttons on the hand controllers 1001′, 1002′ have the same button type but different functions. For example, the second button 1035′ on the right hand controller 1002′ may trigger a signal used to turn a menu on or off. Compared to the hand controllers 1001, 1002 of FIGS. 18A and 18B, some buttons on the hand controllers 1001′, 1002′ may have different button types and / or different functions. For example, the touch input device 1041′ on the left hand controller 1001′ may have a three-way switch button type. Operating or holding the touch input device 1041′ at the center may trigger a signal used to enable or disable a scan mode of the surgical robotic system. Switching the touch input device 1041' forward may activate a zoom-in function provided by the surgical robotic system's camera assembly to enlarge the view displayed to the operator, and switching the first touch input device 1041' backward may provide a zoom-out function provided by the surgical robotic system's camera assembly to reduce the view displayed to the operator, or vice versa. Switching the touch input device 1035' upward may trigger a signal used to traverse a menu when a menu is displayed or when a menu mode is active. The touch input device 1042' of the right hand controller 1002' may have a three-way switch button type. Switching the touch input device 1042' may trigger a signal used to traverse a menu or highlight a portion of a menu when a menu is displayed or when a menu mode is active by pressing the touch input device 1035'.Switching forward on the touch input device 1042' may move the menu up, and switching backward on the touch input device 1042' may move the menu down, or vice versa. Clicking the first touch input device 1042' may trigger a signal used to select a highlighted portion on a menu or a menu or feature when the menu is displayed. In some embodiments, switching the touch input device 1042' may trigger a signal used to select a right elbow bias when the elbow bias function is activated using a menu. Compared to the hand controllers 1001, 1002 of FIGS. 18A and 18B , the hand controllers 1001', 1002' may have first paddles 1021', 1023' and second paddles 1022', 1024' coupled to finger loops 1061, 1062, 1063, 1064, respectively. Each finger loop may be a Velcro type. In some embodiments (not shown), each finger loop may be a hook type. Deflection or depression of the first paddle 1021′, 1023′ and the second paddle 1022′, 1024′ is configured to toggle a signal to control a tool or instrument tip (e.g., open / close a grasper / jaw opening at the instrument tip) at the distal end of a robotic arm of a surgical robotic system. For example, depressing the first paddle 1021′, 1023′ and the second paddle 1022′, 1024′ can change the angle of the grasper jaws at the distal end of the respective robotic arm. In some embodiments, the end effector, tool, or instrument is used to separate tissue, drive a needle driver, grasp items (e.g., mesh, suture, needle), or pick up such items within a body cavity when the item is dropped, and deliver energy (e.g., to cut or coagulate) via an electrosurgical unit (ESU). 18A and 18B, the first buttons 1031′, 1034′ may have a slider button type. Sliding the first buttons 1031′, 1034′ may trigger a signal used to control the clutch function of the corresponding hand controller of the surgical robot system.

[0149] 20 schematically illustrates a graphical user interface 1100 including a camera view portion 1110 displaying a view from a camera assembly and a menu 1120. A hand controller may select items listed in the menu 1120, such as by controlling a touch input device.

[0150] 21 is a feature map table illustrating functions accessed by hand controllers, foot pedals, and menus as taught herein in some embodiments. Zoom in / out control, clutch control, viewpoint mode, instrument mode, scan mode, and insert / eject control may be controlled by one or more hand controllers. Blink control, elbow bias control, motion scale factor control, tissue plane selection control, and brightness and focus control may be controlled by one or more hand controllers via menus (illustrated in FIG. 20). Tracking mode and high force control may be controlled by one or more foot pedals via menus.

[0151] In some embodiments, the hand controller (e.g., as illustrated in FIGS. 6-10, 15, 16, 18, and 19) may not include an enable / disable function as taught herein. Instead, the operator can position their head near the display so that their head is within the surgeon alert sensor range, and the operator can squeeze the paddles to enable the instrument control mode as taught herein (e.g., as illustrated in FIGS. 6-10, 15, 16, 18, and 19). The operator can withdraw their head from the surgeon alert sensor range to disable the instrument control mode. In some embodiments, the swivel mode may be eliminated in the surgical robotic system. For example, the swivel mode functionality may be integrated into the camera mode. The camera mode may have three degrees of freedom for control. The camera mode may be enabled by pressing the second button 1032 on the hand controller 1001′ and then manipulating the hand controllers 1001′, 1002′ with the three degrees of freedom. In some embodiments, the direction of hand controller movement may be opposite to the direction of chest movement. In some embodiments, the paddles 1021-1024, 1021'-1024, and / or finger loops 1061-1064 of the hand controllers 1001, 1002, 1001', 1002' may be automatically adjusted to always align with the instrument tip and / or end effector. In some embodiments, the scan mode may have two degrees of freedom for control without a rolling degree of freedom.

[0152] While various exemplary embodiments described herein assign specific functions to specific buttons and specific touch input devices, those skilled in the art will understand, in light of this disclosure, that which functions are assigned to which buttons and touch input devices may vary in different embodiments. Furthermore, those skilled in the art will understand, in light of this disclosure, that additional functions not explicitly described herein may be assigned to some buttons and some touch input devices in some embodiments. Those skilled in the art will also understand, in light of this disclosure, that some embodiments may not assign some of the functions described herein, or any of the functions described herein, to any of the buttons and / or touch input devices of a hand controller. In some embodiments, one or more functions may be assigned to a foot pedal of a surgical robotic system that includes one or more hand controllers described herein.

[0153] Method for controlling a robotic assembly of a surgical robotic system - Patent Application 20070122997 Embodiments of the present disclosure provide a method for controlling a robotic assembly of a surgical robotic system using a hand controller (e.g., the hand controllers illustrated in Figures 6-10, 15, 16, 18, and 19).

[0154] FIG. 22 is a flowchart illustrating steps 2100 for controlling a robot assembly using a hand controller, performed by a surgical robotic system, according to some embodiments. In step 2110, the system 10 receives a first control mode selection input. For example, while the robot assembly 20 is positioned within an internal cavity of a subject, the system 10 may receive the first control mode selection input and, in response to the first control mode selection input, change the current control mode of the surgical robotic system 10 to the first control mode. The surgical robotic system may include multiple control modes, including one or more of a scan mode, a view mode, a drive mode, a turn mode, and a translation mode, where the first control mode is the scan mode, the view mode, the drive mode, the turn mode, or the translation mode. In some embodiments, the system 10 need not include all control modes. For example, the turn mode may be eliminated from the surgical robotic system 10. Each control mode is described below.

[0155] In step 2120, system 10 changes the position and orientation of at least one of the camera assembly, one or more robotic arm assemblies, or one or more instrument tips. For example, system 10 may take a particular action in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller while the current control mode is a first control mode. If the first control mode is a scan mode, system 10 may hold the robotic arm assembly stationary while rotating the camera assembly a corresponding first rotation relative to a displayed camera view, that is, a view of the camera assembly displayed on an image display of the surgical robot system. When the first control mode is a driving mode, the system 10 may move the corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on the position of the midpoint between the instrument tips of the robot arm of the robot assembly, which is the average tip position, or while translating the robot assembly to translate a chest point of the virtual chest and rotating the robot assembly to rotate the virtual chest, or both, to maintain the distance between the center of the virtual chest and the average tip position within an acceptable distance range and maintain the angular deviation between the line from the chest point to the average tip position and the normal to the virtual chest within an acceptable angular deviation range. When the first control mode is view mode, the system 10 may move the camera assembly and rotate the virtual chest of the robot assembly by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while maintaining the position and orientation of each instrument to maintain the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range.When the first control mode is a swivel mode, the system 10 may rotate the camera assembly while moving the corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view to center the camera assembly view on the average instrument tip position, and cause a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to maintain the camera assembly view centered on the average instrument tip position and maintain the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range. If the first control mode is a translation mode, the system 10 may rotate the camera assembly, or rotate the camera assembly and translate the virtual chest while moving the corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view to center the camera assembly view on the average tip position and maintain the distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0156] In some embodiments, the current control mode may be an instrument control mode in which movement of the right hand controller or the left hand controller causes corresponding movement of an instrument tip at the distal end of the corresponding robotic arm relative to the displayed camera view without moving or changing the orientation of the virtual chest of the robotic assembly. In some embodiments, the first control mode selection input may be received via a foot pedal of the surgical robotic system. In some embodiments, the first control mode selection input may be received via a first hand controller or a second hand controller. In some embodiments, the first control mode selection input may be received via a button or a touch input device on the right hand controller or the left hand controller. In some embodiments, the first control mode may be a view mode. In some embodiments, the first control mode may be a scan mode. In some embodiments, the first control mode may be a drive mode. In some embodiments, the first control mode may be a traverse mode. In some embodiments, the first control mode may be a swivel mode.

[0157] In step 2130, the system 10 receives a second control mode selection input. For example, by controlling the hand controller as described in connection with FIGS. 6-10, 15, 16, 18, and 19, the operator may select a second control mode from multiple control modes of the surgical robotic system. The multiple control modes may include two or more of a scanning mode, a view mode, a traveling mode, a translation mode, and a rotation mode. In some embodiments, the system 10 need not include all control modes. For example, the rotation mode may be eliminated from the surgical robotic system 10. Each control mode is described below. The system 10 may receive the second control mode selection input via the hand controller.

[0158] In step 2140, the system 10 may change the current control mode of the surgical robotic system to a second control mode different from the first control mode in response to the second control mode selection input.

[0159] In step 2150, the system 10 changes the position and orientation of at least one of the camera assembly, the one or more robotic arm assemblies, or the one or more instrument tips in response to a second movement including a second translation and a second rotation of the right hand controller or the left hand controller while the current control mode is the first control mode. For example, if the second control mode is a scan mode, the system 10 may hold the robotic arm assembly stationary while rotating the camera assembly a corresponding second rotation relative to the displayed camera view. When the second control mode is a running mode, the system 10 may move the corresponding instrument tip by a corresponding second translation, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on the average tip position, or while translating the robot assembly to translate a chest point of the virtual chest and rotating the robot assembly to rotate the virtual chest, or both, to maintain the distance between the center of the virtual chest and the average tip position within an acceptable distance range, and to maintain the angular deviation between the line from the chest point to the average tip position and the normal to the virtual chest within an acceptable angular deviation range. When the second control mode is view mode, the system 10 may move the camera assembly and rotate the virtual chest of the robot assembly by a corresponding second movement, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while maintaining the position and orientation of each instrument to maintain the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range.If the second control mode is a swivel mode, the system 10 may rotate the camera assemblies while moving the corresponding instrument tips by corresponding second movements, including corresponding second rotations, relative to the displayed camera view to center the camera assembly view on the average instrument tip position, and cause a change in the orientation of the camera assemblies, or a change in the orientation of the camera assemblies and a change in the orientation of the virtual chest, to keep the camera assembly view centered on the average instrument tip position and maintain an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range. If the second control mode is a translation mode, the system 10 may rotate the camera assemblies, or rotate the camera assemblies and translate the virtual chest while moving the corresponding instrument tips by corresponding second movements, including corresponding second translations and corresponding second rotations, relative to the displayed camera view to center the camera assembly view on the average tip position and maintain an acceptable distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range. In some embodiments, the system 10 need not include all control modes. For example, the swivel mode may be removed from the surgical robotic system 10.

[0160] Control modes and features for the control of surgical robotic systems Specific control modes and features for control of a surgical robotic system are described in more detail below. Some embodiments may include all of the modes and features described herein. Some embodiments may include only a portion of the modes and features described herein. It may be understood that embodiments may optionally provide specific modes based on procedures intended or preferred to be performed on a surgical robotic system to provide improved functionality for a particular purpose, but it should also be understood that some embodiments in accordance with the present technology may not provide any particular mode or feature, or set of modes or features.

[0161] Some embodiments described herein provide systems and methods that employ multiple different control modes for controlling a robotic assembly of a surgical robotic system when the robotic assembly is positioned within an internal body cavity of a subject.

[0162] Each control mode may use sensed movements of one or more hand controllers, and also input from one or more foot pedals, to control the robotic arm assembly and / or camera assembly. The control mode may be changed from a current control mode to a different selected control mode based on operator input (e.g., provided via one or more hand controllers and / or pedals of a surgical robotic system). In different control modes, the same movement of a hand controller may result in different movements of the robotic assembly.

[0163] Some embodiments employ multiple control modes, including an instrument control mode, which may also be referred to herein as an "instrument mode," one or more of a view control mode, which may also be referred to herein as a "view mode," a "camera control mode," a "camera mode," a "framing control mode," a "framing mode," or a "point of view mode," a scan mode, which may also be referred to herein as a "scanning mode" or "survey mode," and a drive control mode, which may also be referred to herein as a "drive mode" or an "auto-track mode." The swivel mode, drive mode, and translate mode may all be referred to herein as "track modes."

[0164] Some embodiments employ a graphical user interface that identifies the current control mode of the surgical robotic system. Some embodiments employ a menu feature where a menu is displayed on the graphical user interface, and input from one or more of the hand controllers can be used to traverse and select menu options.

[0165] Some embodiments employ additional features for controlling the robotic assembly, for example, some embodiments allow for individual control of elbow bias or elbow lift for the right and left robotic arms.

[0166] Graphical user interface for operator displays In some embodiments, a graphical user interface (GUI) of the surgical robotic system includes a display for an operator that includes a camera view portion and at least one information portion that identifies a current control mode of the surgical robotic system. An exemplary GUI is described with respect to FIG. 17. In some embodiments, the identification of the current control mode of the surgical robotic system in the at least one information portion may include a word, a color, a pattern, a symbol, or any combination of the above.

[0167] Instrument Control Mode As described above, some embodiments employ or provide an instrument control mode, which may be described herein as an “instrument mode.” In instrument mode, the surgical robotic system identifies the movement (e.g., translation and / or rotation) of each hand controller and moves (e.g., translates and / or rotates) the instrument tip on the distal end of the corresponding robotic arm in the corresponding material. In instrument control mode, the surgical robotic system may move the instrument tip in a manner directly proportional to the movement of the corresponding hand controller. This may be described as movement including translation and / or rotation of the instrument tip of the robotic arm that is directly controlled by the movement of the respective hand controller. For example, translating a hand controller by a certain amount in a certain direction moves the corresponding instrument tip of the corresponding robotic arm in the corresponding direction (i.e., the same direction relative to the view from the camera assembly displayed to the operator) by a corresponding scaled-down amount (e.g., if the scaling is based on the scale of the view from the camera assembly displayed to the operator). As another example, rotating the hand controller by an angle about an axis will cause the corresponding instrument tip of the corresponding robotic arm to rotate by the same angle about the corresponding axis (e.g., the corresponding axis is the same axis relative to the orientation of the view from the camera assembly displayed to the operator). In the instrument control mode, operator controls can be used to actuate the instrument (e.g., via the gripper controls on the hand controller, via the foot pedal controls) as well as to move or change the orientation of the instrument tip.

[0168] In instrument mode, movement of the hand controller does not change the position, does not change the orientation of the camera assembly (e.g., the orientation and position of the camera assembly may remain fixed), and does not change the position or orientation of the virtual chest. In other words, instrument mode does not reposition or reorient the camera or chest. Instrument control mode is useful for manipulating instrument tips within a working area of ​​an internal body cavity that is accessible without moving the virtual chest of the robotic assembly.

[0169] In some embodiments, the operator may enable or disable instrument control mode via either or both hand controllers. In some embodiments, instrument mode is enabled and disabled using input controls from the hand controller (e.g., by pressing a button such as button 233 in FIG. 6A or button 333 in FIG. 7A, or by interacting with a touch input device such as touch input device 443 in FIG. 9A). When instrument mode is disabled, any movement of the hand controller does not cause any corresponding movement of the associated instrument tip. In some embodiments, when the surgical robotic system is in the disengaged state, an information portion of the GUI of the user display indicates that the current state is disengaged. In some embodiments, engaging the clutch causes an information panel of the graphical user interface to identify that the clutch is engaged (e.g., via text, color, or any other graphical indicator).

[0170] In some embodiments, the tool control mode is the default control mode that the surgical robotic system enters when another control mode, such as the view control mode, or the navigation control mode, is terminated.

[0171] Further explanation of control modes As described herein, some embodiments provide one or more of a scan mode, a camera / view mode, a drive mode, a swivel mode, or a travel mode in addition to the tool control mode, further descriptions of which are provided below.

[0172] Scan Mode As described above, some embodiments employ or provide a scan mode, which may also be described herein as a “scan control mode” or “scanning mode.” In scan mode, rotation of one of the hand controllers causes a corresponding rotation of the camera assembly (e.g., yaw rotation, pitch rotation, roll rotation, or a combination of the above) to change the view provided by the camera assembly. In scan mode, movement of any of the hand controllers does not cause movement of the robotic arm or the virtual chest of the robotic assembly. Scan mode can be used to quickly survey internal body cavities, check the elbow position of the robotic arm, and / or locate surgical tools.

[0173] In some embodiments, the scan mode is initiated and activated using input from one or both of the hand controllers, or using input from one or both of the hand controllers in combination with input from a foot pedal (e.g., via input touch device 241 in FIG. 6A or via input touch device 341 in FIG. 7A), which may be described as activating and deactivating the scan mode. In some embodiments, when the scan mode ends, the orientation and position of the camera assembly return to the orientation and position the camera assembly had when the scan mode was initiated.

[0174] In some embodiments, when the surgical system is in scan mode, in response to a first movement including a first rotation of the right hand controller or the left hand controller, the robotic arm assembly is held stationary while rotating at a corresponding first rotation relative to a view of the camera assembly displayed on the image display, which is the displayed camera view.

[0175] View Control Mode / Camera Control Mode As described above, some embodiments include a view control mode, which may also be referred to herein as a “view mode,” “camera control mode,” “camera mode,” “framing control mode,” “framing mode,” or “point of view mode.” In the view control mode or camera mode, movement (e.g., translation and / or rotation) of one of the hand controllers causes a corresponding movement (e.g., translation and / or rotation) of the camera assembly. Movements of the camera assembly may include, but are not limited to, forward / backward translation, vertical translation, lateral translation, yaw, pitch, roll, or any combination of the above. In the view control mode / camera mode, the instrument tip of the robot arm remains stationary, but other portions of the robot arm may move to achieve the corresponding movement of the camera assembly. For example, the virtual chest of the robot assembly may need to translate and / or change its orientation to achieve the movement of the camera assembly. By keeping the instrument tip in a fixed position and orientation, the view control mode allows the operator to frame a specific view, such as a view of a portion of the internal cavity where the procedure is being performed, without moving the instrument tip or any tissue in contact with it.

[0176] For example, in some embodiments, when the surgical robotic system is in camera control / view mode, in response to a first movement, including a first translation and a first rotation, of the right hand controller or the left hand controller, or both simultaneously, the position and orientation of each instrument is maintained while the camera assembly is moved by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, and the virtual chest of the robot assembly is rotated to maintain the angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.

[0177] In some embodiments, the camera control mode is entered and exited in response to operator input (e.g., operator input via a foot pedal, which may be a dedicated camera control foot pedal or a button on a hand controller). In some embodiments, when the camera control mode is exited, the framing or view is maintained, meaning that the position and orientation of the camera control assembly is maintained.

[0178] In some embodiments, the camera control mode may include swivel mode functionality, as described with respect to the swivel mode section. For example, in addition to the functionality described above, the camera control may further enable the operator to change the camera orientation or the camera assembly orientation and the virtual chest orientation to control the position and orientation of the instrument tip by corresponding movements of the hand controllers while maintaining the center of the camera assembly's view at the average instrument tip position. For example, when in camera control mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, the corresponding instrument tip moves by a corresponding first movement including a corresponding first rotation relative to the displayed camera view, and the camera assembly rotates to center the camera assembly's view at the average instrument tip position, causing a change in camera assembly orientation, or a change in camera assembly orientation and a change in virtual chest orientation, to maintain the camera assembly's view centered at the average instrument tip position and to maintain the angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range.

[0179] Driving mode As described above, some embodiments include a drive-control mode, which may also be referred to herein as a "drive mode." When the drive-control mode is activated, movement of the left and right hand controllers translates into corresponding movement of the end effectors or instrument tips of the robotic assembly. Similar to the instrument control mode, instruments and tools may be operated in the drive-control mode. Unlike the instrument control mode, in the drive-control mode, the camera assembly and chest track the midpoint between the instrument tip of the right robotic arm and the instrument tip of the left robotic arm. Unlike the instrument control mode, in the drive mode, movement of the hand controllers may also cause a displacement and / or change in the orientation of the chest of the robotic assembly, allowing the robotic assembly to "drive" or "follow" the instrument tip. This may be described as the instrument tip directing or leading the movement through the internal body cavity. For example, in some embodiments, the surgical robotic system establishes a cone of movement (e.g., an acceptable range for the distance of the instrument tip from the virtual chest of the robot assembly position, and an acceptable range of deviation of the line connecting the center of the chest to the instrument tip from the normal to the chest), and if the instrument tip or end effector would exceed the cone of movement, the chest and arms of the robotic system automatically move to keep the instrument tip within the cone of movement (see FIG. 30). The cone of movement may not have a fixed length or a fixed angular range, but the length and angular range may change during use based on some other parameter of the surgical robotic system or based on currently selected features and options of the surgical robotic system (e.g., based on the zoom of the displayed camera view).

[0180] For example, in some embodiments, when in travel mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, or both, the corresponding instrument tip moves by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the displayed camera view while the camera assembly rotates to center the camera assembly's view on a location of the midpoint between the instrument tips of the robotic arms of the robot assembly, which is the average tip position, or while the robot assembly translates to translate a chest point of the virtual chest and rotates the robot assembly to rotate the virtual chest, or both, to maintain the distance between the center of the virtual chest and the average tip position within an acceptable distance range, and to maintain the angular deviation between the line from the chest point to the average tip position and the normal to the virtual chest within an acceptable angular deviation range. In some embodiments, there is a threshold that defines an acceptable translational or rotational offset that the instrument tip can have from the virtual center / chest, below which virtual center / chest movement does not occur. Once that threshold is exceeded, the system translates or rotates the virtual chest to properly re-center the instrument tip.

[0181] In some embodiments, the navigation control mode can be used to navigate the robotic assembly to different locations within a patient's internal body cavity or to maintain visualization while a surgical task is being performed. As described above, in navigation mode, the camera assembly automatically tracks the midpoint between instrument tips during their movement. Thus, navigation mode can be useful for task navigation and visualization because it allows the user to maintain a consistent view of the instrument tips. This visualization can be beneficial, for example, in procedures such as suturing around a mesh or creating a flap.

[0182] Swivel Mode As described above, some embodiments include a swivel control mode, which may also be referred to herein as a "swivel mode." The swivel mode allows the operator to change the camera orientation or the camera assembly orientation and the virtual chest orientation to further control the position and orientation of the instrument tip by corresponding movements of the hand controllers while maintaining the center of the camera assembly's view at the average instrument tip position. For example, when in the swivel mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, the corresponding instrument tip moves by a corresponding first movement including a corresponding first rotation relative to the displayed camera view, causing the camera assembly to rotate to center the camera assembly's view at the average instrument tip position, causing a change in camera assembly orientation, or a change in camera assembly orientation and a change in virtual chest orientation, to maintain the camera assembly's view centered at the average instrument tip position and to maintain the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range.

[0183] In some embodiments, activation of a foot pedal activates the swivel mode. In some embodiments, activation of an input function on a hand controller activates the swivel mode.

[0184] In some embodiments, the functionality of the swivel mode may be integrated into the camera mode, and the system need not have a swivel mode, as the camera mode may perform the functions described in the "View Control Mode / Camera Control Mode" section and with respect to the functionality of the swivel mode, as described herein.

[0185] Translational Mode As described above, some embodiments include a translate control mode, which may also be referred to herein as a "translate mode," "translation mode," or "translation control mode." As described above, some embodiments include a drive control mode. The translate control mode allows the operator to control the position and orientation of the instrument tip by corresponding movement of the hand controller, by changing the orientation of the camera assembly and / or translating the chest of the robotic assembly to maintain the center of the camera assembly's view at the average instrument tip position and keep the average distance between the center of the chest and the average tip position within an acceptable range. For example, while in translation mode, in response to a first movement, including a first translation and a first rotation, of the right hand controller or the left hand controller, the corresponding instrument tip moves by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while the camera assembly rotates, or while the camera assembly rotates and the virtual chest translates, to center the camera assembly view on the average tip position and maintain the distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

[0186] In some embodiments, activation of a foot pedal activates the run mode. In some embodiments, activation of an input function on the hand controller activates the run mode.

[0187] Methods for implementing instrument, travel, and view modes with respect to camera center, camera direction, and virtual chest position and orientation Some embodiments implement methods and modes for controlling the movement of at least a portion of a robotic assembly (e.g., a robotic arm) to achieve a desired instrument tip location and orientation. In some embodiments, the methods for control abstract away from direct and separate control of the robotic assembly's virtual chest and camera assembly robotic chest, allowing the operator to focus entirely on the robotic assembly's instrument tips and the operator's view of those instrument tips. Definitions of terms used to describe the control methods are included below.

[0188] As used herein, a "chest point" refers to the center point of the virtual chest of the robotic assembly. As used herein, a "chest normal" refers to the direction of the virtual chest, which is also normal to the chest plane. As used herein, a "chest plane" is a plane passing through the chest point whose normal is the chest direction. As used herein, a "trocar center" is a point in space where a trocar is inserted into a patient and is a support that extends through the trocar to support the robotic assembly within the internal body cavity rotation. As used herein, a "trocar direction" is a direction from the trocar center to the chest point. As used herein, a "trocar plane" is a plant that passes through the trocar center whose normal is the trocar direction. As used herein, a "mean tip position" is a point in space that is midway between the instrument tips of the robotic arm. As used herein, a "camera origin" is a neutral position of the camera assembly relative to its support. In some embodiments, the camera assembly may be able to be pushed forward and pulled back relative to the camera assembly support. As used herein, the "shoulder" of an arm is the location of the start of the arm or the most proximal joint of the arm. As used herein, the "camera / arm root" is the point where the camera or arm support intersects with the trocar plane. As used herein, the "root triangle" is the triangle formed by the camera root and the roots of the two arms. It is always coplanar with the trocar plane. Figure 23 schematically illustrates the relationship between these centers, planes, and directions.

[0189] Those skilled in the art will appreciate that the term "paddle" may also be referred to as a control lever.

[0190] In some embodiments, some control modes and methods, such as the travel mode, focus on maintaining an "ideal robot pose," which may also be called a "desired robot pose," particularly related to the position / orientation of the robot arm shoulder, camera assembly, and trocar.

[0191] Controlling the position and orientation of the instrument tip in instrument control mode The operator controls the position and orientation of the instrument tip using two hand controllers mapped to each instrument tip. Using operator input (e.g., on the hand controllers or via foot pedals), the user can enable and disable control of each instrument tip, which may be described as enabling and disabling instrument control mode. When instrument control mode is enabled, moving and rotating the hand controllers moves and rotates the instrument tip in a similar manner. When instrument control mode is disabled or disengaged, moving the hand controllers does not cause corresponding movement of the instrument tip. Instrument control mode may be used to achieve a "ratchet" type movement for larger movements by turning off instrument control (e.g., disengaging instrument control mode or engaging the clutch), repositioning one or both hand controllers, re-enabling instrument control (e.g., enabling instrument control mode or disengaging the clutch), and continuing to move.

[0192] Translational movements of the instrument tip are relative to the camera's current orientation. For example, enabling instrument control mode and moving the hand controller forward moves the corresponding instrument tip forward in the frame of reference of the camera view (i.e., the direction the camera is currently facing). Moving the hand controller up moves the instrument tip upward in the camera's frame of reference (i.e., the hand moves up relative to the frame of view of the camera image displayed to the operator, which is the "base" of the camera frustum). In some embodiments, the scale of these movements is different in different modes.

[0193] Rotational movements of the hand controller are mapped to rotations relative to the current orientation of the instrument tip: for example, rolling the hand controller to the left causes the instrument tip to rotate a similar amount about its forward axis, pitching the hand controller up causes the instrument tip to pitch a similar amount about its left axis, etc.

[0194] "Chest" and "Camera Offset" FIG. 23 schematically illustrates a trocar plane 2200, a chest plane 2202, and various points and directions associated with the virtual chest 140 and camera assembly 44 of the robot assembly 20, and the trocar 50, according to some embodiments.

[0195] The chest point may be the center point 2203 of the virtual chest 140 of the robot assembly 20. The chest normal 2201 may be the direction of the virtual chest 140, which is also normal to the chest plane 2202. The chest plane 2202 may be a plane passing through the chest point 2203 whose normal is the chest direction 2201. The trocar center 2250 may be a point in space where the trocar 50 is inserted into the patient and is a support that extends through the trocar 50 to support the robot assembly 20 within the internal body cavity rotation. The trocar direction 2230 may be the direction from the trocar center 2250 to the chest point 2203. The trocar plane 2200 is a plane passing through the trocar center 2250 whose normal is the trocar direction. The mean tip position 2300 (as illustrated in FIG. 24) may be a point in space midway between the instrument tips 120A, 120B of the robotic arm assemblies 42A, 42B (as illustrated in FIG. 5). The camera origin 2204 may be a neutral position of the camera assembly 44 relative to its support. In some embodiments, the camera assembly 44 may be able to be pushed forward and pulled back relative to its support. The shoulder 126 of the robotic arm assembly 42 may be the start of the robotic arm assembly 42 or the location of the proximal-most joint of the robotic arm assembly 42 (as illustrated in FIG. 5). The camera root 2210 may be the point where the camera assembly support intersects with the trocar plane 2200. The arm root 2220 may be the point where the support of the robotic arm assembly 44 intersects with the trocar plane. The root triangle may be the triangle formed by the camera root 2210 and the two arm roots 2220. It is always flush with the trocar plane 2200 .

[0196] Physically, the virtual chest 140 can be a triangle formed by the two shoulders 126A, 126B and the camera origin 2210. Conceptually, it is the projection of the root triangle onto the chest plane 2202. To calculate this projection, rays 2206, 2208 are cast from the camera root 2210 of the camera assembly 44 and the arm roots 2220 of the robot arm assemblies 42A, 42B in the trocar direction 2230 and intersect with the chest plane 2202. Each drive that moves a portion of the robot assembly 20 should be inserted and outsert (e.g., retracted) to keep the shoulders 126A, 126B and the camera origin 2204 on the chest plane 2202.

[0197] During operation, camera assembly 44 always orients itself relative to chest plane 2202. That is, as chest plane 2202 rotates, camera assembly 44 may rotate in a similar manner. The operator may introduce an offset (called a camera offset) to change the angle of camera assembly 44 relative to chest plane 2202.

[0198] Control of the robot body in running mode Aspects of a method for implementing the drive mode are described below according to some embodiments. Other embodiments may have different or other implementations of the drive mode. In the drive mode, the positioning of the robot body (e.g., the virtual chest) is primarily determined by the position of the instrument tip. In some embodiments, as the user works and controls the instrument tip, the chest is repositioned to maintain the desired arm configuration, which may be the "ideal" arm configuration. In some embodiments, as the user works and controls the instrument tip, the chest is repositioned to maintain the configuration the robot assembly was in when entering the drive mode and when exiting the drive cone, which may be the desired arm configuration. This requires moving (i.e., translating and / or rotating) the virtual chest to reposition the chest point and reorient the chest as needed to maintain the desired arm configuration. In some embodiments, yaw and pitch to position the arms may also be employed to achieve the desired positioning of the robot assembly.

[0199] The method for relocating the chest point attempts to keep the average tip position always within a certain distance range, which is as follows: 1) Draw a line from the chest point to the average tip position and measure its length. 2) If the length of the line exceeds some value (e.g., 12 cm), translate the chest point toward the mean tip position. 3) If the length of the line is below some value (e.g., 6 cm), translate the chest point away from the mean tip position. 4) If the line length is between these two values, do not translate the chest point.

[0200] In some embodiments, a "soft boundary" may be implemented where the chest moves slowly when the distance is within a certain range, e.g., 10 cm to 12 cm, and moves faster when the distance is greater than that range, e.g., greater than 12 cm, or where the speed at which the chest moves is proportional to the distance.

[0201] The method for reorienting the chest attempts to keep the chest facing or nearly facing the instrument tip. It is as follows. 1) Draw a line from the chest point to the average tip position. 2) If the angle between that line and the chest normal exceeds a given value (e.g., 10 degrees), rotate the chest normal toward that line. 3) Do not rotate the chest normal if the angle is less than a given value.

[0202] In some embodiments, this method of reorienting the chest may also include soft boundary conditions as the chest normal rotates.

[0203] 24 schematically illustrates positioning and orienting the chest plane 2202 to keep the distance 2330 from the chest point 2203 to the average instrument tip position 2300 within an acceptable range 2320 and the angular deviation of the average instrument tip position 2300 from the chest normal / chest direction 2201 within an acceptable range 2310, according to some embodiments. The net result of the combined chest reorientation and chest translation is that the robotic chest remains properly positioned and oriented toward the operator's workspace. As mentioned above, the chest point 2203 and chest normal 2201 change, but the trocar 50 should be pivoted and the robotic drive should insert / outsert to keep the trocar direction 2230 and shoulder 126 / camera origin 2204 exactly on the chest plane 2202.

[0204] As the chest moves / rotates, the chest normal 2201 may become perpendicular or nearly perpendicular to the trocar direction 2230. This would result in a very distorted projection of the chest, requiring the shoulders 126 or camera origin 2204 to be very far from each other. To prevent this, the chest normal 2201 may be constrained to always be more than a specified minimum angular amount away from perpendicular to the trocar direction 2230 (e.g., more than 20 degrees).

[0205] Controlling the view in view control mode At any time, the operator may enable view control mode (e.g., using a toggle control such as a foot pedal or toggle control). Upon entering view control mode, controls for both robotic instrument tips are disabled and cannot be re-enabled until the user exits view control mode.

[0206] While in view control mode, the user can rotate a single hand controller to rotate the robot camera. Rotation of the hand controller is directly mapped to rotation of the robot camera. Thus, rotating the hand controller up about its left axis rotates the camera by a corresponding scaled amount (e.g., scaled down) about its left axis, and yawing the controller to the right rotates the camera by a similar amount about its up axis.

[0207] Moving the hand controllers has varying effects based on the direction of movement. Moving the hand controller along the forward axis pushes the robot camera forward or backward. Note that the position of the camera origin does not change because the chest normal does not change; it simply changes the distance of the robot camera from the camera origin.

[0208] Moving the hand controller along the left-right axis and vertically moves the chest point. All of these movements are relative to the reference frame of the robotic camera. Thus, moving the hand controller up moves the chest point toward the robotic camera's up axis, and moving the hand controller to the left moves the chest point toward the robotic camera's left axis.

[0209] In some embodiments, none of the different controls in the view control mode are exclusive; they are all active simultaneously, allowing for smooth, combined pushing, rotating, and chest positioning.

[0210] In some embodiments, when the operator exits view control mode, the angular offset between the camera orientation and the chest orientation is stored as the camera offset. At any time, the user can press the "reset" button, which slowly returns the camera offset to 0.

[0211] Advantages over some conventional control methods in surgical robotic systems Some conventional surgical robotic systems employ gesture control in a dedicated driving mode or driving state to navigate the robotic assembly within an internal body cavity. For example, in one conventional system, the operator grasps the target workspace and “pull” it toward the operator while in a dedicated driving control mode. The driving mode described herein offers several advantages over this “grab and pull” method. First, in the driving mode described herein, the operator operates and navigates the instrument as needed within a single control mode. In contrast, the “grab and pull” method requires the operator to activate a separate, dedicated driving mode, move the robotic assembly, and then return to the separate control mode for robotic arm manipulation. This increases the cognitive load on the operator and requires more time and more steps. In contrast, in the driving mode described herein, the workspace moves with the instrument tip as needed as the operator normally works, effectively eliminating the need for a separate, dedicated driving control mode in many cases. Second, gesture control is not discoverable (e.g., gestures need to be explicitly taught and memorized by the operator), in contrast to movement in the present cruise-control mode, which can be discovered through normal operation of the surgical robotic system. Finally, cruise-control mode does not require the operator to understand and keep track of the positions of the individual components of the robotic assembly (e.g., chest, camera, etc.), but instead requires the operator to focus on the two things that will concern them most during any typical endoscopic surgical procedure: controlling the instrument tip and the surgical camera view. This is substantially simpler than controlling individual components, potentially lowering the barrier to entry for using surgical robotic systems.

[0212] Rotational scaling for suture motion Some embodiments of a hand controller according to the present disclosure may be used to control suturing by a surgical robot. Moving the controller with a suturing motion sufficient for a robotic arm to perform suturing may require a roll motion greater than a human wrist can perform. Therefore, in embodiments, when a suturing mode is selected, rotational scaling may be provided for the suturing motion.

[0213] Some embodiments may provide a suturing mode in which rotational movement of a portion of the hand controller results in a larger angle of rotation at the instrument tip. For example, in some embodiments of the suturing mode, the surgical robotic system performs a mathematical operation to produce a larger roll output in the surgical robotic arm than roll movements made with the hand controller. In some embodiments, the mathematical operation may be as follows: α commanded = S × α β commanded =β gamma commanded =γ

[0214] where α is the roll input from the hand controller, β is the pitch input from the hand controller, γ is the yaw input from the hand controller, S is a scale factor, and α commanded is the roll output sent to the robotic surgical device, and β commanded is the pitch output delivered to the robotic surgical device, and γ commanded is the yaw power delivered to the surgical device.

[0215] The scale factor (S) can be selected to be greater than 1. For example, the scale factor can be 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, or 2.0. A scale factor of 1.4 has been tested and found to be satisfactory. When the scale factor is applied, the surgical robotic device exhibits a virtual wrist roll that is greater than that performed by the user's wrist and reflected by the hand controller.

[0216] Foot pedal array for surgical robotic systems Embodiments of the present disclosure provide a foot pedal array for receiving input from an operator to control a robotic assembly of a surgical robotic system, an operator console including such a foot pedal array, and a surgical robotic system including such a foot pedal array. Some embodiments include a pair of foot pedals and a column sensor that enables detection of an operator's foot being positioned on a column of foot pedals or on a foot pedal. In some embodiments, a column sensor and at least one row sensor are employed to detect an operator's foot being positioned on a particular foot pedal. In some embodiments, a column sensor, or a column sensor and at least one row sensor, may be used to identify that an operator's foot is positioned on a foot pedal where it obstructs the sensor's beam before the operator presses or activates the foot pedal. In some embodiments, the surgical robotic system may provide a user with information identifying a foot pedal, or a function of the foot pedal, that is in proximity to the operator's foot in response to sensor data. This information may allow the operator to confirm that a foot pedal will be pressed or activated before the user presses the foot pedal, which reduces the risk of unintended activation of a function based on pedal activation. This may reduce the risk associated with a user selecting an incorrect foot pedal, for example, as a result of not being able to see the foot pedal located on the foot pedal array of the user console. In embodiments in which some foot pedals are used to control an electrosurgical device or system, such a safety feature may be particularly advantageous.

[0217] Some systems use a foot pedal array to provide control of modes or functions of a surgical robotic system. In some embodiments, one or more foot pedals or pairs of foot pedals can select one or more modes of a surgical robotic device, one or more foot pedals or pairs of foot pedals can operate functions of the surgical robotic system, one or more foot pedals or pairs of foot pedals can control articulation and movement of a surgical robot, and one or more foot pedals or pairs of foot pedals can control visualization via the surgical robotic system. For example, in some embodiments, one or more pairs of foot pedals can control electrosurgical functions of the surgical robotic system, and a pair of foot pedals can be used to select a control mode of the surgical robotic system, including, for example, a drive mode and a camera mode, which may also be referred to herein as a view mode, in which input via at least one hand controller of the operating system (e.g., sensed movement of at least one hand controller) causes movement of one or more robotic arms of the surgical robotic system and / or a camera assembly of the surgical robotic system that differs from when the robotic system is in a standard mode (e.g., a mode in which one or more robotic arms will perform movements corresponding to the sensed movement of the at least one hand controller).

[0218] In particular, the present disclosure is not limited to drive mode and camera mode, but may include other articulation and visualization functions of the system. For example, in some embodiments, control modes may include swivel mode and translate mode. In some embodiments, such foot pedal control of changing modes of the surgical robotic system may reduce the complexity of one or more hand controllers and may reduce operator fatigue compared to controlling changes in modes of operation via hand controllers.

[0219] Further advantages of some embodiments of the present technology may include increased speed and efficiency in operator control of the system as a result of easier access to modes and controls, and improved reliability due to the safety benefits of the sensor system in embodiments that have a sensor system, which may reduce the overall time of the surgical procedure providing health benefits to the patient.

[0220] Embodiments include a foot pedal assembly for use in a robotic surgical system, an operator or surgeon console including such a foot pedal assembly, and a robotic surgical system including such a foot pedal assembly.

[0221] In one aspect, the present disclosure is directed to a foot pedal assembly for a surgical robotic system, including a foot pedal array for receiving input from an operator to control a robotic assembly of the surgical robotic system. The foot pedal array includes a foot pedal tray having a left foot portion corresponding to a portion of the foot pedal tray more easily accessible by the operator's left foot and a right foot portion corresponding to a portion of the foot pedal tray more easily accessible by the operator's right foot when the operator is sitting or standing facing the foot pedal array. The foot pedal tray also includes a first foot pedal tier and a second foot pedal tier positioned lower than and forward of the first foot pedal tier and extending toward the operator in use. The foot pedal assembly also includes a first pair of foot pedals located on a first foot pedal hierarchy in the right foot portion, the first pair including a first foot pedal and a second foot pedal adjacent to the first foot pedal, and a second pair of foot pedals located on a second foot pedal hierarchy in the right foot portion, the second pair of foot pedals including a third foot pedal and a fourth foot pedal adjacent to the third foot pedal. In some embodiments, the foot pedal assembly also includes a third pair of foot pedals located on the left foot portion, the third pair of foot pedals including a fifth foot pedal and a sixth foot pedal. The foot pedal assembly further includes a first row sensor and a second row sensor. The first row sensor includes a first emitter configured to direct a first beam onto at least a portion of the first foot pedal and the third foot pedal, and a first receiver configured to receive the first beam. The second array sensor includes a second emitter configured to direct a second beam onto at least a portion of the second foot pedal and the fourth foot pedal, and a second receiver configured to receive the second beam.

[0222] In some embodiments, the second foot pedal tier has a height relative to the floor that is less than the height relative to the floor of the first foot pedal tier when the foot pedal array is in use.

[0223] In some embodiments, the first receiver and the second receiver are disposed on or inserted into a second foot pedal tier of the foot pedal tray.

[0224] In some embodiments, the first array sensor is configured to generate a signal indicative of partial or complete blocking of the first beam received at the first receiver caused by an object disposed between the first emitter and the first receiver, generate a signal indicative of an unblocked first beam received at the first receiver, or both. In some embodiments, the second array sensor is configured to generate a signal indicative of partial or complete blocking of the second beam received at the second receiver caused by an object disposed between the second emitter and the second receiver, generate a signal indicative of an unblocked second beam received at the second receiver, or both.

[0225] In some embodiments, the foot pedal array is configured such that a foot positioned on and adjacent to or at the first foot pedal or the third foot pedal generates a signal indicative of partial or complete blockage of the first beam, and a foot positioned on and adjacent to or at the second foot pedal or the fourth foot pedal generates a signal indicative of partial or complete blockage of the second beam.

[0226] In some embodiments, the first emitter and the second emitter are positioned on a second foot pedal tier beyond the first foot pedal and the second foot pedal.

[0227] In some embodiments, the fifth foot pedal is located on a first foot pedal tier and the sixth foot pedal is located on a second foot pedal tier. In some embodiments where the fifth foot pedal is located on the first foot pedal tier and the sixth foot pedal is located on the second foot pedal tier, the foot pedal array further includes a third column sensor including a third emitter configured to direct a third beam onto at least a portion of the fifth foot pedal and the sixth foot pedal, and a third receiver configured to receive the third beam.

[0228] In some embodiments, the fifth foot pedal and the sixth foot pedal are both located on the second foot pedal tier, or both located on the first foot pedal tier. In some embodiments where the fifth foot pedal and the sixth foot pedal are both located on the same foot pedal tier, the foot pedal array further includes a third column sensor and a fourth column sensor. The third column sensor includes a third emitter configured to direct a third beam onto at least a portion of the fifth foot pedal and a third receiver configured to receive the third beam. In some embodiments, the foot pedal array also includes a fourth column sensor including a fourth emitter configured to direct a fourth beam onto at least a portion of the sixth foot pedal and a fourth receiver configured to receive the fourth beam.

[0229] In some embodiments, the foot pedal array further includes a first row sensor including a first row emitter configured to direct a first row beam along the first foot pedal layer and across at least a portion of the first and second foot pedals, or along the second foot pedal layer and across at least a portion of the third and fourth foot pedals, and a first row receiver configured to receive the first row beam. In some embodiments, the first row beam is also directed across at least a portion of the fifth foot pedal, across at least a portion of the sixth foot pedal, or both. In some embodiments, the first row beam is directed along a different foot pedal layer than that of the fifth and sixth foot pedals.

[0230] In some embodiments, the first row sensor is configured to generate a signal indicative of partial or complete blocking of the first row beam received at the first row receiver caused by an object disposed between the first row emitter and the first row receiver, generate a signal indicative of an unblocked first row beam received at the first row receiver, or both.

[0231] In some embodiments, the foot pedal array is configured such that a foot positioned on and adjacent to or at the first foot pedal or the second foot pedal generates a signal indicative of partial or complete blockage of the first row beam.

[0232] In some embodiments, the foot pedal array comprises: The foot is configured such that a foot on and adjacent to or positioned on the first foot pedal generates a signal from the first column sensor indicating partial or complete blocking of the first beam and a simultaneous signal from the first row sensor indicating partial or complete blocking of the first row beam; a foot on and adjacent to or positioned on the third foot pedal generates a signal from the first column sensor indicating partial or complete blocking of the first beam but does not generate a simultaneous signal from the first row sensor indicating partial or complete blocking of the first row beam; a foot on and adjacent to or positioned on the second foot pedal generates a signal from the second column sensor indicating partial or complete blocking of the second beam but does not generate a simultaneous signal from the first row sensor indicating partial or complete blocking of the first row beam; and a foot on and adjacent to or positioned on the fourth foot pedal generates a signal from the second column sensor indicating partial or complete blocking of the second beam but does not generate a simultaneous signal from the first row sensor indicating partial or complete blocking of the first row beam.

[0233] In some embodiments, the foot pedal array further includes a second row sensor including a second row emitter configured to direct a second row beam along the first foot pedal layer and across at least a portion of the first and second foot pedals, or along the second foot pedal layer and across at least a portion of the third and fourth foot pedals, and a second row receiver configured to receive the second row beam. In some embodiments, the second row beam is also directed across at least a portion of the fifth foot pedal, across at least a portion of the sixth foot pedal, or both.

[0234] In some embodiments, the foot pedal array further includes a first sidewall that at least partially defines an edge of a left foot portion of a foot pedal tray in the first foot pedal tier, and a second sidewall that faces the first sidewall and at least partially faces an edge of a right foot portion of a foot pedal tray in the first foot pedal tier.

[0235] In some embodiments, a first row transmitter is disposed on or within or supported on one of the first sidewall or the second sidewall, and a first row receiver is disposed on or within or supported on the other of the first sidewall or the second sidewall.

[0236] In some embodiments, the foot pedal array includes a processor configured to receive one or more signals from the first row sensor, the second row sensor, or both. In some embodiments, the first pair of foot pedals and the second pair of foot pedals are configured to generate signals to control electrosurgical functions of the surgical robotic system. In some embodiments, the third pair of foot pedals is configured to generate signals to change or control a mode of operation of the robotic assembly. In some embodiments, the signal generated by the fifth foot pedal changes or controls a drive mode of operation of the robotic assembly, and the signal generated by the sixth foot pedal changes or controls a camera motion mode of operation, which may also be referred to herein as a view mode of operation, of the robotic assembly. In some embodiments, the signal generated by the fifth foot pedal changes or controls a translation mode of operation of the robotic assembly, and the signal generated by the sixth foot pedal changes or controls a swing mode of operation of the robotic assembly.

[0237] In another aspect, the present disclosure is directed to an operator console for receiving input from an operator to control a robotic assembly of a surgical robotic system, the operator console including: a foot pedal array arranged such that each of a first foot pedal, a second foot pedal, a third foot pedal, a fourth foot pedal, a fifth foot pedal, and a sixth foot pedal is accessible to one or both feet of the operator; and a plurality of controls operable by one or both hands of the operator.

[0238] In another aspect, an operator console for receiving input from an operator to control a robotic assembly of a surgical robotic system, the operator console including any of the foot pedal assemblies described herein.

[0239] In some embodiments, a system according to the present disclosure may increase the safety of a surgical procedure by allowing a user to identify the foot pedal they are approaching with their foot before they press the foot pedal, thereby reducing the likelihood that the user will unintentionally select the wrong foot pedal when approaching the foot pedal with their foot. In some embodiments, a system according to the present disclosure may provide improved ease of use for the user.

[0240] As noted above, embodiments of the present disclosure provide a foot pedal array for receiving input from an operator to control a robotic assembly of a surgical robotic system, an operator console including such a foot pedal array, and a surgical robotic system including such a foot pedal array. Furthermore, some embodiments of the foot pedal array and operator console described herein may be employed with endoscopic surgical systems that are non-robotic or only partially robotic.

[0241] The foot pedal array, and the surgeon or operator console employing the foot pedal array, can be understood with reference to the specific embodiment shown in FIGS.

[0242] FIG. 25 illustrates an operator console 2500 including a foot pedal array 2501 according to some embodiments. FIG. 26 illustrates a detailed view of the foot pedal array 2501 of the operator console 2500. The operator console 2500 also includes a cart base 2521, two hand controllers 2512, 2514, and a cart 2520 on which a display device 2510 is provided, according to some embodiments. The display device 2510 may be, for example, a monitor or screen. In other embodiments, a user may use a headset, which may complement or replace the display device 2510. In some embodiments, the cart base 2521 includes four locking wheels that allow the cart 2520 to be moved. The foot pedal array 2501 is located below the cart base 2521 according to some embodiments. The foot pedal array 2501 may be attached directly to the cart base 2521 or may be suspended below the cart base 2521 according to some embodiments. In some embodiments, the foot pedal assembly 2501 may be integrated into or at least partially housed within the cart base. In some embodiments, the foot pedal array 2501 may be removably attached to the operator console 2500 to facilitate cleaning, repair, or replacement of the foot pedal array 2501 with a different foot pedal array, as provided herein. In some embodiments, the operator console 2500, or a portion of the operator console (e.g., the cart base 2521), may be provided separately from the foot pedal array 2501, and the foot pedal array 2501 may be removably attached to or connected to the operator console 2500. The connection of the foot pedal array 2501 to the operator console 2500 may also be adjustable to allow different configurations to be selected by a user, according to some embodiments. The foot pedal array 2501 may be positioned slightly above the ground or floor to allow movement of the operator console 2501, according to some embodiments.

[0243] The foot pedal array 2501 includes a foot pedal tray 2505 having a first foot pedal level 2506 and a second foot pedal level 2507. The second foot pedal level 2507 is positioned lower (e.g., closer to the floor) and forward relative to the first foot pedal level 2506 such that the second foot pedal level 2507 extends further toward the operator when the foot pedal array 2501 is positioned for use. The foot pedal tray 2505 may generally be described as having a left foot portion 2505L corresponding to a portion of the foot pedal tray more easily accessible by the operator's left foot when the operator is sitting or standing facing the foot pedal array 2501, than a right foot portion 2505R corresponding to a portion of the foot pedal tray more accessible by the operator's right foot. In some embodiments, the foot pedal tray 2505 includes a recess 2503 at its proximal end located near where a user will stand or sit during use. The recess 2503 may facilitate user access to the foot pedal array 2501 .

[0244] 25 and 26, the foot pedal tray 2505, according to some embodiments, may be provided with a plurality of foot pedals, including a first pair of foot pedals 2530, a second pair of foot pedals 2532, and a third pair of foot pedals 2534. The first pair of foot pedals 2530 is located on a first foot pedal level 2506 in the right foot portion 2505R of the foot pedal tray and includes a first foot pedal 2530a and a second foot pedal 2530b adjacent to the first foot pedal. The second pair of foot pedals 2532 is located on a second foot pedal level 2507 in the right foot portion 2505R of the foot pedal tray and includes a third foot pedal 2532a and a fourth foot pedal 2532b adjacent to the third foot pedal. The third pair of foot pedals 2534 is located on the left foot portion 2505L of the foot pedal tray and includes a fifth foot pedal 2534a and a sixth foot pedal 2534b. In some embodiments, both the fifth foot pedal 2534a and the sixth foot pedal 2534b of the third pair of foot pedals 2534 are located on the second foot pedal tier 2507 as shown. In other embodiments, the third pair of foot pedals may be located on the first foot pedal tier of the foot pedal tray 2505. In other embodiments, the fifth and sixth foot pedals may be located on different foot pedal tiers of the foot pedal tray. The foot pedal array 2501 also includes a first column sensor, which itself includes a first emitter 2540 and a first receiver 2550. The foot pedal array 2501 also includes a second column sensor, which itself includes a second emitter 2542 and a second receiver 2552. In some embodiments, any or all of the column sensors may be photoelectric sensors. In some embodiments, any or all of the emitters for the sensors may be optical emitters, such as infrared emitters, and any or all of the receivers may be optical detectors, such as photoelectric detectors or photodiodes.

[0245] As illustrated, emitters 2540 and 2542 are located in the first foot pedal hierarchy 2506 distal to or beyond the first foot pedal pair 2530, and receivers 2552 and 2550 are located in the second foot pedal hierarchy 2507 proximal to the second foot pedal pair 2532, although in some embodiments the positions may be reversed with the emitters located on the second foot pedal hierarchy proximal to the second foot pedal pair and the receivers located in the first foot pedal hierarchy distal to or beyond the first foot pedal pair. In some embodiments, both the transmitter and receiver are located in the first foot pedal hierarchy distal to the first foot pedal pair, and a reflector that reflects the emitted light back to the receiver is located in the second foot pedal hierarchy proximal to the second foot pedal pair. In some embodiments, both the transmitter and receiver are located in the second foot pedal level proximal to the second pair of foot pedals, and a reflector that reflects the emitted light back to the receiver is located in the first foot pedal level distal to the first pair of foot pedals. As used herein, the term "array sensor" refers to a sensor and emitter that directs a beam across the row of foot pedals generally in a forward / backward or proximal / distal direction. As shown in FIG. 26 , the first and second row sensors are configured such that a first beam 2580 emitted by the first emitter 2540 and a second beam 2582 emitted by the second emitter 2542 travel down the respective row of foot pedals, specifically foot pedals 2530 a and 2532 b for the first row sensor and foot pedals 2530 b and 2532 a for the second row sensor, to the first receiver 2550 or the second receiver 2552, respectively. 25 and 26 , in some embodiments, the first emitter 2540 and the second emitter 2542 are each disposed on a holder 2592 that protrudes from the base of the first foot pedal level 2506. In some embodiments, the holder may be “T-shaped” as shown. In some embodiments, the first receiver 2552 and the second receiver 2550 are inserted into the base of the second foot pedal level 2507.

[0246] In some embodiments, the first array sensor is configured to generate a signal indicative of a partial or complete blockage of the first beam 2580 received at the first receiver 2550 caused by an object (e.g., an operator's foot) located between the first emitter 2540 and the first receiver 2550, generate a signal indicative of an unblocked first beam received at the first receiver 2550, or both. In some embodiments, the second array sensor is configured to generate a signal indicative of a partial or complete blockage of the second beam 2582 received at the second receiver 2552 caused by an object (e.g., an operator's foot) located between the second emitter 2542 and the second receiver 2552, generate a signal indicative of an unblocked second beam received at the first receiver, or both.

[0247] In some embodiments, the foot pedal array 2501 is configured such that a foot positioned on and adjacent to or at the first foot pedal 2530a or the second foot pedal 2530b generates a signal indicating partial or complete blocking of the first beam, and a foot positioned on and adjacent to or at the third foot pedal 2532a or the fourth foot pedal 2532b generates a signal indicating partial or complete blocking of the first beam.

[0248] In some embodiments, the foot pedal array 2501 also includes a third column sensor including a third emitter 2544 configured to direct a third beam 2584 over at least a portion of the fifth foot pedal 2534a, as shown in FIG. 26 . The third column sensor also includes a third receiver 2556. Although the emitter 2544 directs a beam onto only one foot pedal, the sensor is still considered a “column” sensor because the beam is directed in the anterior-posterior or proximal-distal direction. In some embodiments, where the fifth foot pedal 2534a is on the same foot pedal hierarchy as the sixth foot pedal 2534a, the foot pedal array 2501 may also include a fourth column sensor including a fourth emitter 2546 configured to direct a fourth beam 2586 over at least a portion of the fourth foot pedal 2534b, as shown in FIG. 26 . The fourth column sensor also includes a fourth receiver 2556. The features and aspects described herein with respect to the first and second column sensor embodiments also apply to the third and fourth column sensors.

[0249] 27 illustrates another embodiment of an operator console 2700. Features of the operator console 2700 that are similar to features of the operator console 2500 described above are identified with the same reference numerals for convenience. In the foot pedal assembly 2701 of the operator console 2700, the third foot pedal pair 2734 is positioned such that the fifth foot pedal 2734a is on the first foot pedal level 2506 and the sixth foot pedal 2734b of the third foot pedal pair 2534 is on the second foot pedal level 2507. As shown in FIG. 27, for the operator console 2700, the third column sensor includes a third emitter 2744 and a third receiver 2754. Because the fifth foot pedal 2734a and the sixth foot pedal 2734b are both in the same row, the third emitter 2744 directs a beam across at least a portion of both the fifth foot pedal 2734a and the sixth foot pedal 2734b. In such an embodiment, one row sensor may be used for the third foot pedal pair 2734 instead of two row sensors.

[0250] 28 and 29 illustrate a foot pedal array 2801 according to another embodiment. With respect to the arrangement of the first pair of foot pedals 2530, the second pair of foot pedals 2532, the third set of foot pedals 2534, and the corresponding first, second, third, and fourth column sensors, the features of the foot pedal array 2801 and foot pedal tray 2805 are similar to those of the foot pedal array 2501 described above. The foot pedal array 2801 also includes a first column sensor including a first column emitter 2840 configured to direct a first column beam 2860 laterally along the second foot pedal layer 2507 and across at least a portion of the third foot pedal 2532 a and the fourth foot pedal 2532 b. The first row sensor also includes a first row receiver 2850 configured to receive the first row beam 2850. In embodiments in which a fifth foot pedal 2534 a and a sixth foot pedal 2534 b are also on the second foot pedal tier 2507, the first row beam 2850 is also directed across at least a portion of the fifth foot pedal 2534 a and the sixth foot pedal 2534 b. The term row may be used herein to refer to an axis running perpendicular to the column, such as laterally from the left foot portion of the foot pedal array 2501 to the right foot portion of the foot pedal array 2501, or vice versa.

[0251] 25 and 26 , the first array sensor is configured to generate a signal indicative of partial or complete blockage of the first beam 2580 received at the first receiver 2550 caused by an object (e.g., an operator's foot) located between the first emitter 2540 and the first receiver 2550, generate a signal indicative of an unblocked first beam received at the first receiver 2550, or both. The second array sensor is configured to generate a signal indicative of partial or complete blockage of the second beam 2582 received at the second receiver 2552 caused by an object (e.g., an operator's foot) located between the second emitter 2542 and the second receiver 2552, generate a signal indicative of an unblocked second beam 2582 received at the second receiver 2552, or both. The third column sensor is configured to generate a signal indicative of partial or complete blockage of the third beam 2584 received at the third receiver 2554 caused by an object (e.g., an operator's foot) located between the third emitter 2544 and the third receiver 2554, generate a signal indicative of an unblocked third beam 2584 received at the third receiver 2554, or both. The fourth column sensor is configured to generate a signal indicative of partial or complete blockage of the fourth beam 2586 received at the fourth receiver 2556 caused by an object (e.g., an operator's foot) located between the fourth emitter 2546 and the third receiver 2556, generate a signal indicative of an unblocked second beam 2586 received at the second receiver 2554, or both. In the foot pedal assembly 2801 of Figures 28 and 29, the first row sensor is configured to generate a signal indicating partial or complete blocking of the first row beam 2860 received at the first row receiver 2850 caused by an object positioned between the first row emitter 2840 and the first row receiver 2850, or to generate a signal indicating an unblocked first row beam 2860 received at the first row receiver 2850, or both.Additional row sensors in combination with the first, second, third, and fourth beam sensors allow the foot pedal array 2801 to identify which individual foot pedal has an object (e.g., foot 2880) on and adjacent to or at it based on which beams are blocked or which beams are not blocked.

[0252] 28 , a foot positioned on and adjacent to or on the first foot pedal 2530a generates a signal from the first column sensor indicating partial or complete blockage of the first beam 2580 and a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860. A foot positioned on and adjacent to or on the third foot pedal 2532a generates a signal from the first column sensor indicating partial or complete blockage of the first beam 2580 but does not generate a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860. A foot positioned on and adjacent to or on the second foot pedal 2530b generates a signal from the second column sensor indicating partial or complete blockage of the second beam 2582 and a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860. 29 , a foot positioned on and adjacent to or on the fourth foot pedal 2532b generates a signal from the second column sensor indicating partial or complete blockage 2882 of the second beam 2582 without generating a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860. A foot positioned on and adjacent to or on the fifth foot pedal 2534a generates a signal from the third column sensor indicating partial or complete blockage of the third beam 2586 without generating a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860. A foot positioned on and adjacent to or on the sixth foot pedal 2534b generates a signal from the third column sensor indicating partial or complete blockage of the fourth beam 2588 without generating a simultaneous signal from the first row sensor indicating partial or complete blockage of the first row beam 2860.

[0253] In some embodiments, the first column sensor may be located on the second foot pedal hierarchy instead of the first foot pedal hierarchy. In some embodiments, the first column sensor may be located on the first foot pedal hierarchy and the second column sensor may be located on the second foot pedal hierarchy. In some embodiments where the fifth and sixth foot pedals are on different foot pedal hierarchies and in a single column, the row sensor may be used to distinguish between an object on the fifth foot pedal and an object on the sixth foot pedal.

[0254] In some embodiments, the first row emitters 2840 are in the left portion 2505L of the foot pedal tray and the first row receivers 2850 are in the right portion 2505R of the foot pedal tray. In some embodiments, the first row emitters are in the right portion of the foot pedal tray and the first row receivers are in the left portion of the foot pedal tray. In some embodiments, the foot pedal tray includes a left side wall 2870 and a right side wall 2872 facing the left side wall, and the first row emitters 2840 and the first row receivers 2850 are attached to, extend from, or mounted to the left side wall 2870 or the right side wall 2872. In some embodiments, the first side wall (e.g., the left side wall 2870) at least partially defines and borders the left foot portion of the foot pedal tray 2805 of the first foot pedal level 2507. In some embodiments, the second sidewall (e.g., right sidewall 2872) at least partially defines the edge of the right foot portion of the foot pedal tray 2805 of the first foot pedal level 2506. In some embodiments, both the first row emitters and the first row receivers are located on the same side of the foot pedal tray, and a reflector is located on the opposite side of the foot pedal tray to reflect the emitted beams back to the receivers.

[0255] Function of foot pedals in surgical robotic systems The foot pedals described herein can be used to operate or control various functions of a surgical robotic system, according to some embodiments. For example, the first pair of foot pedals 2530 and / or the second pair of foot pedals 2532 can be configured to generate signals to control at least one electrosurgical function of the surgical robotic system. In some embodiments, the second pair of foot pedals 2532 can constitute a secondary electrosurgical foot pedal and the first pair of foot pedals 2530 can constitute a primary electrosurgical foot pedal, or vice versa. In some embodiments, the third pair of foot pedals 2534 can be configured to generate signals to change or control the mode of operation (or function) of the robotic assembly. In some embodiments, the signal generated by the fifth foot pedal 2534a or the signal generated by the sixth foot pedal 2534b may change or control the mode of operation or mode of control of the robot assembly from a default mode, in which movement of at least one hand controller causes a corresponding movement of a corresponding robot arm, to a different mode of controlling movement, such as a run mode, in which movement of at least one hand controller may cause movement of the base of the robot arm or the virtual chest of the robot assembly while keeping the camera assembly view centered on the instrument tip of the robot arm if the robot assembly has only one robot arm, or at a position between the instrument tips of the instrument arms if the robot assembly has two robot arms. In some embodiments, when the travel mode is activated, movement of one of the hand controllers causes a corresponding movement of the instrument tip of the corresponding robot arm relative to the view of the camera assembly, rotates the camera assembly to center the view of the camera assembly on the average tip position, and translates and / or rotates the virtual chest of the robot assembly to maintain the distance between the center of the virtual chest of the robot assembly and the average instrument tip position within an acceptable distance range, and maintains the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range.In some embodiments, while in the running mode, the chest and / or camera assembly of the robotic assembly automatically translates or reorients to maintain the configuration the robotic assembly had when entering the running mode.

[0256] In some embodiments, the signal generated by the other of the sixth foot pedal 2534b or the fifth foot pedal 2534a may change or control the mode of operation (or function) or mode of control (or articulation) of the robotic assembly from a default mode, in which movement of at least one hand controller causes a corresponding movement of the robotic arm, to a camera mode, also referred to herein as a view mode, in which movement of at least one hand controller changes the orientation and / or position of the camera assembly without changing the position or orientation of the at least one robotic arm. In some embodiments, the different mode is a view mode, in which movement of one of the hand controllers causes a corresponding change in the orientation of the camera assembly and the position of the camera assembly due to translation of the virtual chest, and a rotation of the virtual chest of the robotic assembly to maintain the angular deviation between a line from the center of the virtual chest to the location of the midpoint between the instrument tips of the robotic arms of the robotic assembly, which is the average instrument tip position, while maintaining the position and orientation of the instrument tips of all robotic arms of the robotic assembly.

[0257] In some embodiments, the different mode is a translation mode in which movement of one of the hand controllers causes a corresponding movement of the instrument tip of the corresponding robotic arm relative to the view of the camera assembly, causes a rotation of the camera assembly to center the view of the camera assembly on the average tip position, and causes a translation of the virtual chest to maintain the distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range. In some embodiments, the translation mode maintains the robot assembly in the configuration and orientation it had when entering the translation mode.

[0258] In some embodiments, the different mode is a swivel mode in which movement of one of the hand controllers causes a corresponding movement of the instrument tip of the corresponding robotic arm relative to the view of the camera assembly, causing a change in orientation of the camera assembly, or a change in orientation of the camera assembly and a change in orientation of the virtual chest to keep the view of the camera assembly centered on the average instrument tip position and to keep the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within an acceptable angular deviation range. Further disclosure regarding the different modes of operation and control can be found in the provisional application entitled "Hand Controllers and Control Modes for a Robotic Surgical System," filed on even date herewith, which is incorporated herein in its entirety.

[0259] In some embodiments, when an object, such as an operator's foot, breaks one or more beams of the foot pedal assembly, one or more signals from the first column sensor, the second column sensor, the third column sensor, the fourth column sensor, and / or at least one row sensor may cause the surgical robotic system to display information in a graphical user interface regarding the function of the foot pedal on which the operator's foot is currently resting or on. For example, FIG. 30 schematically illustrates a graphical user interface (GUI) 3000 for an operator, including a central area for displaying images based on image input from a camera assembly. The GUI may include one or more portions, e.g., a first information portion 3020 and a second information portion 3030, that display information, which may include any information regarding the control mode, the driving mode, the current configuration of the robotic assembly, and foot pedal selection information. In some embodiments, at least one information portion includes foot pedal selection information. For example, first information portion 3020 may include first foot pedal group selection information 3022, where the first foot pedal group includes a third pair of foot pedals, and the foot pedal selection information indicates which foot pedal the user's feet are on and / or the function of the foot pedal the user's feet are on. In some embodiments, second information portion 3030 includes second foot pedal group selection information 3032. In some embodiments, the first foot pedal group includes a first pair of foot pedals and a second pair of foot pedals, and the foot pedal selection information indicates which foot pedal the user's feet are on and / or the function of the foot pedal the user's feet are on. In some embodiments, the first foot pedal group selection information and the second foot pedal group selection information may be included in the same information portion of the GUI. In some embodiments, the second foot pedal group includes the first pair of foot pedals, and the second pair of foot pedals is included in a third foot pedal group located in a third information portion of the GUI.

[0260] While some embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Numerous variations, changes, and substitutions will now occur to those skilled in the art without departing from the invention. It will be understood that various alternatives to the embodiments of the invention described herein may be employed in practicing the invention. The following claims define the scope of the invention, and it is intended that methods and structures within the scope of these claims and their equivalents be covered thereby.

Claims

1. One or more hand controllers for controlling a surgical robotic system including a robotic assembly configured to be positioned in an internal body cavity of a subject, each of the one or more hand controllers comprising: a contoured housing having a top surface, an inner side surface adjacent the top surface, an outer side surface facing away from the inner side surface, and a bottom surface facing away from the top surface; a plurality of buttons including a first button positioned on the top surface and a second button positioned on one of the top surface, the inner side surface, or the outer side surface; a first touch input device positioned on the top surface; a first paddle disposed on either the inner side or the outer side; the contoured housing is configured to be grasped on the top surface with a user's thumb on the inner side and at least a portion of the user's fingers, the user's palm, or both, and the first button is configured to be operated by the user's index finger; For at least one of the hand controllers, at least one of the plurality of buttons is resetting the orientation of the surgical robotic assembly relative to the plurality of robotic arms and the virtual chest of the robotic assembly to a default orientation while maintaining the position and orientation of the instrument tip relative to each of the plurality of robotic arms; or one or more hand controllers that are mapped to a function selected from one or more of functions for activating or exiting a view mode of the surgical robotic system, wherein activating the view mode causes a change in orientation of a camera assembly, or causes the change in orientation of the camera assembly and the change in orientation of the virtual chest to center the camera assembly view at a location midway between the instrument tips of the multiple robotic arms, which is an average instrument tip position, and when the view mode is activated, movement of one of the hand controllers causes a corresponding movement of an instrument tip of a corresponding one of the multiple robotic arms relative to the camera assembly view, causing the change in orientation of the camera assembly, or the change in orientation of the camera assembly and the change in orientation of the virtual chest to keep the camera assembly view centered at the average instrument tip position.

2. The one or more hand controllers of claim 1 , wherein for each of the one or more hand controllers, a contact surface of the first paddle is disposed on the outer side of the housing.

3. 2. The one or more hand controllers of claim 1, wherein each of the one or more hand controllers further comprises a second paddle, wherein a contact surface of the first paddle is disposed on the outer side surface of the housing and a contact surface of the second paddle is disposed on the inner side surface of the housing, or wherein the contact surface of the first paddle is disposed on the inner side surface of the housing and the contact surface of the second paddle is disposed on the outer side surface of the housing.

4. 4. The one or more hand controllers of claim 3, wherein the first paddle is engaged with the second paddle via one or more gears, such that movement of the first paddle causes reciprocating movement of the second paddle and movement of the second paddle causes reciprocating movement of the first paddle.

5. 5. One or more hand controllers according to claim 1, wherein for each of the hand controllers, the first touch input device is a scroll wheel, a rocker button, a joystick, a pointing stick, a touchpad, or a trackball.

6. The one or more hand controllers of any one of claims 1 to 5, wherein at least one of the one or more hand controllers further comprises a second touch input device.

7. 7. The one or more hand controllers of claim 1, wherein, for each of the hand controllers, deflection of the first paddle is configured to generate a signal that the surgical robotic system uses as an input to control a robotic subassembly.

8. For at least one of the hand controllers, at least one of the plurality of buttons is a function that triggers the activation of a clutch for a hand controller, the activation of the clutch allowing or enabling movement of the respective hand controller without causing movement of the robot assembly; or the ability to activate or exit an instrument control mode of the surgical robotic system, wherein, when in the instrument mode, movement of one of the hand controllers causes a corresponding movement in a corresponding robotic arm of the robotic assembly; or the ability to activate or exit a scan mode of the surgical robotic system, wherein when in the scan mode, movement of one of the hand controllers causes a corresponding change in orientation of a camera assembly of the robotic assembly, but the robotic arm of the robotic assembly remains stationary if the robotic assembly includes one robotic arm, or all robotic arms of the robotic assembly remain stationary if the robotic assembly includes more than one robotic arm; or a function to activate or deactivate a menu feature of the surgical robotic system, the menu feature causing a menu to be displayed on a graphical user interface of the surgical robotic system and input from a touch input device or button of the one or more hand controllers to be used to select an item in the menu; or the ability to navigate among and / or select options from the menus displayed on the graphical user interface of the surgical robotic system; or the ability to activate or deactivate an elbow biasing feature, the elbow biasing feature allowing adjustment of the magnitude and direction of elevation of the elbow of a robotic arm while maintaining the position of the tool tip of the robotic arm relative to each robotic arm of the robotic assembly; or for one robot arm of said robot assembly, or for each robot arm of said robot assembly, the ability to control the adjustment of the elevation of the elbow of a robot arm of said robot assembly; or 8. One or more hand controllers according to any one of claims 1 to 7, which are mapped to a function selected from one or more of the following functions: change or select the zoom of a view from a camera assembly of the robot assembly.

9. The one or more hand controllers of claim 8 , wherein for at least one of the hand controllers, at least two of the plurality of buttons are mapped to a function selected from the one or more functions.

10. The one or more hand controllers of claim 8 , wherein for at least one of the hand controllers, at least three of the plurality of buttons are mapped to functions selected from the one or more functions.

11. 11. The one or more hand controllers of claim 8, wherein for at least one of the one or more hand controllers, the first touch input device has one or more functions selected from the one or more functions.

12. at least one of the one or more hand controllers further comprises a second touch input device; 12. The one or more hand controllers of claim 8, wherein for at least one of the one or more hand controllers, the first touch input device or the second touch input device has one or more functions selected from the one or more functions.

13. at least one of the one or more hand controllers further comprises a second touch input device; 13. The one or more hand controllers of claim 12, wherein for at least one of the one or more hand controllers, the first touch input device and the second touch input device each have one or more functions selected from the one or more functions.

14. 14. The one or more hand controllers of any one of claims 1-13, wherein, for at least one of the one or more hand controllers, actuation of the first button generates a signal that triggers activation of a clutch feature of the surgical robotic system for the hand controller, activation of the clutch feature allowing or enabling movement of the respective hand controller without causing movement of the robotic assembly.

15. One or more hand controllers according to any one of claims 1 to 14, wherein for each hand controller, the outer side of the housing is substantially concave.

16. 16. The one or more hand controllers of claim 1, wherein each of the one or more hand controllers further comprises a thumb pad located on either the inner side or the outer side.

17. 17. The one or more hand controllers of any one of claims 1 to 16, wherein at least one of the one or more hand controllers includes a sensor configured to sense contact of the operator's hand with the hand controller or to sense proximity of the operator's hand to the hand controller.

18. 18. The one or more hand controllers of claim 17, wherein the input from the sensor is used to activate or exit an instrument control mode of the surgical robotic system, wherein, when in the instrument mode, movement of one of the hand controllers causes a corresponding movement in a corresponding robotic arm of the robotic assembly.

19. One or more hand controllers for controlling a surgical robotic system including a robotic assembly configured to be positioned in an internal body cavity of a subject, each of the one or more hand controllers comprising: a contoured housing having a top surface, an inner side surface adjacent the top surface, an outer side surface facing away from the inner side surface, and a bottom surface facing away from the top surface; a plurality of buttons including a first button positioned on the top surface and a second button positioned on one of the top surface, the inner side surface, or the outer side surface; a first touch input device positioned on the top surface; a first paddle disposed on the outer side; a second paddle disposed on the inner side surface, one or more hand controllers, wherein the contoured housing is configured to be grasped on the top surface with a user's thumb on the inner side and at least a portion of the user's fingers, the user's palm, or both, and the first button is configured to be operated by the user's index finger.

20. 1. A hand controller system for controlling a surgical robotic system including a robotic assembly configured to be positioned within an internal body cavity of a subject, the hand controller system comprising: A first hand controller according to any one of claims 1 to 20, configured to be removably coupled or connected to an operator console via a first hand controller support; and a second hand controller according to any one of claims 1 to 20, configured to be removably coupled or connected to the operator console via a second hand controller support, wherein an inner surface of the first hand controller faces an inner surface of the second hand controller when the first and second hand controllers are in a neutral position during use.

21. 1. A hand controller system for controlling a surgical robotic system including a robotic assembly configured to be positioned in an internal body cavity of a subject, the hand controller system comprising: a first hand controller configured to be removably coupled or connected to an operator console via a first hand controller support, a contoured housing having a top surface, a first side surface adjacent the top surface, a second side surface facing away from the first side surface, and a bottom surface facing away from the top surface; a first button and a first scroll wheel disposed on the top surface; a first paddle having a contact surface disposed on the first side or the second side; a first hand controller including a second button disposed on the first side or the second side; a second hand controller configured to be removably coupled or connected to the operator console via a second hand controller support, a contoured housing having a top surface, a first side surface adjacent the top surface, a second side surface facing away from the first side surface, and a bottom surface facing away from the top surface; a first button and a first scroll wheel disposed on the top surface; a first paddle having a contact surface disposed on the first side or the second side; a second hand controller comprising: a second button disposed on the first side or the second side, wherein an inner surface of the first hand controller faces an inner surface of the second hand controller when the first and second hand controllers are in a neutral position during use.

22. 1. A surgeon console comprising: A hand controller system according to claim 1, claim 19 or claim 20; A plurality of foot pedals, activating a view mode of the surgical robot system, wherein when the view mode is activated, movement of one of the hand controllers causes a corresponding change in orientation and position of the camera assembly and a rotation of the virtual chest of the robot assembly to maintain the positions and orientations of the instrument tips of all robot arms of the robot assembly while maintaining an angular deviation between a line from the center of the virtual chest to a position of the midpoint between the instrument tips of the robot arms of the robot assembly, which is an average instrument tip position, and a normal to the virtual chest within an allowable angular deviation range; a foot pedal configured to activate a view mode wherein activating the view mode further causes a change in orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of a virtual chest of the robot assembly, to center the view of the camera assembly at a location midway between instrument tips of the robot arms of the robot assembly, which is an average instrument tip position, and wherein, when the view mode is activated, movement of one of the hand controllers causes a corresponding movement of an instrument tip of the corresponding robot arm relative to the view of the camera assembly, causing a change in orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to keep the view of the camera assembly centered at the average instrument tip position and maintain an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range; a foot pedal configured, during operation, to activate a drive mode of the surgical robot system, wherein when the drive mode is activated, movement of one of the hand controllers causes a corresponding movement of an instrument tip of a corresponding robot arm with a corresponding movement including a corresponding translation and a corresponding rotation relative to the view of the camera assembly, rotates the camera assembly to center the view of the camera assembly on the average tip position, translates and / or rotates a virtual chest of the robot assembly to maintain a distance between a center of the virtual chest of the robot assembly and the average instrument tip position within an acceptable distance range, and maintains an angular deviation between a line from the center of the virtual chest to the average instrument tip position and a normal to the virtual chest within an acceptable angular deviation range; a foot pedal configured, when operated, to activate a translation mode of the surgical robotic system, wherein when a transverse mode is activated, movement of one of the hand controllers causes a corresponding movement of an instrument tip of a corresponding robotic arm relative to the view of the camera assembly, causes a rotation of the camera assembly to center the view of the camera assembly on the average tip position, and causes a translation of the virtual chest to maintain a distance between the center of the virtual chest and the average instrument tip position within an acceptable distance range.

23. 1. A method for controlling a robotic assembly of a surgical robotic system, the surgical robotic system comprising a video display and right and left hand controllers configured to sense hand movements of an operator, the robotic assembly comprising a camera assembly and a robotic arm assembly including a first robotic arm and a second robotic arm, each of the first robotic arm and the second robotic arm having an associated instrument tip, the positions and orientations of the robotic arms and camera assembly defining a virtual chest of the robotic assembly, the method comprising: receiving a first control mode selection input while the robot assembly is positioned within the internal cavity of the subject; and changing a default control mode of the surgical robot system to a first control mode in response to the first control mode selection input, wherein the surgical robot system has a plurality of control modes including one or more of a scan mode, a view mode, a drive mode, and a translate mode, and the first control mode is the scan mode, the view mode, the drive mode, or the translate mode; The method includes, while the robot assembly is in the first control mode, in response to a first movement of the right hand controller or the left hand controller, the first movement including a first translation and a first rotation: if the first control mode is the scan mode, holding the robotic arm assembly stationary while rotating the camera assembly a corresponding first rotation relative to a view of the camera assembly displayed on the video display of the surgical robot system, the view being a displayed camera view; if the first control mode is the travel mode, moving a corresponding instrument tip by a corresponding first movement, including a corresponding first translation and a corresponding first rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on a location of a midpoint between instrument tips of the robot arm of the robot assembly, which is an average tip position, or while translating the robot assembly to translate a chest point of the virtual chest and rotating the robot assembly to rotate the virtual chest, or both, to maintain a distance between the center of the virtual chest and the average tip position within an acceptable distance range and maintain an angular deviation between a line from the chest point to the average tip position and a normal to the virtual chest within an acceptable angular deviation range; If the first control mode is the view mode, then rotating a virtual chest of the robot assembly while moving the camera assembly by a corresponding first movement, the first movement including a corresponding first translation of the virtual chest and a corresponding first rotation of the camera assembly relative to the displayed camera view. maintaining the position and orientation of a virtual chest (competant) to maintain an angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range; moving a corresponding instrument tip by a corresponding first translation, including a corresponding first rotation, relative to the displayed camera view while rotating the camera assembly to center the view of the camera assembly on the average instrument tip position; causing a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to maintain the view of the camera assembly centered on the average instrument tip position and to maintain an angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range; if the first control mode is the translation mode, rotating the camera assembly, or rotating the camera assembly and translating the virtual chest while moving a corresponding instrument tip by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the displayed camera view to center the view of the camera assembly on the average tip position and to maintain a distance between the center of the virtual chest and the average instrument tip position within the acceptable distance range.

24. 24. The method of claim 23, wherein the current control mode is an instrument control mode in which movement of the right hand controller or the left hand controller causes corresponding movement of an instrument tip at the distal end of a corresponding robotic arm relative to the displayed camera view without moving or changing the orientation of the virtual chest of the robotic assembly.

25. 25. The method of claim 23 or 24, wherein the first control mode selection input is received via a foot pedal of the surgical robotic system.

26. 25. The method of claim 23 or 24, wherein the first control mode selection input is received via the first hand controller or the second hand controller.

27. 27. The method of claim 26, wherein the first control mode selection input is received via a button or a touch input device on the right hand controller or the left hand controller.

28. The method of any one of claims 23 to 27, wherein the first control mode is the view mode.

29. The method according to any one of claims 23 to 27, wherein the first control mode is the scan mode.

30. The method of any one of claims 23 to 27, wherein the first control mode is the driving mode.

31. The method of any one of claims 23 to 27, wherein the first control mode is the transverse mode.

32. The method of any one of claims 23 to 27, wherein the first control mode is the view mode.

33. receiving a second control mode selection input; and changing a current control mode of the surgical robot system to a second control mode different from the first control mode in response to the second control mode selection input, wherein the plurality of control modes of the surgical robot system include two or more of a scanning mode, the view mode, the travel mode, and the translation mode; The method further includes, while the current control mode is the second control mode, in response to a second movement of the right hand controller or the left hand controller, the second movement including a second translation and a second rotation: if the second control mode is the scan mode, holding the robot arm assembly stationary while rotating the camera assembly a corresponding second rotation relative to a displayed camera view; if the second control mode is the travel mode, moving the corresponding instrument tip by a corresponding second translation, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while rotating the camera assembly to center the view of the camera assembly on an average tip position, or while translating the robot assembly to translate the chest point of the virtual chest and rotating the robot assembly to rotate the virtual chest, or both, to maintain the distance between the center of the virtual chest and the average tip position within the acceptable distance range and to maintain an angular deviation between the line from the chest point to the average tip position and the normal to the virtual chest within the acceptable angular deviation range; If the second control mode is the view mode, move the camera assembly by a corresponding second movement, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, and rotate the virtual chest of the robot assembly while maintaining the position and orientation of each instrument to maintain the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range, and rotate the camera assembly by a corresponding second movement, including a corresponding second translation and a corresponding second rotation, relative to the displayed camera view, while rotating the camera assembly. moving a corresponding instrument tip by a corresponding second movement including a second rotation about the center of the virtual chest to center the view of the camera assembly on the average instrument tip position, causing a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to keep the view of the camera assembly centered on the average instrument tip position and to keep the angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range; 33. The method of any one of claims 23 to 32, further comprising, if the second control mode is the translation mode, rotating the camera assembly, or rotating the camera assembly and translating the virtual chest while moving a corresponding instrument tip by a corresponding second movement comprising a corresponding second translation and a corresponding second rotation relative to the displayed camera view to center the view of the camera assembly on the average tip position and to maintain the distance between the center of the virtual chest and the average instrument tip position within the acceptable distance range.

34. 34. The method of any one of claims 23 to 33, wherein the virtual chest is defined by a chest plane extending between a first pivot point of a most proximal joint of the first robotic arm, a second pivot point of a most proximal joint of the second robotic arm, and a camera imaging center point of the camera assembly when the camera is in a default position relative to the plunge.

35. receiving a first function selection input corresponding to an input function located on the left hand controller, the right hand controller, or both; The method of any one of claims 23 to 34, further comprising: performing a first function in response to receiving the first function selection input.

36. performing the first function, displaying a menu on a portion of the video display while holding the position and orientation of the camera assembly and the robotic arm assembly constant; resetting the orientation and position of the camera assembly to align the camera view with the mean instrument tip position and the virtual chest; or 36. The method of claim 35, comprising activating a clutch for a hand controller to allow movement of each hand controller without causing movement of the robot assembly.

37. the first function includes displaying the menu on the portion of the video display while holding constant the positions and orientations of the camera assembly and the robotic arm assembly, the method comprising: receiving one or more second function selection inputs corresponding to one or more input functions located on the left hand controller, the right hand controller, or both; 37. The method of claim 35 or 36, comprising: in response to receiving the one or more second function selection inputs, graphically indicating options on a portion of the image display for one or more menu items, one of which is selected based on the one or more second function inputs, and displaying one or more sub-menus on the portion of the image display corresponding to the one or more second function inputs.

38. receiving at least one third feature selection input that selects an option on the visual display corresponding to an input function located on the left hand controller, the right hand controller, or both; if the at least one third feature selection input selects a zoom level, changing a zoom level of the displayed camera view in response to receiving the at least one third feature selection input and storing information about the new zoom level for zoom level dependent control mode modification; or and if the at least one third feature selection input comprises selecting an elbow bias level, or selecting an elbow bias menu and selecting an elbow bias level from the elbow bias menu, modifying the elbow bias of a first robotic arm, the elbow bias of a second robotic arm, or both, in response to receiving the third feature selection input.

39. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic surgical system, perform the method of any one of claims 23 to 38.

40. 1. A surgical robotic system for performing a surgical procedure within an internal cavity of a subject, the surgical robotic system comprising: a right hand controller and a left hand controller operative to operate the surgical robotic system; a camera assembly; a robotic arm assembly configured to be inserted into the internal cavity during use, the robotic arm assembly comprising: a first robotic arm including or coupled to a first instrument tip disposed at a distal end of the first robotic arm; a robotic arm assembly including a second robotic arm including or coupled to a second instrument tip disposed at a distal end of the second robotic arm; an image display for outputting images from said camera assembly; at least one computing module or control unit, receiving from and responsive to the right hand controller and the left hand controller, generating control signals based on a current control mode of the surgical robotic system; receiving a control mode selection input and, in response thereto, changing the current control mode of the surgical robotic system to a selected one of a plurality of control modes of the surgical robotic system; and at least one computing module or control unit configured to receive a function selection input from the first hand controller and / or the left hand controller and generate a control signal to perform a corresponding function of a plurality of functions.

41. the plurality of control modes include one or more of a scan mode, a view mode, a travel mode, and a translation mode; while the current control mode is the scan mode, in response to a first movement comprising a first rotation of the right hand controller or the left hand controller, the robotic arm assembly is held stationary while rotating at a corresponding first rotation relative to a displayed camera view, the displayed camera view being a view of the camera assembly displayed on the video display; while the current control mode is the drive mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, a corresponding instrument tip moves by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the displayed camera view while the camera assembly rotates to center the view of the camera assembly on a location of a midpoint between instrument tips of the robot arm of the robot assembly, which is an average tip position, or while the robot assembly translates to translate a chest point of the virtual chest and rotates the robot assembly to rotate the virtual chest, or both, to maintain a distance between the center of the virtual chest and the average tip position within an acceptable distance range and maintain an angular deviation between a line from the chest point to the average tip position and a normal to the virtual chest within an acceptable angular deviation range; While the current control mode is a view mode, in response to a first movement including a first translation and a first rotation of the right hand controller or the left hand controller, the position and orientation of each instrument is held while the camera assembly moves by a corresponding first movement including a corresponding first translation and a corresponding first rotation relative to the displayed camera view and a virtual chest of a robot assembly rotates to maintain an angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range, and the corresponding instrument tip is moving the camera assembly by a corresponding first translation, including a corresponding first rotation, relative to the displayed camera view to center the view of the camera assembly on the average instrument tip position, causing a change in the orientation of the camera assembly, or a change in the orientation of the camera assembly and a change in the orientation of the virtual chest, to maintain the view of the camera assembly centered on the average instrument tip position and to maintain an angular deviation between the line from the center of the virtual chest to the average instrument tip position and the normal to the virtual chest within the allowable angular deviation range; 41. The surgical robotic system of claim 40, wherein while the current control mode is the translation mode, in response to a first movement comprising a first translation and a first rotation of the right hand controller or the left hand controller, the corresponding instrument tip moves by a corresponding first movement comprising a corresponding first translation and a corresponding first rotation relative to the displayed camera view while the camera assembly rotates, or while the camera assembly rotates and the virtual chest translates, to center the view of the camera assembly on the average tip position and maintain a distance between the center of the virtual chest and the average instrument tip position within the acceptable distance range.

42. 42. The surgical robotic system of claim 41, wherein the plurality of modes further includes an instrument control mode in which movement of the right hand controller or the left hand controller causes corresponding movement of an instrument tip at the distal end of a corresponding robotic arm relative to the displayed camera view without moving or changing an orientation of the virtual chest of the robotic assembly.

43. 43. The surgical robot system of claim 41 or 42, wherein the plurality of modes includes two or more of the scan mode, the view mode, the travel mode, and the translation mode.

44. 43. The surgical robot system of claim 41 or 42, wherein the plurality of modes includes three or more of the scan mode, the view mode, the travel mode, and the translation mode.

45. The surgical robot system of any one of claims 41 to 44, wherein the plurality of modes includes the scan mode.

46. The surgical robot system of any one of claims 41 to 45, wherein the plurality of modes includes the view mode.

47. The surgical robot system of any one of claims 41 to 46, wherein the plurality of modes includes the travel mode.

48. The surgical robot system of any one of claims 41 to 47, wherein the plurality of modes includes the translational mode.

49. 49. The surgical robotic system of claim 40, wherein the surgical robotic system includes at least one pedal, and wherein the control mode selection input is received via the at least one pedal.

50. 49. The surgical robotic system of claim 40, wherein the surgical robotic system includes at least one pedal, and wherein the control mode selection input is received via the at least one pedal, or via the right hand controller or the left hand controller.

51. 49. The surgical robotic system of claim 40, wherein the first control mode selection input is received via a button or a touch input device on the right hand controller or the left hand controller, or via at least one pedal.

52. 52. The surgical robotic system of claim 41, wherein the virtual chest is defined by a chest plane extending between a first pivot point of a left robotic arm's most proximal joint, a second pivot point of a right robotic arm's most proximal joint, and a camera imaging center point of the camera assembly when the camera assembly is in a default position relative to the plunge.

53. The plurality of functions are displaying a menu on a portion of the video display while holding the position and orientation of the camera assembly and the robotic arm assembly constant; displaying a submenu on a portion of the image display; selecting an option from the menu or submenu displayed on the portion of the image display; resetting the orientation and position of the camera assembly to align the camera view with the mean instrument tip position and the virtual chest; resetting the pose of the robotic assembly relative to the robotic arms and virtual chest to a default pose while maintaining the position and orientation of the instrument tip relative to each of the robotic arms; activating a clutch for a hand controller to allow movement of each hand controller without causing movement of the robot assembly; activating or exiting an instrument control mode of the surgical robotic system, wherein, when in the instrument mode, movement of one of the hand controllers causes a corresponding movement in a corresponding robotic arm of the robotic assembly; activating or deactivating an elbow biasing feature, the elbow biasing feature allowing adjustment of the magnitude and direction of elevation of the elbow of the robotic arm while maintaining the position of the tool tip of the robotic arm relative to each robotic arm of the robotic assembly; controlling, for one robot arm of the robot assembly or for each robot arm of the robot assembly, an adjustment of an elbow elevation of a robot arm of the robot assembly; and changing or selecting a zoom of a view from a camera assembly of the robot assembly.

Images