Interaction Between the User Interface and the Master Controller

Abstract

A method is provided for controlling actuation of a click event by a manual selector movably attached to a mounting structure, the method comprising: in a first control state, applying a holding force to the manual selector; in a second control state, applying a tactile force to the manual selector that increases as a function of increasing displacement of the manual selector from a neutral position; and in a third control state, applying a click event signal that causes generation of a click event in a display system, and applying a tactile force to the manual selector that decreases in magnitude to a decreasing magnitude.

Description

[Technical Field]

[0001] (Reference to Related Application) This application claims the benefit of priority to U.S. Patent Application No. 63 / 120,202, filed December 1, 2020, and U.S. Patent Application No. 63 / 187,879, filed May 12, 2021, each of which is incorporated by reference in its entirety. [Background technology]

[0002] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during diagnostic or surgical procedures, thereby reducing patient recovery time, discomfort, and adverse side effects. Teleoperated surgical systems using robotic technology (so-called surgical robotic systems) are sometimes used to overcome the limitations of manual laparoscopic and open surgery. Advances in telepresence systems provide surgeons with an inside view of the patient's body, an increased number of degrees of surgical instrument movement, and the ability for surgical coordination over long distances. Teleoperated control of surgical robotic techniques typically involves user interaction with hand-controlled manipulators that control the movement of surgical instruments and finger-controlled selectors that trigger the occurrence of robotic system events. Haptic feedback can enhance a user's teleoperated control of surgical robotic techniques. Summary of the Invention

[0003] In one aspect, a method for controlling actuation of a click event by a manual selector movably attached to a mounting structure is provided. A sensor senses a displacement distance of the manual selector from a neutral position. While the manual sensor is at a displacement distance less than a first threshold distance from the neutral position, one or more motors are controlled according to a first control state to apply a maintenance force. While the manual sensor is at a displacement distance between the first threshold distance from the neutral position and a second threshold distance from the neutral position, the motors are controlled according to a second control state to apply a haptic force to the manual selector that increases as a function of increasing displacement of the manual selector from the neutral displacement position. Once the manual sensor meets the second threshold distance from the neutral position, a click event signal is applied to trigger the generation of a click event in the display system. Also, once the manual sensor meets the second threshold distance from the neutral position, the one or more motors are controlled according to a third control state to reduce the magnitude of the haptic force applied to the manual selector to a reduced magnitude that is less than the maximum magnitude of the haptic force applied during the second control state.

[0004] In another aspect, a method is provided for controlling cursor movement within a first two-dimensional (2D) plane based on movement of a user input device within a second 2D plane and movement of a manual selector movably attached to the user input device. The cursor is moved within the first 2D plane to follow movement of the user input device within the second 2D plane according to a constant movement ratio, while the manual selector moves relative to the user input device at a speed less than a first threshold speed. In response to a speed of movement of the manual selector relative to the controller being between the first threshold speed and a second threshold speed, the cursor is moved within the first 2D plane to follow movement of the user interface device within the second 2D plane according to a movement ratio that decreases as a function of an increase in the speed of movement of the manual selector relative to the user input device. In response to the speed of movement of the manual selector relative to the user input device decreasing below a second threshold speed, the cursor is moved within the first 2D plane to follow movement of the user input device within the second 2D plane according to a movement ratio that increases as a function of the decrease in the speed of movement of the manual selector relative to the user input device.

[0005] In the drawings, which are not necessarily drawn to scale, like numbers may describe similar components in different views. Like numbers with different suffix letters may represent different instances of similar components. Several embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is an illustration illustrating an exemplary teleoperated surgical system. [Figure 2A] FIG. 1 is an illustration illustrating an exemplary user input control system according to one embodiment. [Figure 2B] FIG. 1 is an illustration illustrating an example tool operation controller of an example user input control system. [Figure 2C]FIG. 1 is an illustration illustrating an example armrest of an example user input control system. [Figure 3] 1 illustrates an exemplary virtual surgical site viewable in the display system viewing plane during 3D mode operation. [Figure 4A] 1 illustrates an exemplary graphical user interface viewable in the display system viewing plane during 2D mode operation. [Figure 4B] 1 illustrates an exemplary graphical user interface viewable in the display system viewing plane during 2D mode operation. [Figure 4C] 1 illustrates an exemplary graphical user interface viewable in the display system viewing plane during 2D mode operation. [Figure 4D] 1 illustrates an exemplary graphical user interface viewable in the display system viewing plane during 2D mode operation. [Figure 5] FIG. 1 is an illustration showing an exemplary visual plane and an exemplary tactile plane. [Figure 6A] FIG. 10 is an illustration showing details of an exemplary user input device that acts as a mount for a manual selector. [Figure 6B] FIG. 1 is a functional block diagram illustrating a control system for receiving user input at a user input device including a manual selector and controlling the user input device to provide haptic feedback. [Figure 7] FIG. 10 is an illustrative diagram showing a first control function curve representing tactile force versus displacement of a displaceable manual selector position during a click event, illustrating a time-aligned sequence of grip button displacement and a time-aligned sequence with a hand formation sequence. [Figure 8] FIG. 10 is an exemplary flow diagram illustrating a control process for controlling the delivery of a haptic force and the triggering of a click event based on the displacement of a manual selector relative to a user input device. [Figure 9] FIG. 10 is an illustration depicting the configuration of an input controller to perform a first transformation in the absence of a click event. [Figure 10] FIG. 10 is an illustration depicting the configuration of an input controller to perform a second transformation in the presence of a click event. [Figure 11] FIG. 10 is an illustrative diagram showing a second control function curve representing an exemplary second conversion function for determining controller movement filtering with respect to the time between click events, and also showing a time-aligned sequence of grip button displacements, a time alignment with a sequence of hand formations, and a time-aligned sequence of view plane instances. [Figure 12] FIG. 1 is an exemplary block diagram of an exemplary computer system. DETAILED DESCRIPTION OF THE INVENTION

[0007] The following description is presented to enable any person skilled in the art to make and use a medical device simulator system and method. Various modifications to the embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments and applications without departing from the scope of the inventive subject matter. Moreover, in the following description, numerous details are set forth for purposes of explanation. However, those skilled in the art will understand that the inventive subject matter may be practiced without these specific details. In other instances, well-known machine components, processes, and data structures are shown in block diagram form so as not to obscure the disclosure with unnecessary detail. Flow diagrams in the figures referenced below are used to represent processes. A computer system may be configured to perform some of these processes. Modules in the flow diagrams representing computer-implemented processes represent the configuration of a computer system in accordance with computer program code to perform the actions described with reference to those modules. Thus, the inventive subject matter is not intended to be limited to the embodiments shown, but is to be accorded the widest scope consistent with the principles and configurations disclosed herein.

[0008] (Remotely operated surgical system) 1 is a schematic diagram illustrating an exemplary teleoperated surgical system 100. Teleoperated surgical system 100 includes an instrument manipulator assembly 102, which may include one or more links for manipulating the movement of a surgical instrument 104 in response to user input controls in performing various procedures on a patient 106. Instrument manipulator assembly 102 is attached to or located near a surgical table 108. A user input control system 110 enables a user 112 to view the surgical site and control instrument manipulator assembly 102.

[0009] In alternative embodiments, the exemplary teleoperated surgical system 100 may include more than one instrument manipulator assembly 102. The exact number of manipulator assemblies will depend, among other things, on the surgical procedure and the space constraints within the operating room.

[0010] The user input control system 110 can be located in the same room as the operating table 108. However, it should be understood that a user 112, such as a surgeon or clinician, can be located in a different room or in a different building altogether from the patient 106. The user input control system 110 generally includes a vision system, including a visualization system 116 and a display system 120, and a motion system, including one or more user input devices 204, one or more instrument manipulator assemblies 102, and an instrument motion and input device haptic feedback haptic controller 118 (referred to herein as an “input controller”).

[0011] The one or more user input devices 204 are operatively coupled to the one or more instrument manipulator assemblies 102 to control the movement of the one or more instruments 104 in response to user input provided at the user input devices 204. In the exemplary teleoperated surgical system 100, the one or more user input devices 204 and the one or more instrument manipulator assemblies 102 are communicatively coupled to an input controller 118. An exemplary motion controller processes user input received at the one or more user input devices 204 to control the motion of the one or more instrument manipulator assemblies 102. The exemplary input controller 118 generates haptic feedback signals used to adjust the state of haptic forces at the one or more user input devices 204 based on the movement of the one or more instrument manipulator assemblies 102 and / or based on the movement of the user input devices 204.

[0012] The user input device 204 may include any number of different input devices, such as a gravity counterbalance arm, joystick, trackball, glove, trigger grip, twistable knob, twistable grip, slider, lever push button, etc. In some embodiments, the user input device 204 may have as many degrees of freedom as the associated surgical instrument 104 to provide the user 112 with a sense of telepresence or a sense that the user 112 is one with the instrument 104 so that they have a strong sense of direct control over the instrument 104. In some embodiments, the user input device 204 is a manual input device that moves with six or more degrees of freedom and may include an actuatable handle or other control configuration (e.g., one or more buttons, switches, etc.) for actuating the instrument (e.g., to close the grasping jaws, apply a potential to an electrode, deliver a medical procedure, etc.).

[0013] The visualization system 116 provides the user 112 with simultaneous two-dimensional or three-dimensional images of the surgical site as the user 112 manipulates one or more instruments. The visualization system 116 may include a viewing scope assembly such that visual images may be captured by an endoscope positioned within the surgical site. The visualization system 116 may be implemented as hardware, firmware, software, or a combination thereof that interacts with or is otherwise executed by one or more computer processors, which may include the processors of the control system 110.

[0014] A display system 120 may display visual images of the surgical site and surgical instrument 104 captured by the visualization system 116. The display system 120 and user input device 204 may be oriented such that the relative positions of the visual imaging device and surgical instrument 104 within the scope assembly resemble the relative positions of a surgeon's eyes and hands, so that an operator (e.g., user 112) may manipulate the surgical instrument 104 with the user input device 204 as if viewing a working volume adjacent to the instrument 104 in substantially true presence. By "true presence" it is meant that the image presentation is a true perspective image that simulates the perspective of an operator physically manipulating the surgical instrument 104.

[0015] The implement motion input controller 118 includes at least one processor circuit (not shown), and typically multiple processor circuits, that provide control between the user input devices 204, the user input control system 110, and the display system 120. The input controller 118 also includes software programming instructions for implementing some or all of the methods described herein. While the input controller 118 is shown as a single block in the simplified schematic diagram of FIG. 1, the input controller 118 may include multiple data processing circuits (e.g., on the user input devices 204 and / or the user input control system 110). Any of a wide variety of centralized or distributed data processing architectures may be utilized. Moreover, one or more processing circuits may be implemented in a virtual machine. Similarly, the programming code may be implemented as multiple separate programs or subroutines, or may be integrated into numerous other aspects of the teleoperation system described herein. In various embodiments, the input controller 118 may support wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

[0016] The exemplary input controller 118 may include a servo controller that provides haptic force and / or torque feedback to the user input device 204 based on forces and torques sensed at the surgical instrument 104. The exemplary input controller 118 may include a servo controller that provides haptic force and / or torque feedback to the user input device 204 based on forces and torques sensed at the input device. Any suitable conventional or specialized servo controller may be used. The servo controller may be separate from or integrated with the instrument manipulator assembly 102. The servo controller may be separate from or integrated with the user input device 204. In the exemplary medical system, the exemplary servo controller and manipulator assembly 102 are provided as part of a robotic arm cart that is positioned adjacent to the patient 106. In the exemplary medical system, the exemplary servo controller and user input device 204 is positioned adjacent to a user to provide input at the user input device 204 .

[0017] For the purposes of this document, the surgical instrument 104 may be referred to as a "controlled device."

[0018] In the exemplary teleoperated surgical system 100, an input controller 118 controls at least one control device 104 (e.g., a “surgical instrument”) and may control the movement of one or more linkages 102-1 of one or more instrument manipulator assemblies 102. The exemplary instrument manipulator assembly 102 includes one or more motors coupled to control the movement of an end effector associated with the instrument 104. The exemplary instrument manipulator assembly 102 includes one or more motors coupled to control the movement of one or more end effectors coupled to the instrument 104. The linkages 102-1 may be referred to as a setup structure, which includes one or more links coupled to joints 102-2 that allow the setup structure to be positioned and held at a position and orientation in space. Motors coupled to control the movement of one or more end effectors of the instrument are further coupled to the surgical instrument 104 to advance the surgical instrument 104 into a natural or surgically created anatomical orifice, move the surgical instrument 104, and move the end effector of the instrument in multiple degrees of freedom, which may include three degrees of linear motion (e.g., x, y, and z linear motion) and three degrees of rotational motion (e.g., roll, pitch, yaw). The motors of the exemplary manipulator assembly 102 may be configured to actuate the effectors of the surgical instrument 104, such as an articulatable effector for grasping tissue within the jaws of a biopsy device, an effector for obtaining a tissue sample or dispensing a medication, or another effector for providing other treatments, such as, for example, U.S. Patent No. 6,671,581, entitled "Camera Referenced Control in a Minimally Invasive Surgical Apparatus," which is incorporated by reference, and which is described in more detail below.

[0019] In the exemplary teleoperated surgery system 100, for training purposes, the display system 120 may display a virtual environment that simulates a surgical site within a patient. The virtual environment may include various biological structures in addition to the surgical instruments 104. A user 112 manipulates the virtual instruments within the virtual environment to train, certify, or experiment with various skills or procedures without potentially harming a real patient.

[0020] In either a live or simulated surgical procedure, the display system 120 may be used to present a user interface to a user (e.g., user 112). In one embodiment, the display system 120 provides a 3D view, such as a stereo display. In another exemplary teleoperated surgical system, the display system 120 is used to project 3D images, for example, from a high-definition endoscopic camera. The user interface may be displayed as an overlay, for example, by using a semi-transparent interface, or may be displayed in place of the view of the surgical field.

[0021] FIG. 2A illustrates an exemplary user input control system 110. A user may sit at the user input control system 110 and have access to a display system 120, a user input device 204, and a footswitch panel 206. The footswitch panel 206 may act as a clutch, allowing the user to switch between performing various tasks, such as swapping between various surgical instruments or controlling a video or camera configuration. While seated at the user input control system 110, the user may rest their arms on armrests 208. When operating in live surgery, the display system 120 displays the surgical field as captured from a camera inserted through a small opening into the surgical site, sometimes called a portal or cannula. For training purposes, a simulated environment may be displayed on the display system 120, where the simulated environment may be a stereoscopic display of the surgical site and virtual control devices (e.g., surgical instruments). When the user moves the user input device 204, the virtual surgical instruments may move in a corresponding manner in the stereoscopic display.

[0022] FIG. 2B is an illustration showing an example user input device 204 operatively coupled to the user input control system 110. The example user input device 204 includes a gimbal mount 225, which includes an articulated arm portion including multiple links 227 connected to each other by pivot joints 229. A user grasps a finger loop 210 by positioning their thumb and index finger over a displaceable manual selector 212, such as a pincher push button. In the example user input device 204, the user's thumb and index finger are typically held on the displaceable manual selector 212 by a strap that is threaded through a slot to create the finger loop 210. The example selector 212 includes first and second grip buttons 503 a, 503 b, described below with reference to FIG. 5, which are spaced apart a distance so that the selector 212 can be grasped between the user's thumb and index finger, for example, with the user's thumb engaging one grip button and the user's index finger engaging the other grip button. At least one grip button is movable to decrease the displacement distance between the grip buttons in response to a user squeezing force applied to the grip buttons engaged therebetween. The joints 229 of the exemplary user input device 204 are operatively connected to motors or the like to provide, for example, force feedback, gravity compensation, etc. Additionally, a suitably positioned sensor, e.g., an encoder, potentiometer, etc., is located on each joint 229 of the exemplary user input device 204 to enable the joint position of the exemplary user input device 204 to be determined by the user input controller 118 to control the movement of one or more instruments operatively coupled to the user input device or to control the haptic feedback force applied to the one or more input devices 204.

[0023] The exemplary teleoperated surgical system 100 includes two user input devices 204, each with two finger loops 210 into which a user may insert the index finger and thumb of each hand. Each of the two user input devices 204 may control a surgical or virtual surgical instrument. The user may be provided with software or hardware mechanisms for switching between multiple instruments for one or both instrument motion controllers 204. For example, the user may be provided with three instruments, such as two forceps and a retractor. One or both of the forceps may be an energy instrument capable of cauterizing tissue. The user may first use the forceps with each instrument motion controller 204, then switch the user input device 204 to control the retractor to expose a portion of the surgical field, and then switch the user input device 204 back to the forceps to continue cutting, probing, or dissecting tissue.

[0024] While using the exemplary user input device 204, the user is provided with a full 3D range of motion (x, y, and z axes) along with rotational motion (roll, pitch, yaw) in addition to pinching motion with the index finger and thumb (or any two fingers inserted into the loop 210). Thus, by moving the appropriate user input device 204, the user can manipulate the corresponding surgical instrument through the full range of motion.

[0025] 2C illustrates an armrest 208 of user input control system 110, according to one embodiment. Armrest 208 may include one or more touch control devices, such as a touchscreen, soft buttons, mechanical buttons, etc. In the example shown in FIG. 2C, a single touchscreen 214 is shown through which a user may configure various video, audio, or other system settings.

[0026] (Graphical User Interface Overview) During operation, the user may be presented with user interfaces at various times. For example, a user interface may be presented to allow the user to select from a selection of training modules. As another example, a user interface may be provided to allow the user to configure various aspects of the operation of the user input control system 110. When a user has one or both hands operating the example user input device 204, it may be inconvenient to have to release the example user input device 204 and then operate another input mechanism, such as a touchscreen interface integrated into the armrest 208 of the user input control system 110.

[0027] FIG. 3 is an illustration showing a virtual surgical site displayed by the display system 120 operating in 3D mode. The virtual surgical site 300 may be displayed on the display system 120 and includes two virtual control devices 302. When operating in this mode, the user input device 204 can move in 3D within free space (within the boundaries of the virtual surgical site 300) to control the control devices 302. In the second mode, the user input device 204 is restricted to movement within a virtual surface, which may be planar or have a contour such as a gentle curve. The second mode can be used, for example, to present a graphical user interface including control elements (e.g., buttons, knobs, sliders, pull-down menus) that can be controlled using the user input device 204, for example, as a pointing and clicking device. For example, the second mode can be used to present 2D images, such as preoperative images. The second mode is useful for providing a working space for the user input device 204 that can be roughly aligned with the 2D virtual surface.

[0028] 4A shows a screen display 400 of a first exemplary graphical user interface displayed by display system 120 operating in 2D mode. First graphical user interface 400 is optionally displayed as an overlay to a surgical site view or as a standalone interface. A cursor 402 is displayed within first user interface 400 and is used to activate one or more user interface controls, such as a button, slider, option list, etc. Cursor 402 may be controlled by user input device 204. Using a servo control coupled to user input device 204, a user may be provided with haptic feedback to provide the sensation of touching first user interface 400. For example, when a user uses the input device 204 to effect virtual operation of a user interface control structure, such as selecting a virtual button, sliding a virtual slider control, or moving a virtual dial displayed in the user interface, motors of the user input device 204 coupled to the input device 204 may provide sensory feedback to the user, for example, by causing the input device 204 to vibrate, vibrate, exert a counterforce against the user's movement, or otherwise react to the actuation of the user interface control. FIG. 4A shows an exemplary login display 401 including a user-selectable menu pull-down arrow 404. A user can move a cursor 402 to overlay the menu selection arrow 402, after which the user can select the overlaid menu selection arrow 402 by actuating (e.g., squeezing) the displaceable manual selector 212 to provide a click event that causes a surgical system operation to occur, such as displaying a pull-down menu (not shown) in the display system 120 or energizing an electrosurgical instrument (not shown) in the surgical site 300.

[0029] As used herein, a "click event" may refer to a displacement of the displaceable manual selector 212 relative to the user input device 204 that causes the controller 118 to send a signal to the display system to select or actuate a selectable element within a user interface display. As used herein, a "click event" may refer to a displacement of the displaceable manual selector 212 relative to the user input device 204 that causes the controller 118 to send a signal to one or more motors of the manipulator assembly to cause movement of one or more of a real or virtual instrument, instrument end effector, or manipulator linkage. In the exemplary teleoperated surgical system 100, a click event causes selection or actuation of a selectable element within a user interface display during 2D operation of the system 100, and a click event causes movement of a real or virtual component of the manipulator assembly during 3D mode operation of the system 100.

[0030] For example, during 2D mode operation, a user can use a cursor to select a menu item from a pull-down menu. The login screen 401 also includes a keypad with a number of control elements (i.e., virtual buttons) 406. The user can move the cursor 402 to overlay a keypad control element, after which the user can select an overlaid keypad button by squeezing the displaceable manual selector 212 to provide a click event that causes the display of the selected element in the display area 408. In the exemplary teleoperated surgical system 100, the keyboard control panel can be used to select, for example, an energy level of an electrosurgical instrument.

[0031] 4B illustrates an exemplary second graphical user interface screen display 421 including a plurality of control element menus 420 and an instrument manipulator assembly image 422 depicting an exemplary instrument manipulator assembly 102. In the exemplary teleoperated surgical system 100, a user can select a control element from within the control element menu 420 using the user input device 204 in a 2D mode of operation, after which a different menu (not shown) corresponding to the selected menu element is displayed.

[0032] FIG. 4C illustrates a screen display 421 of the exemplary second graphical user interface of FIG. 4B with an endowrist manipulation menu 425 displayed overlaying an instrument manipulator assembly image 422 in response to a user selection of the endowrist manipulation control element of FIGS. 4B-4C.

[0033] 4D shows the example second graphical user interface screen display 421 of FIGS. 4B-4C with a cursor mode menu 427 displayed overlaying an instrument manipulator assembly image 422 in response to a user selection of the cursor mode control element of FIGS. 4B-4D. It will be appreciated that more or fewer screens may be used in the second user interface 400.

[0034] The illustrative first and second graphical user interface displays 400, 402 are shown as 2D interfaces. Thus, when a user is controlling a cursor within the graphical user interface, the user input device 204 is constrained to a 2D region. In contrast, when in a surgical simulation mode (e.g., the first mode), the user input device 204 is allowed full or near-full 3D freedom of movement. However, in a graphical user interface mode (e.g., the second mode), the user input device 204 is constrained to 2D movement. The 2D movement may be constrained to a planar region or may be constrained to a region with a gentle curvature, such as a gentle convex curvature. The 2D region constraining the movement of the user input device 204 can be oriented in space at approximately the same angle as the user's hand and user input device 204 are displayed on the display system 202. Such a correlation may help the user orient their hand in 3D space relative to the displayed graphical user interface.

[0035] An example is shown in FIG. 5 , which shows a viewing plane 502 and a haptic plane 504. The viewing plane 502 represents a planar area in which a graphical user interface image is presented to a user by a user within a display system 202 capable of displaying, for example, the exemplary first and second graphical user interface screen displays 400, 421 of FIGS. 4A-4D . Cartesian coordinate locations within the viewing plane 502 correspond to Cartesian coordinate locations within the haptic plane 504. A user can select a control element displayed at a Cartesian coordinate location within the viewing plane 502 by moving one of the user input devices 204 to the corresponding Cartesian coordinate location within the haptic plane and clicking at the corresponding location. More specifically, the movement of a cursor within the graphical user interface displayed in the viewing plane 502 follows the movement of the user input device 204 within the haptic plane 504. Thus, as the user moves the user input device 204 within the 2D tactile region 504, a cursor correspondingly moves and is displayed within one of the first and second graphical user interface screen displays 400, 421 displayed within the visual image region 502. In the second mode, movement of the user's hand causes corresponding movement of the user input device 204 within the 2D region 504, which causes corresponding movement of a cursor image within the viewing region 502. To select a desired control element displayed in the visual image region 502, the user can engage the user input device 204 with their hand and move their hand to cause corresponding action of the user input device 204 to cause corresponding movement of the cursor to visually align the cursor with the desired control element. With the cursor aligned, the user can select the desired control element by, for example, applying a finger movement to apply movement to the displaceable manual selector 212 to affect a click-to-select (“click”) user interface action. In response to the click, the processor circuitry implements the action associated with the selected control element.

[0036] The user input device 204 is constrained within the 2D tactile plane region 504. If the user attempts to move the user input device 204 "up" or "down" relative to the z-axis of the tactile plane 504, the user may encounter resistance from such movement. If the user changes the orientation of the view plane 502, for example, using a display configuration setting, the tactile plane 504 may adjust to maintain an orientation approximately parallel to the view plane 502. In various embodiments, the tactile plane may be oriented with a fixed or dynamic angular offset relative to the view plane. Alternatively, the tactile plane may be oriented with a fixed or dynamic angular offset relative to the ground. The user may also change constraints, such as the position or orientation of the tactile plane.

[0037] The one or more processor circuits are configured to scale the movement of the user input device 204 when imparting a corresponding movement to a controlled device in a first (3D) mode and to impart a corresponding movement to a cursor in a second (2D) mode. In the first mode, the scaling allows a user to perform complex medical procedures with greater ease than traditional open surgery. The scaling includes scaling the commanded movement of the user input device 204 according to a scale factor before imparting the corresponding movement to the controlled device or cursor. The scaling takes into account changes in velocity and position of the user input device 204 and converts these into corresponding scaled changes in the position of the controlled device or cursor. The scale factor is adjustable and can be different during operation in the first and second modes. In the second mode, for example, the scale factor can be 1 (i.e., no scaling). U.S. Patent No. 7,843,158, entitled "Medical Robotic System Adapted to Inhibit Motions Resulting in Excessive End Effector Forces," which is incorporated by reference, contains further information regarding scaling.

[0038] (User input device with manual click selector) FIG. 6A is a perspective view showing details of an exemplary user input device 204 that acts as a mount for a displaceable manual click selector 212. The exemplary user input device 204 is attached to a gimbal mount assembly 225 and coupled to the input controller 118 of FIG. 2A. The gimbal mount 225 includes multiple controller motion sensors 529 (only one shown) for sensing overall controller 225 motion. The exemplary user input device 204 may include a controller motion sensor (first sensor) 529 for sensing controller motion in multiple degrees of freedom. The exemplary user input device 204 includes an exemplary displaceable manual selector 212 that includes a pinch mount member configured as an elongated handle 530 that includes a longitudinal axis 531. The exemplary manual selector 212 is integrally formed with the exemplary user input device 204 such that the manual selector 212 moves integrally with 2D movement of the user input device 204 as well as such that the manual selector 212 moves integrally with 3D movement of the user input device 204. The exemplary manual selector 212 includes first and second articulatable pincher grip buttons 530a, 530b attached to a handle 530. The handle 530 acts as a mounting member for attaching the grip buttons 530a, 530b of the manual selector 212. The first and second grip buttons 530a, 530b of the manual selector 212 are angled upright from opposite sides of the handle 530.

[0039] The first and second grip buttons 530a, 530b are fixed to the handle 530 for articulation relative thereto. The first and second grip buttons 530a, 530b are tilted relative to the handle 530 with their distal ends spaced closer together and their proximal ends spaced further apart. As used herein, the term "proximal" refers to a location closer to the manipulator support structure and farther from the patient's anatomy, and the term "distal" refers to a location farther from the manipulator support structure and closer to the patient. The first and second grip buttons 530a, 530b have an angle α between their distal ends that can vary according to the force a user applies thereto. In the exemplary teleoperated surgical system 100, the grip buttons 530a, 530b are in a neutral position when the user is not applying a pinch force to move the grip buttons 530a, 530b closer together. In the exemplary user input device 204, the grip buttons 530a, 530b are maximally displaced from one another in the neutral position. In the exemplary user input device 204, the angle α is an acute angle when the grip members are in the neutral position. In the exemplary user input device 204, in the neutral position, one or more motors 545 apply a tactile counterforce to the grip buttons 530a, 530b that resists a user-applied force that moves the grip buttons 530a, 530b toward one another, and the user must overcome this tactile counterforce to move the grip buttons 530a, 530b toward one another. In an alternative exemplary user input device 204, in the neutral position, a spring-like biasing member (not shown) provides a reaction force to resist the displacement of the grip buttons 530a, 530b toward one another, and the user must overcome this reaction force to move the grip buttons 530a, 530b toward one another.

[0040] In the exemplary user input device 204, the one or more motors 545 move the grip buttons 530a, 530b to a neutral position under (i) a first condition when the user does not apply force to the grip buttons 530a, 530b, and (ii) a second condition when the grip buttons 530a, 530b have a displacement position distance between them that meets a predetermined threshold. In an alternative exemplary user input device 204, a biasing member (not shown), such as a spring, moves the grip buttons 530a, 530b to a neutral position under (i) a first condition when the user does not apply force to the grip buttons 530a, 530b, and (ii) a second condition when the grip buttons 530a, 530b have a displacement position distance between them that meets a predetermined threshold. In the exemplary user input device 204, the second condition occurs when the grip buttons 530a, 530b reach a second grip threshold (T G A click event occurs when the grip buttons 530a, 530b have a displacement position distance between them that is less than the second grip threshold (T G 2) A click event greater than the distance initiates an action, such as a display action in the display system 120 or an instrument action within the surgical site 300, upon or before reaching a displacement position. Thus, an action has been initiated in response to a decrease in the displacement distance between the grip buttons 530a, 530b by the time the displacement distance position is in the first control state 776 of graph 770, described below. For example, a control element in the control element menu 420 in the screen display 421 of the second graphical user interface of FIGS. 4B-4C is selected, and a screen display corresponding to the selected control element is displayed. For example, if the click event is the selection of an endowrist manipulation control element, the endowrist manipulation menu 425 of FIG. 4C is displayed.

[0041] In the exemplary user input device 204, a user can apply forces to each grip button 530 a, 530 b in respective directions toward the handle 530 to reduce displacement therebetween until the grip members abut the handle mount 530, which acts as a stop surface, after which there is no displacement between the grip buttons and the handle mount. More specifically, according to some embodiments, the first and second grip buttons 530 a, 530 b are fixed to the handle for pivoting about a master pivot axis 536. One or more motors 545 or other biasing members urge the grip buttons 530 a, 530 b apart. In the exemplary user input device 204, the one or more motors 545 are configured to apply a variable tactile force in a radially outward direction from the mount member 530 toward the grip buttons 530 a, 530 b during user-applied radially inward movement of the grip buttons 530 a, 530 b toward the handle 530. In the exemplary user input device 204, the one or more motors 545 may include a single motor (not shown) capable of applying a tactile force to both grip buttons 530a, 530b. In an alternative exemplary user input device 204, the one or more motors 545 may include a first motor (not shown) that applies a tactile force to the first grip button 530a and a second motor (not shown) that applies a tactile force to the second grip button 530b. The handle 530 includes one or more displacement sensors (second sensors) 547, such as Hall-effect devices, that sense movement of the grip buttons 530a, 530b along the first path and their displacement from a neutral displacement position. Finger loops (not shown) may be attached to the handle to prevent slippage from the grip buttons. A wide variety of grip button configurations may be used within the scope of the present disclosure, including, for example, any surgical instrument handle that optionally includes rigid or flexible loops for the thumb and / or fingers.The control relationship between the grip button and the controlled device is described in more detail in U.S. Pat. No. 6,594,552, entitled "Grip Strength with Tactile Feedback for Robotic Surgery," the complete disclosure of which is expressly incorporated by reference.

[0042] In a first (3D) mode of operation, the user input device 204 and grip buttons 530a, 530b are operatively coupled through a kinematic arrangement to control movement of the controlled device 104, for example, in response to 3D movement of the user input device 204 and movement of the grip buttons 530a, 530b about the master pivot axis 536. In a second (2D) mode of operation, the controller 204 and grip buttons 530a, 530b are operatively coupled to control 2D cursor movement within the view plane 502 and control element selection within the view plane 502.

[0043] In the exemplary teleoperated surgical system 100, one or more motors are selectably configured to apply a variable tactile force to the grip buttons 530a, 530b in a radially outward direction from the handle 530. A user can use their fingers to apply force to the grip buttons 530a, 530b, urging them toward the handle 530 located between them and toward each other to move them closer together. As described below, in a second (2D) mode, a user can use the manual selector 212 to apply a finger force radially inward toward the handle 530 to overcome neutral resistance and a motor-controlled tactile force and press the grip buttons 530a, 530b toward each other, resulting in a click event to select a graphical user interface control element. As described below, a variable tactile force is applied to the grip buttons 530a, 530b of the manual selector 212 to provide tactile feedback indicating when a click event occurs within the viewing plane 502.

[0044] The exemplary user input device 204 includes a four-degree-of-freedom gimbal mount 225 to allow a user to rotate the actuatable mount member handle 530 about three axes: axis 534a, axis 534b, and axis 534c. During operation in the first (3D) mode, a controlled device, physical or virtual, such as an instrument 104, follows the 3D movement of the user input device 204. During operation in the second (2D) mode, a controlled user interface element, such as a cursor within the 2D viewing area 502, follows the 2D movement of the user input device 204 within the 2D area 504.

[0045] More specifically, the handle mount 530 portion of the user input device 204 is coupled to a first elbow-shaped link 514 by a first pivot joint 536. The first link 532 is coupled to a second elbow-shaped link 537 by a pivot joint 520. The second link 537 is pivotally coupled to a third elbow-shaped link 538 by a pivot joint 524. In some embodiments, motors in the arm 538 and gimbal 225 can actively apply position and orientation forces to mount the member handle 530, thereby providing tactile feedback to the surgeon. The gimbal 225 includes links 532, 537, and 538. The gimbal 225 is mounted to a platform 540 for rotation about axis 534d, and the links 532, 537, and 538 define additional axes 534a, 534b, and 534c. Handle 530 is attached to gimbal 225 by an actively actuated joint for movement about axis 534d. Gimbal 225 therefore provides four actuated orientational degrees of freedom, including a redundant orientational degree of freedom. The gimbal 225, arm 538, and drive motors for these joints are described in detail in U.S. Patent No. 6,714,839, entitled "Master Having Redundant Degrees of Freedom," the complete disclosure of which is expressly incorporated by reference.

[0046] 6B is a functional block diagram illustrating a control system 680 that controls receiving user input at a user input device 204, including a manual selector 212, and provides haptic feedback at the user input device 204. An input controller 118 is coupled to control the manipulator assembly 102 and coupled to control the display system 120. The user input device 204 is configured to receive a first user input action 652, such as a user's hand movement, that imparts movement to the user input device 204 as a whole. The manual selector 212, which is movably mounted relative to the user input device 204, is configured to receive a second user input 654, such as a user finger movement, that imparts movement to the manual selector 212 relative to the user input device 204. One or more first sensors 547a are configured to sense movement of the user input device 204 as a whole and provide a corresponding first sensor signal (S1) 549a to the input controller 118. The one or more second sensors 547b are configured to sense movement of the manual selector 212 relative to the user input device 204 and provide a corresponding second sensor signal (S2) 549b to the input controller 118. The one or more motors 545 are coupled to receive motor control signals 551 from the input controller 118 and to apply a haptic feedback force 553 to the manual selector 212. The input controller 118 provides motor control signals (M C ) 551 to cause one or more motors to apply a haptic feedback force 553 to the manual selector 212.

[0047] The user selects the mode by the mode selection signal (S M) 555 is provided to allow the input controller 118 to operate in either a 3D mode or a 2D mode. In the 3D mode, the displaceable manual selector 212 can move in 3D in coordination with the movement of the user input device 204 in response to a first user input action 652, and can be displaced relative to the user input device 204 in response to a second user input. In the 2D mode, the displaceable manual selector 212 can move in 2D in coordination with the user input device 204 in response to a first user input, and can be displaced relative to the user input device 204 in response to a second user input action 654.

[0048] In the 3D mode, the input controller 118 controls the manipulator assembly 102, which includes one or more motors (not shown) that control the movement of the instrument 104, the instrument end effector, and one or more manipulator links, in response to motion-imparting user input that causes one or more of the displacement of the displaceable manual selector 612 and the entire user input device 204 relative to the user input device 204 to which the selector is attached. In the 2D mode, the input controller 118 controls the display system 120, which includes a screen display of a graphical user interface including one or more control elements, such as a menu, cursor, slider, knob, or button, in response to user input that causes one or more of the displacement of the displaceable manual selector 612 and the entire user input device 204 relative to the user input device 204.

[0049] The first displacement sensor 547a is coupled to sense the displacement of the entire user input device 204 and to provide a corresponding first sensor signal 549a to the input controller 118 that is indicative of the displacement of the user input device 204. The second sensor 547b is coupled to sense the displacement of the manual selector 212 relative to the user input device 204 and to provide a corresponding second sensor signal 549b to the input controller 118 that is indicative of the displacement of the manual selector 212 relative to the user input device 204 to which the selector 212 is movably mounted. The input controller 118 includes one or more processor circuits configured with executable instructions for providing control signals to control the manipulator assembly 102 in response to the first and second sensor signals when in the 3D mode, and for providing control signals to control the display system 120 in response to one or more of the first and second sensor signals when in the 2D mode. The input controller 118 is further configured to provide motor control signals 553 to one or more motors 545 to apply a haptic feedback force FH to the manual selector 212 based on one or more of the first and second sensor signals 549a, 549b.

[0050] (Haptic feedback to indicate click events) FIG. 7 is an illustration showing a first control function curve 720 representing tactile force versus displacement of the displaceable manual selector 212 position during a click event time-aligned with a sequence of grip button displacements and a sequence of hand formations. The example user input device 204 of FIG. 6A includes a handle 530 that serves as a mount for a pair of opposing grip buttons 530 a, 530 b. As described above, alternative example manual selectors include a gravity-balanced arm, a joystick, a trackball, a glove, a trigger grip, a twistable knob, a twistable grip, a slider, a lever, a push button, and the like. A user can cause a click event by applying a second user input action that causes a predetermined displacement of the displaceable manual selector 212. For example, for the manual selector 212 of FIG. 6A, a user can cause a click event by decreasing the displacement of the grip buttons 530 a, 530 b from a neutral position displacement to a click event completion position displacement distance. 6A , each grip button is equally displaced from the handle mount 530 of the user input device throughout movement of the grip buttons 530 a, 530 b between the open and closed positions, but the displacement distance between the grip buttons decreases as the grip buttons 530 a, 530 b move from the open position to the closed position and increases as the grip buttons 530 a, 530 b move from the closed position to the open position. Moreover, in the example user input device 204 of FIG. 6A , each individual grip button 530 a, 530 b has a displacement distance between it and the handle mount 530 of the user input device, which displacement distance decreases as the grip button moves from the neutral displacement position to the click event completion displacement position and increases as the grip button moves from the click event displacement position to the neutral displacement position.

[0051] Figure 7 represents an example sequence of displacement positions of grip buttons 530a, 530b of example manual selector 212 of Figure 6A during pincher closure resulting from a second user input action 646 caused by a user's finger during a click event, via a sequence of user finger positions 752, 754, 756. Figure 7 also shows a corresponding example sequence of displacement positions 762, 764, 768, 770 of grip buttons 530a, 530b of example manual selector 212 of Figure 6A. For simplicity and clarity of explanation, exemplary user's fingers 758, 760 and sequence of finger positions 752, 754, 756 are shown separated from the respective grip buttons 530a, 530b they contact and the corresponding sequence of grip button displacement positions 762, 764, 768, 770, although it will be understood that in actual use, each finger 758, 760 will contact a respective grip button 530a, 530b. For example, in actual exemplary use, cursor finger 758 will contact first grip button 530a and thumb 760 will contact second grip button 530b.

[0052] 7 shows an example sequence of displacement positions 762, 764, 768, 770 of the grip buttons 530a, 530b of FIG. 6A, each position corresponding to a different movement distance relative to the user input device handle 530 moved by the grip button during a click event. Thus, at the example displacement distance indicated by grip button displacement position 762 and corresponding finger position 752, the grip button has moved a first (smallest) distance relative to the user input device handle 530 in the example sequence. At the example displacement distance indicated by grip button displacement position 764 and corresponding finger position 754, the grip button has moved a second distance relative to the user input device handle 530 that is greater than the first distance. At the example displacement distance indicated by grip button displacement position 768 and corresponding finger position 756, the grip button has moved a third distance relative to the user input device handle 530 that is greater than the sum of the first and second distances. At the exemplary displacement distance indicated by grip button displacement position 770 and corresponding finger position 756, the grip button has moved a fourth distance relative to the handle 530 of the user input device, which is greater than the first, second, and third distances.

[0053] 7 also shows a corresponding sequence of finger positions of a hand 759 that imparts a second user input action 654 to the grips 530a, 530b. The finger position sequence begins with a fully open finger position 752, followed by a partially closed / partially open finger position 754, followed by a substantially closed finger position 756. A sequence of grip button displacement positions is shown about the handle mount 530. The grip button sequence begins with a fully open grip button displacement position 762, followed by a partially closed / partially open grip button position 764, followed by a nearly fully closed grip button position 768, followed by a fully closed grip button position 770.

[0054] 7 includes an example curve representing a control function curve 770 that controls a combination of click event triggering and haptic feedback control functions implemented using the manual selector 212, the second sensor 547b, the input controller 118, one or more motors 545, and a stop surface, such as the handle 530 of the input device. The input controller 118 includes one or more processor circuits programmed with executable instructions for implementing the control function curve 770. The control function 770 controls the triggering of click events and the haptic feedback force applied in response to a second user input action 654 that displaces the manual selector 212. More specifically, the input controller 118 is configured to perform the control function in response to a second sensor signal S2 provided by the one or more second sensors to control the triggering of click events at the display system 120 or the manipulator assembly 102 and cause the one or more motors 545 to apply a haptic force to the manual selector 212. According to control function 770, the user must displace the manual selector 212 at least a specified displacement distance to trigger a click event. Additionally, according to control function 770, a haptic feedback force is applied to the manual selector 212 while the user is displacing the manual selector 212. The haptic feedback force responds to the user's increasing displacement of the manual selector 212 to indicate both the build-up and triggering of a click event to the user. The stop surface applies a sudden reaction force that stops further displacement of the manual selector 212 after a specified amount of further displacement following triggering of a click event.

[0055] The control function curve 770 has multiple states. A first control state 772 occurs when the displacement of the manual selector 212 exceeds a first threshold displacement T D1 For the user input device 204 of FIG. 6A, in the first control state 772, the grip buttons 530a, 530b are in a neutral or rest state where the distance is less than the maximum displacement at the neutral position and the first threshold distance T D1It is displaced from the handle 530 by a distance between them. In an exemplary teleoperation surgical system 100, in the first control state 772, the elastic member applies a holding force to the manual selector 212 to bias it to the neutral position. In the case of the user input device 204 of FIG. 6A, in the first control state 772, a biasing member such as a spring member biases the grip buttons 530a, 530b that are displaced maximally from each other and from the handle 530, while the input controller 118 causes a constant zero force to be applied to one or more motors. In an alternative exemplary teleoperation surgical system 100, in the first control state, the input controller 118 causes one or more motors 545 to apply an elastic holding force to the manual selector 212 to bias it to the neutral displacement position. In an alternative first control state 772, one or more motors 545 are controlled to generate a holding force like a spring to maintain the manual selector 212 in the neutral position. In an alternative first control state 772, one or more motors apply a force that the user overcomes, causing it to displace the selector 212 by less than the first displacement threshold distance, and then, in response to the user removing that force, provide a force to return the manual selector 212 to the neutral position. While in the first control state 772, the user may impart movement to the manual selector such that the displacement is the first threshold distance T D1 and, conditional on that, may displace the manual selector somewhat from the neutral position. The magnitude and direction of such a holding force on the user's finger for the exemplary user input device of FIG. 6A are represented by the arrows 752a, 752b. The first and second directions extend radially outward from the longitudinal axis 531 of the handle 530.

[0056] The second control state 774 is a haptic feedback force increasing state in which the haptic force increases at a first rate relative to increasing displacement of the manual selector 212. The second control state 774 includes a displacement of the manual selector 212 that meets the first threshold distance TD1 but does not yet meet the second threshold distance TD2. During the second control state 774, the one or more second sensors 547b sense an increase in the displacement of the manual selector 212 from the neutral position and send a corresponding second sensor signal value S2 to the input controller 118 to report the increase in displacement. The input controller 118 then generates motor control signals 553 to cause the one or more motors 545 to apply a haptic force to the manual selector 212 that increases at a first rate relative to increasing displacement of the manual selector 212 from the neutral position. The first rate can be linear or nonlinear, provided that the user has time to recognize and react to the haptic sensation of the increasing contact force by tracking or aborting an impending click event. In the example user input device 204 of FIG. 6A , the grip buttons 530 a, 530 b are displaced by increasingly smaller amounts from each other as they are increasingly displaced from their more widely spaced neutral positions. The accumulation of haptic force, which increases in magnitude at a first rate relative to the increasing displacement of the manual selector 212 during the second control state 774, provides a tactile indication or warning to the user that a click event is imminent. That is, the greater the magnitude, the closer the current displacement is to causing a click event to occur. The second rate is selected so that the user has time to react in response to the indication of an impending click event, so that the user can make a conscious decision, for example, to continue the displacement and proceed with the click event or to stop the displacement and abort the click event. Thus, the increasing haptic feedback during the second control state 774 alerts the user that a click event is increasingly imminent as the displacement of the manual selector 212 increases.

[0057] In the example user input device 204 of FIG. 6A , the magnitude of the second feedback force during the second control state is represented by the length of the arrows 754 a, 754 b. Grip button 530 a applies a force 754 a to index finger 758 in a first direction, and grip button 530 b applies a force 754 b to thumb 760 in a second direction opposite the first direction. The first and second directions extend radially outward from the longitudinal axis 531 of the handle 530. It will be appreciated that to move grip buttons 530 a, 530 b closer to one another during the second control state 774, the user's fingers 758, 760 apply respective displacement forces in directions opposite the haptic force directions 754 a, 754 b. The user-applied forces have a magnitude large enough to overcome the haptic forces 754 a, 754 b. It will also be understood that the magnitude of the tactile forces 754a, 754b applied by the respective grip buttons 530a, 530b at any example instance in the second control state 774 is greater than the forces 752a, 752b during the sustained force during the first control state 772, as represented by the lengths of the arrows 754a, 754b being greater than the lengths of the arrows 752a, 752b.

[0058] The third control state 776 is when the displacement of the manual selector 212 exceeds the second threshold distance T D2 The second sensor 547b detects a click event trigger state that occurs when the displacement of the manual selector 212 is greater than a second threshold displacement T D2 a second sensor signal indicating when the input voltage Vcc reaches the input controller 118;

[0059] The example input controller 118, when operating in 2D mode, detects when the manual sensor 212 detects a second threshold displacement T D2 , transmits a click event trigger signal to the display system 120 to, for example, cause selection of a visual UI control element overlaid by the cursor 402 in the control element menu 420. The example input controller 118, when operating in 3D mode, is configured to operate when the manual sensor 212 reaches a second threshold displacement T D2In response to reaching the click event trigger signal is sent to the manipulator assembly 102 causing actuation of one or more motors (not shown) that actuate, for example, links or instrument end effectors.

[0060] Additionally, the example input controller 118 may generate a motor control signal M in response to the manual sensor 212 reaching a second threshold displacement TD2 when operating in either 2D or 3D mode. C on line 551 to cause one or more motors 545 to apply a step-down haptic feedback force to the manual selector 212. The step-down haptic feedback force decreases the haptic feedback force from a peak value at the instant the displacement of the manual selector 212 crosses the second threshold displacement TD2 to a level that matches or nearly matches the force applied during the first control state 772. More specifically, the haptic force decreases at a second rate relative to increasing displacement of the manual selector 212. The magnitude of the second rate is greater than the magnitude of the first rate. In the exemplary system, the second rate has a magnitude selected to provide the user with a substantially instantaneous haptic indication that a click event has been triggered. In the exemplary user input device 204 of FIG. 6A , the magnitude of the third feedback force during the third control state 776 is represented by the length of arrows 756a, 756b. Grip button 530a applies a force 764a to index finger 758 in a first direction, and grip button 530b applies a force 756b to thumb 760 in a second direction opposite the first direction. In the exemplary system, the control function returns to the first control state 772 in response to one or more sensors 247b providing a signal indicating when the displacement returns to the neutral displacement distance.

[0061] In the exemplary user input device 204, the selector 212 selects the third threshold displacement T D36A , the manual selector 212 includes displaceable grip buttons 530 a, 530 b, and the handle 530 acts as the stop surface. The impact of the manual selector 212 against the stop surface provides a sudden reaction force that provides the user with an additional tactile sensation indicating that the click event is complete and that the manual selector 212 is ready to return to the second control state 772, which is a reset or neutral state. Thus, the stop surface 530 provides tactile feedback when the control function 700 has actually returned to the first control state 772, and the one or more motors 545 provide a maintaining force in the first control state.

[0062] 8 is an exemplary flow diagram illustrating a control process 800 according to the control functions of FIG. 7 for controlling the provision of forces and the triggering of click events based on the displacement of the manual selector 212 relative to the user input device 204. The input controller 118 includes one or more processor circuits programmed with executable instructions for implementing the control process 800. In operation 802 during the first control state 772, the input controller 118 controls the one or more motors 545 to apply a neutral hold force (F HN ), while one or more second sensors 547b detect when the user moves the manual selector 212 at least a first threshold distance T D1 A first threshold determination operation 804 monitors one or more second sensors 547b to determine whether the displacement of the manual selector is greater than or equal to the first threshold displacement T D16A , the first threshold determination operation 804 determines when the displacement position of the grip buttons 530 a, 530 b from the neutral position meets a first displacement threshold. The example input controller 118 provides motor control signals that cause one or more motors 545 to generate a zero magnitude haptic force while the selector 212 is displaced by less than the first threshold tactile distance. A biasing member, such as a spring, maintains the selector 212 in the neutral position while the selector 212 is displaced by less than the first threshold tactile distance.

[0063] The control process begins when the displacement reaches a first threshold displacement distance T D1 , transitions to a second control state 774 in response to the displacement satisfying the first threshold displacement distance. Process 800 remains in first control state 772 while the displacement has not yet satisfied the first threshold displacement distance. At operation 806 during second control state 774, input controller 118 causes one or more motors 545 to apply a haptic feedback force to manual selector 212 that increases at a first rate that is a function of the increasing displacement of the grip button from the neutral position. In an exemplary system, operation 806 may set the rate of accumulation of the haptic feedback force to vary as a linear or non-linear function of displacement, for example. A second threshold determination operation 808 monitors one or more second sensors 547b to determine whether the displacement of the manual selector meets a second threshold displacement T D2 In the example manual selector 212 of FIG. 6A, a second threshold determination operation 808 determines when the displacement position of the grip buttons 530a, 530b from the neutral position of the grip buttons 530a, 530b satisfies a second displacement threshold distance.

[0064] The control process continues until the displacement reaches a second threshold displacement distance T D2, transitioning to a third control state 776 in response to the displacement satisfying the second threshold displacement distance. The process 800 remains in the second control state 774 while the displacement does not yet satisfy the second threshold displacement distance. At operation 809, the input controller 118 sends a control signal to the display system 120 to trigger the occurrence of a click event. At operation 810, during the third control state 776, the input controller 118 causes the one or more motors 545 to apply a haptic feedback force to the manual selector 212 at a second rate that decreases from a peak value when transitioning from the second control state to the third control state to a level that matches or approximately matches the force applied during the first control state 772. The magnitude of the second rate is greater than the magnitude of the first rate. During a determining operation 812, the input controller 118 monitors sensor signals provided by the one or more sensors 547b to determine whether the displacement of the selector 212 meets the first threshold displacement distance T D1 (e.g., the first threshold displacement T D1 In response to the displacement satisfying the first displacement distance, control process 800 transitions back to operation 802.

[0065] The change in haptic force between the first, second, and third control states provides the user with a tactile indication of the status of the click event. During the second control state 774, a buildup of haptic force indicates to the user that a trigger event is increasingly imminent. During the third control state 776, a rapid decrease in haptic force indicates the triggering of a click event. During the first control state 772, a sustained force maintains hand-held reinforcements to a user that a tactile event has occurred. A hard stop at the physical stop surface 530 indicates to the user that the manual selector 212 has returned to the neutral first control state 772.

[0066] (Isolating cursor movement from push button movement) During operation of the display system 120 in the second (2D) mode, the digits of the user's hand can be used to select a control element displayed on the viewing plane 502. During the second mode of operation, the input controller 118 causes movement of the cursor 402 in the graphical user interface displayed on the viewing plane 502 to follow movement of the user input device 204 in the tactile plane 504. Cartesian coordinate locations in the viewing plane 502 correspond to Cartesian coordinate locations in the tactile plane 504. A user selects a control element by first moving the user input device 204 to a coordinate location in the tactile plane 504 that corresponds to the control element coordinate location in the visual plane 502 to visually align the cursor 402 with the control element, and second applying an actuation motion to the manual selector 212. The user's finger can apply a motion to the manual selector 212 to increase its displacement from a neutral position and trigger a click event, for example, as described above with reference to FIGS. 7-8 .

[0067] The movement of a user's fingers can affect the movement of the rest of the user's hand. Referring to the example manual selector 212 of FIG. 6A , when a user applies a closing force to the grip buttons 530a, 530b to perform a click, the user's finger movement can cause a corresponding movement of the controller 204 to which the grip buttons 530a, 530b are attached. The user-applied movement of the grip buttons 530a, 530b can cause a slight unintended change, or jump, in the position of the controller 204. Such an unintended change in the position of the controller 204 in the tactile plane 504 can cause a corresponding unintended change in the position of the cursor 402 in the viewing plane 502. As a result, for example, when the grip buttons 530a, 530b are used to click on a control element in a graphical user interface menu, the cursor 402 may tend to jump on the control element during the click process. Thus, a user suddenly activating a click event can cause a sudden, unintended jump in cursor movement, causing the cursor to misalign with the target control element, which can result in the selection of an incorrect target. This can be particularly frustrating if the targeted clickable control element is not large and the user misses the targeted element.

[0068] A missed target can be particularly harmful in a surgical environment. For example, the viewing plane 502 can display a preoperative MRI or CT scan instead of or in addition to a control element. The target element to which the surgeon user intends to visually align the cursor for selection may be a delicate anatomical structure, such as a nerve or blood vessel. The surgeon may intend to expand the view of the structure or transition to another anatomical structure represented in the MRI or CT scan that is located above or below the targeted structure. Thus, precise alignment between the cursor and the target element may be required.

[0069] 6B , the example input controller 118 is configured to calculate a motion transformation function that provides first and second cursor motion input signals, which are cursor motion control signals indicative of the movement of the user input device 204 sensed by the one or more first sensors 547a, to the display system 120 on line 561. The display system displays the movement of the cursor 402 in the graphical user interface 400 following the movement of the user input device 204.

[0070] In the exemplary user input device 204, a click event typically involves a rapid movement of the manual selector 212. The controller system 118 is configured with instructions to coordinate a transformation function during the occurrence of a user motion input to the manual selector 212 attached to the user input device 204 to execute a click event, where such user motion input may also affect the movement of the user input device. Figure 9 is a diagram illustrating the configuration of the input controller 118 to implement a first transformation F1 in the absence of a click event. Figure 10 is a diagram illustrating the configuration of the input controller 118 to perform a second transformation F2 in the presence of a click event.

[0071] 9 , based on the input controller 118 determining that the movement of the user's digits (e.g., fingers and / or thumb) and / or overall hand movement indicates that the user is not in the process of causing activation of a click event signal, the input controller 118 applies a first transformation function F1 that causes the movement of the cursor 402 in the viewing plane 502 to track the movement of the user input device 204 in the tactile plane 504. More specifically, the example input controller 118 is configured to match the Cartesian coordinates of the cursor 402 in the viewing plane 502 with the Cartesian coordinates of the user input device 204 in the tactile plane 504. Moreover, in the absence of a click event, the example input controller 118 can scale the movement of the user input device 204 to the movement of the cursor 402 by a specified amount. For example, the input controller 118 can be calibrated for a predetermined one-to-one (1:1) movement ratio of user input device movement to cursor movement, where the distance the user input device 204 moves in the tactile plane 504 exactly matches the distance the cursor 402 moves in the viewing plane 502. Alternatively, for example, the input controller 118 can be calibrated for a predetermined two-to-one (2:1) movement ratio, where the distance the user input device 204 moves in the tactile plane 504 is exactly twice the distance the cursor 402 moves in the viewing plane 502.

[0072] 10 , based on input controller 118 determining that the user's digit (e.g., finger and / or thumb) movement and / or overall hand movement indicates that the user is in the process of initiating a click event, input controller 118 applies a second transformation function F2 that reduces the rate of cursor movement in response to movement of the user input device during the time interval in which the user is initiating the click event. The reduced movement rate causes a decrease in cursor movement in viewing plane 502 during the click in response to movement of the user input device in tactile plane 504.

[0073] FIG. 11 is an illustration showing a second control function curve 820 representing an exemplary second transformation function F2 for determining the filtering of controller movements with respect to time during a click event, and also showing the time-aligned sequence of grip button displacement, the time alignment with the sequence of hand formations, and the time-aligned sequence of view plane instances. As used herein, selector velocity refers to the rate of increase in displacement from a neutral position of the manual selector 212 relative to the base of the user input device 204. In the exemplary case of FIG. 6A , selector velocity refers to the rate of increase in the displacement of the grip buttons 530 a, 530 b relative to the handle 530. As used herein, an increase in controller movement filtering corresponds to a decrease in the ratio of the movement of the user input device 204 to the movement of the cursor 402. That is, the greater the controller movement filtering, the less movement of the input device 204 in the tactile plane 504 that causes a corresponding movement of the cursor 402 in the view plane 502.

[0074] The exemplary second transformation function F2 includes first, second, and third filtering functions. As shown by the second control function curve 820, the input controller 118 is configured to apply the first filtering function 822 over a first time interval T1, during which the displacement of the manual selector 212 from a neutral position relative to the user input device 204 exceeds a first threshold displacement rate (T R1 As shown by second control function curve 820, input controller 118 is configured to apply a second filtering function 824 over a second time interval T2, during which displacement of manual selector 212 from a neutral position relative to user input device 204 changes at a rate less than a first threshold rate T R1 As shown by the second control function curve 820, the input controller 118 detects when the grip buttons 530a, 530b are no longer moving at the first threshold velocity T R16A , the displacement of the grip buttons 530 a, 530 b from a neutral position relative to the handle 530 increases during each of the first, second, and third time intervals.

[0075] Referring to the second control function curve 820, during a first time interval T1, the manual selector 212 selects T R1 6A and 11 , the directions of the arrows 852a, 852b indicate a decrease in the displacement distance between the fingers 758, 760 and the grip buttons 530a, 530b. Referring to the examples of FIGS. 6A and 11 , the directions of the arrows 852a, 852b indicate a decrease in the displacement distance between the fingers 758, 760 and the grip buttons 530a, 530b. The length of the arrows 852a, 852b indicates the rate at which the grip buttons move towards each other. The example input controller 118 can be configured to use a first filtering function 822 to increase the motion filtering of the user input device (with a corresponding decrease in the movement rate), for example, as a linear, logarithmic, or exponential function of time. Thus, during a first time interval T1, the selector displacement rate increases or decreases over a first threshold rate T R1 Although less than , the motion filtering of the user input device increases with increasing selector rate.

[0076] During the second time interval T2, the manual selector 212 selects the first threshold speed T R1 In an alternative exemplary surgical system 100, the manual selector 212 moves relative to the selector at a speed above the first threshold speed T R1 While the cursor 402 moves relative to the selector at a higher speed, the second filtering function 822 stops the movement of the cursor 402 so that the cursor movement does not follow the movement of the user input device 204. Referring to the examples of FIGS. 6A and 11 , the direction of the arrows 854a, 854b indicates a decrease in the displacement distance between the fingers 758, 760 and between the grip buttons 530a, 530b. Because the grip buttons 530a, 530b are moving at a faster speed during the second time interval T2 than during the first time interval, the arrows 854a, 854b have a longer length than the arrows 852a, 852b. The example input controller 118 can be configured to cause the movement ratio to become zero during the second time interval. That is, the cursor 402 does not move in response to the movement of the controller 204. More specifically, for example, the example input controller 118 can be configured to stop the movement of the cursor during the second time interval T2 (i.e., transition to a movement ratio of 1:0). Alternatively, for example, the example input controller 118 can be configured to reduce the movement ratio to 1:0.1 during the second time interval T2. Thus, during the second time interval T2, the filtering of the user input device motion and the corresponding movement ratio remain constant, while the movement rate of the grip buttons 530a, 530b decreases to 1:0.1. R1 or T R1 The duration of the second time interval T2 is determined by the time the speed of movement of the selectors 212, 530a, 530b exceeds the first threshold speed T R1 or the first threshold speed T R1 The time period is determined based on the length of time that the

[0077] In the exemplary surgical system 100, a click event 809 is triggered during the second time interval T2. The triggering of the click event can be controlled, for example, according to the first control function curve 720 of FIG. 7 and the control process 800 of FIG. 8.

[0078] During the third time interval T3, the speed of movement of the manual selector 212 is T R1 As the velocity of the grip buttons 530a, 530b relative to the handle 530 decreases to a velocity less than T2, the third filtering function 826 decreases the controller movement filtering as a function of time, which corresponds to an increase in the movement ratio as a function of time. Thus, movement of the cursor 402 follows movement of the user input device 204 according to the second dynamic movement ratio during the third time interval T3. Referring to the example manual selector 212 and user input device 204 of FIG. 6A , because the movement velocity of the grip buttons 530a, 530b relative to the handle 530 is greater during the second time interval T2 than during the third time interval T3, the arrows 856a, 856b have shorter lengths than the arrows 854a, 854b. The example input controller 118 can be configured to use the third filtering function 826 to decrease the controller movement filtering (and correspondingly increase the movement ratio), for example, as a linear function of time, a logarithmic function of time, or an exponential function of time. Additionally, the example input controller 118 can be configured to use a third filtering function 826 to increase the movement ratio as a function of time, for example, until the movement ratio regains a pre-click movement ratio of 1:1. Thus, during the third time interval T3, the filtering of user input device motion decreases over time, which means that the movement ratio increases over time.

[0079] 11 , a sequence of viewing window display instances 502a, 502b, 502c, 502d pictorially represent the change in cursor movement speed within viewing plane 502 at different points along second control function curve 820. In each successive viewing window instance 502a, 502b, 502c, 502d, a respective arrow 503a, 503b, 503d is associated with cursor 402. The length of each arrow in each respective viewing window display instance represents the magnitude of the movement ratio during the display time of the respective viewing window, the ratio of movement of the user input device to movement of the corresponding cursor.

[0080] Prior to the initiation of an exemplary click event, the movement ratio filter functions to impart a maximum movement ratio, represented by the long length of each arrow 503a within each viewing window display 502a.

[0081] During the first time interval T1, T R1 In response to movement of the manual selector 212 relative to the user input device 204 at a velocity less than , the example input controller 118 is configured with a second transformation function F2 to increase the movement filtering of the user input device as a function of the velocity of movement of the manual selector 212 relative to the user input device 204, corresponding to a decrease in the movement rate with an increase in the decreasing rate of grip button displacement. The decrease in the movement rate is represented by a shorter length of the arrow 503b in the viewing window display 502b.

[0082] During the second time interval T2, the manual selector 212 R1 In response to moving relative to the user input device 204 at these or similar speeds, the input controller 118 is configured with a second transformation function F2 to provide a constant minimum movement rate despite continued increases in the movement rate, which is represented by the absence of an arrow in the viewing window display 502c.

[0083] During a third time interval T3, in response to movement of the manual selector 212 relative to the user input device 204 at a reduced speed that is no longer equal to or greater than TR1, the input controller 118 is configured with a second transformation function F2 to decrease the controller action filtering (and correspondingly increase the movement ratio) as a function of time. The increase in the movement ratio is represented by the reappearance in the viewing window display 502d and by the arrow 503d. Note that the shorter length of the arrow 503d indicates that the movement ratio has not yet returned to the pre-click level.

[0084] (Computer Hardware and Storage Devices) FIG. 12 is an exemplary block diagram illustrating an exemplary machine upon which any one or more of the techniques (e.g., methodologies) discussed herein may be implemented, according to an exemplary embodiment. FIG. 12 shows an illustrative diagram of a more specific computer system 1200 that can be used, for example, to implement controller system 118. Computer system 1200 can be configured, for example, to implement a computerized training module. In alternative embodiments, computer system 1200 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked configuration, computer system 1200 may operate in the capacity of a server or client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Computer system 1200 may be a server computer, a client computer, a personal computer (PC), a tablet PC, a personal digital assistant (PDA), a mobile phone, or any machine capable of executing a set of instructions (sequence or otherwise) that specify actions to be taken by the machine. Furthermore, although only a single machine (i.e., computer system 1200) is shown, the term "machine" should be taken to include any collection of machines that individually or jointly execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein.

[0085] The exemplary computer system 1200 includes a processor 1202 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), or both), a main memory 1204, and a static memory 1206, which communicate with each other via a bus 1208. The computer system 1200 may further include a video display unit 1210 (e.g., a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a touch screen, or a cathode ray tube (CRT)) that may be used to display, for example, the positions of the surgical instrument 124 and the flexible instrument 120. The computer system 1200 also includes an alphanumeric input device 1212 (e.g., a keyboard, a physical keyboard, a virtual keyboard using software), a cursor control device or input sensor 1214 (e.g., a mouse, a trackpad, a trackball, a sensor or reader, a machine-readable information reader, a barcode reader), a disk drive unit 1216, a signal generation device 1218 (e.g., a speaker), and a network interface device or transceiver 1220.

[0086] The disk drive unit 1216 includes a non-transitory machine-readable storage device medium 1222 on which one or more sets of instructions 1224 (e.g., software) embodying any one or more of the methodologies or functions described herein are stored. The instructions 1224 may reside, completely or at least partially, within the main memory 1204, the static memory 1206, and / or the processor 1202 during execution by the computer system 1200, with the main memory 1204 and the processor 1202 also constituting non-transitory machine-readable storage device media. The non-transitory machine-readable storage device medium 1222 may also store integrated circuit designs and waveform structures. The instructions 1224 may further be transmitted or received over a network 1226 via a network interface device or transceiver 1220. While the machine-readable storage device medium 1222 is shown in the exemplary embodiment to be a single medium, terms such as "machine-readable medium," "computer-readable medium," and the like should be understood to include a single medium or multiple media (e.g., centralized or distributed databases, and / or associated caches and servers) that store one or more sets of instructions 1224. "Machine-readable medium" should also be interpreted to include any medium that can store, encode, or otherwise retain a set of instructions for execution by a machine, causing the machine to perform any one or more of the methodologies of this disclosure. The term "machine-readable medium" includes, but is not limited to, solid-state memory, optical and magnetic media, and carrier wave signals.

[0087] It will be understood that, for clarity, the above description may describe some embodiments with reference to different functional units or processors. However, it will be apparent that any suitable distribution of functionality between different functional units, processors, or domains may be used without detracting from the disclosure. For example, functionality shown to be performed by separate processors or controllers may be performed by the same processor or controller. Hence, references to specific functional units are to be understood only as references to suitable means for providing the described functionality, and not as indicative of a strict logical or physical structure or organization.

[0088] Although the present disclosure will be described in connection with several embodiments, it is not intended to be limited to the specific form set forth herein. Those skilled in the art will recognize that various configurations of the described embodiments may be combined in accordance with the present disclosure. Furthermore, it will be understood that various modifications and changes may be made by those skilled in the art without departing from the scope of the present disclosure.

[0089] Additionally, the foregoing Detailed Description may show that various features are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed embodiment. Accordingly, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separate embodiment.

[0090] The foregoing description and drawings of embodiments according to the present invention merely illustrate the principles of the inventive subject matter. It will thus be understood that various modifications can be made by those skilled in the art without departing from the scope of the inventive subject matter as defined in the appended claims.

[0091] Thus, while certain exemplary embodiments of the present invention have been described and illustrated in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of the broad inventive subject matter and are not limiting, and that embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art.

[0092] (Various examples) Example 1 may include a method of controlling actuation of a click event by a manual selector movably attached to a mounting structure, the method including: using one or more sensors to sense an amount of displacement distance of the manual selector from a neutral position; controlling one or more motors according to a first control state to apply a maintenance force in response to the manual selector being at a displacement distance less than a first threshold distance from the neutral position; controlling one or more motors according to a second control state to apply a haptic force to the manual selector that increases as a function of increasing displacement of the manual selector from the neutral displacement position in response to sensing the manual selector being at a displacement distance between the first threshold distance from the neutral position and a second threshold distance from the neutral position; applying a click event signal to cause occurrence of a click event at a display system in response to the manual selector satisfying the second threshold distance from the neutral position; and controlling one or more motors according to a third control state to decrease a magnitude of the haptic force applied to the manual selector to a reduced magnitude that is less than a maximum magnitude of haptic force applied during the second control state.

[0093] Example 2 can include the method described in Example 1, where the retention force is zero.

[0094] Example 3 can include the method of example 1, wherein the maintenance force is less than the haptic force applied by the one or more motors according to the second control state.

[0095] Example 4 can include the method of Example 1, wherein controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to impart a haptic force to the manual selector that decreases in magnitude to a decreasing magnitude as a function of increasing displacement of the manual selector from a neutral displacement position.

[0096] Example 5 can include the method of Example 4, wherein controlling the one or more motors according to the second control state includes controlling the one or more motors to increase the magnitude of the haptic force at a first rate, and controlling the one or more motors according to the third control state includes controlling the one or more motors to decrease the magnitude of the haptic force at a second rate, wherein the magnitude of the first rate is less than the magnitude of the second rate.

[0097] Example 6 can include the method of Example 4, wherein controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to impart a haptic force to the manual selector that decreases in magnitude as a function of increasing displacement of the manual selector from a neutral displacement position to a decreasing magnitude.

[0098] Example 7 can include the method of Example 4, where controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to apply a momentarily decreasing haptic force to the manual selector to a decreased magnitude.

[0099] Example 8 can include the method of example 1, further including using one or more motors to move the manual selector to a neutral position.

[0100] Example 9 can include the method of example 1, further including using a resilient member to move the manual selector to a neutral position.

[0101] Example 10 can include the method of example 1, further including using a stop surface to stop movement of the manual selector.

[0102] Example 11 can include the method of example 1, further including using a stop surface on the mounting structure to stop movement of the manual selector.

[0103] Example 12 can include the method of Example 1, further including using a stop surface positioned to apply a reaction force to the manual selector to stop further displacement of the manual selector when the manual selector reaches a third threshold displacement distance from the neutral position.

[0104] Example 13 can include the method of Example 1, further including controlling the one or more motors according to a third control state to impart the reduced magnitude to the manual selector at a displacement distance between the displacement distance at which the reduced magnitude is first imparted to the manual selector and a third threshold distance.

[0105] Example 14 can include the method of example 1, wherein the one or more sensors are configured to sense displacement of the manual selector relative to the mounting structure.

[0106] Example 15 can include the method of example 1, using one or more sensors to sense the amount of displacement of the manual selector from a neutral position during movement of the manual selector.

[0107] Example 16 is an apparatus for controlling actuation of click events by a manual selector movably attached to a mounting structure, the apparatus including one or more sensors configured to sense a position of the manual selector, one or more motors configured to apply a tactile force to the manual selector, processing circuitry, and a memory system storing instructions that, when executed by the processing circuitry, cause the processing circuitry to perform the following operations: using the one or more sensors to sense an amount of displacement of the manual selector from a neutral position; controlling the one or more motors according to a first control state to apply a maintaining force in response to the manual selector being at a displacement distance less than a first threshold distance from the neutral position; and controlling the one or more motors according to a first control state to apply a maintaining force in response to the manual selector being at a displacement distance between the first threshold distance from the neutral position and a second threshold distance from the neutral position. controlling one or more motors according to a second control state to apply a haptic force to the manual selector that increases as a function of increasing displacement of the manual selector from the neutral displacement position; applying a click event signal to cause occurrence of a click event in the display system in response to the manual selector meeting a second threshold distance from the neutral displacement position; controlling one or more motors according to a third control state to apply a haptic force to the manual selector that decreases in magnitude as a function of increasing displacement of the manual selector from the neutral displacement position, the decreased magnitude being less than a maximum magnitude of the haptic force applied during the second control state; and controlling one or more motors according to the third control state to decrease the magnitude of the haptic force applied to the manual selector to a decreased magnitude that is less than the maximum magnitude of the haptic force applied during the second control state.

[0108] Example 17 can include the device described in Example 16, where the retention force is zero.

[0109] Example 18 can include the device of Example 16, wherein the maintenance force is less than the haptic force applied by the one or more motors according to the second control state.

[0110] Example 19 can include the device of Example 16, wherein the act of controlling the one or more motors according to the third control state occurs after the act of applying the click event.

[0111] Example 20 can include the device of Example 16, wherein controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to impart a haptic force to the manual selector that decreases in magnitude to a decreasing magnitude as a function of increasing displacement of the manual selector from a neutral displacement position.

[0112] Example 21 can include the device of Example 20, wherein controlling the one or more motors according to the second control state includes controlling the one or more motors to increase the magnitude of the haptic force at a first rate, and controlling the one or more motors according to the third control state includes controlling the one or more motors to decrease the magnitude of the haptic force at a second rate, wherein the magnitude of the first rate is less than the magnitude of the second rate.

[0113] Example 22 can include the device of Example 20, wherein controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to impart a haptic force to the manual selector that decreases in magnitude to a decreasing magnitude as a function of increasing displacement of the manual selector from a neutral displacement position.

[0114] Example 23 can include the device of Example 20, wherein controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to apply a momentarily decreasing haptic force to the manual selector to a decreased magnitude.

[0115] Example 24 may include the apparatus of Example 16, further including instructions that, when executed, cause a processor to perform the following operations, the operations including controlling one or more motors to move the manual selector to a neutral position:

[0116] Example 25 can include the device of Example 16, further including a resilient member configured to move the manual selector to a neutral position.

[0117] Example 26 can include the device of Example 16, further including a stop surface configured to stop movement of the manual selector when the manual selector reaches a third threshold displacement distance from the neutral position.

[0118] Example 27 is a method of controlling movement of a cursor in a first two-dimensional (2D) plane based on movement of a user input device in a second 2D plane and based on movement of a manual selector movably attached to the user input device, the method including: causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane according to a constant movement ratio while the manual selector moves relative to the user input device at a speed less than a first threshold speed; and, in response to a speed of movement of the manual selector relative to the controller being between the first threshold speed and a second threshold speed. and, in response to the speed of movement of the manual selector relative to the user input device being below a second threshold speed, causing movement of the cursor in the first 2D plane to follow movement of the user interface device in the second 2D plane according to a movement ratio that decreases as a function of an increase in the speed of movement of the manual selector relative to the user input device; and, in response to the speed of movement of the manual selector relative to the user input device being below a second threshold speed, causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane according to a movement ratio that increases as a function of a decrease in the speed of movement of the manual selector relative to the user input device.

[0119] Example 28 can include the method of Example 27, further including, in response to the manual selector moving relative to the user input device at a velocity greater than a second threshold velocity, causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane according to a second constant movement ratio less than the first movement ratio.

[0120] Example 29 can include the method of Example 27, further including, in response to the manual selector moving relative to the user input device at a velocity greater than a second threshold velocity, causing movement of the cursor in the first 2D plane to stop following movement of the user input device in the second 2D plane.

[0121] Example 30 is an apparatus for controlling movement of a cursor in a first two-dimensional (2D) image display plane in a display system based on movement of a user input device in a second 2D tactile plane and based on movement of a manual selector movably attached to the user input device, the apparatus including one or more sensors configured to sense movement of the manual selector, processing circuitry, and a memory system storing instructions that, when executed by the processing circuitry, cause the processing circuitry to perform the following operations: moving the cursor in the first 2D plane relative to the user input device in the second 2D plane according to a constant movement ratio while the manual selector moves relative to the user input device at a speed less than a first threshold speed: and in response to the speed of movement of the manual selector relative to the user input device being between a first threshold speed and a second threshold speed, causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane at a movement ratio that decreases as a function of an increase in the speed of movement of the manual selector relative to the user input device, and in response to the speed of movement of the manual selector relative to the user input device decreasing below the second threshold speed, causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane at a movement ratio that increases as a function of a decrease in the speed of movement of the manual selector relative to the user input device.

[0122] Example 31 can include the apparatus of Example 30, further including instructions that, when executed, cause a processor to perform the following operations: in response to the manual selector moving relative to the user input device at a velocity greater than a second threshold velocity, causing the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane according to a second constant movement ratio that is less than the first movement ratio.

[0123] Example 32 may include the apparatus of Example 30, further including instructions that, when executed, cause a processor to perform the following operations: in response to the manual selector moving relative to the user input device at a velocity greater than a second threshold velocity, causing a cursor in the first 2D plane to stop following movement of the user input device in the second 2D plane.

Claims

1. 1. An apparatus for controlling actuation of a click event by a manual selector movably attached to a mounting structure, comprising: one or more sensors configured to sense the position of the manual selector; one or more motors configured to apply a tactile force to the manual selector; a processing circuit configuration; a memory system for storing instructions; The instructions, when executed by the processing circuitry, cause the processing circuitry to perform the following operations: using the one or more sensors to sense the amount of displacement of the manual selector from a neutral position; controlling one or more motors according to a first control state to apply a maintenance force in response to sensing the manual selector at a displacement distance less than a first threshold distance from the neutral position; controlling one or more motors according to a second control state to apply a haptic force to the manual selector that increases in magnitude as a function of increasing displacement of the manual selector from the neutral position in response to sensing the manual selector at a displacement distance between a first threshold distance from the neutral position and a second threshold distance from the neutral position; in response to sensing the manual selector satisfying the second threshold distance from the neutral position; applying a click event signal to cause occurrence of said click event at a display system; controlling the one or more motors according to a third control state to reduce the magnitude of the haptic force applied to the manual selector to a reduced magnitude that is less than a maximum magnitude of the haptic force applied during the second control state. Device.

2. The device of claim 1 , wherein the retention force is zero.

3. The device of claim 1 , wherein the maintaining force is less than a haptic force applied by the one or more motors according to the second control state.

4. The apparatus of claim 1 , wherein the act of controlling the one or more motors in accordance with the third control state occurs after the act of applying the click event.

5. 2. The device of claim 1, wherein controlling the one or more motors in accordance with the third control state comprises controlling the one or more motors to apply a haptic force to the manual selector that decreases in magnitude to the decreased magnitude as a function of increasing displacement of the manual selector from the neutral position.

6. controlling the one or more motors in accordance with the second control state includes controlling the one or more motors to increase the magnitude of the haptic force at a first rate; controlling the one or more motors in accordance with the third control state includes controlling the one or more motors to decrease the magnitude of the haptic force at a second rate; The magnitude of the first velocity is less than the magnitude of the second velocity.

6. The apparatus of claim 5.

7. 2. The device of claim 1, wherein controlling the one or more motors in accordance with the third control state comprises controlling the one or more motors such that a rate at which the magnitude of the haptic force applied to the manual selector decreases to the reduced magnitude in the third control state is shorter than a rate at which the magnitude of the haptic force applied to the manual selector increases as a function of the increasing displacement in the second control state.

8. The method further includes instructions that, when executed, cause the processing circuitry to perform the following operations: controlling the one or more motors to move the manual selector to the neutral position.

10. The apparatus of claim 1.

9. The device described in claim 1, further comprising an elastic member configured to bias the manual selector to the neutral position in the first control state.

10. The apparatus further includes a stop surface configured to stop movement of the manual selector when the manual selector reaches a third threshold distance from the neutral position, the operation comprising: responsive to a velocity of movement of the manual selector relative to the processing circuitry being between a first threshold velocity and a second threshold velocity, causing movement of a cursor in a first 2D plane to follow movement of a user interface device in a second 2D plane according to a movement ratio that decreases as a function of increasing velocity of movement of the manual selector relative to a user input device; and in response to a speed of movement of the manual selector relative to the user input device decreasing below the second threshold speed, causing movement of the cursor in the first 2D plane to follow movement of the user input device in the second 2D plane according to a movement ratio that increases as a function of the decrease in speed of movement of the manual selector relative to the user input device.

10. The apparatus of claim 1.

Images