System and method for haptic feedback of menu item selection in a remote control system

Abstract

The present invention relates to systems and methods for haptic feedback for selection of menu items in a teleoperated system. The haptic feedback method includes providing a teleoperated control system including a first mode of operation for operating a teleoperated instrument in response to movement of a control device in a first degree of freedom and a second mode of operation for controlling a graphical user interface. The method further includes entering the second mode of operation of the teleoperated control system. While in the second mode of operation, the method includes tracking movement of the control device in a second degree of freedom that is different from the first degree of freedom. While in the second mode of operation and in response to movement of the control device in the second degree of freedom, the method includes applying a first haptic force to the control device by the teleoperated control system.

Description

[0001] This application is a continuation of Chinese Patent Application No. 2018800428788 (PCT / US2018 / 040475), filed on June 29, 2018, entered into the National Stage on December 26, 2019, entitled “System and Method for Haptic Feedback of Selection of Menu Items in a Teleoperated System,” which claims the benefit of U.S. Provisional Application No. 62 / 529,038, filed on July 6, 2017, the entire contents of which are incorporated herein by reference.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 529,038, filed on July 6, 2017, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0004] The present disclosure relates to systems and methods for controlling operation of a teleoperated control system, and more particularly to systems and methods for providing haptic feedback to a user when accessing a graphical user interface menu using a control device of the teleoperated control system. BACKGROUND

[0005] Minimally invasive medical techniques are aimed at reducing the amount of tissue that is damaged during an invasive medical procedure, thereby reducing patient recovery time, discomfort, and harmful side effects. Such minimally invasive techniques can be performed through natural orifices in a patient’s anatomy or through one or more surgical incisions. Through these natural orifices or incisions, a clinician can insert a medical tool to access a target tissue location. Minimally invasive medical tools include instruments such as therapeutic, diagnostic, and surgical instruments. Minimally invasive medical tools can also include imaging instruments such as endoscopic instruments. Some minimally invasive medical instruments can be teleoperated or computer-assisted. Various control devices can be used to control the teleoperated or computer-assisted medical tools. To expand the functionality of the teleoperated system without adding additional structural controls to the user’s console, a graphical user interface can be used. When accessing the graphical user interface using a control device that is also used to control an instrument within a patient’s body, systems and methods provide haptic feedback to the user. SUMMARY

[0006] Embodiments of the present invention are summarized in the appended claims.

[0007] In one embodiment, a haptic feedback method includes providing a teleoperational control system including a first mode of operation for operating a teleoperational instrument in response to movement of a control device in a first degree of freedom and a second mode of operation for controlling a graphical user interface. The method also includes engaging the second mode of operation of the teleoperational control system. While in the second mode of operation, the method includes tracking movement of the control device in a second degree of freedom that is different from the first degree of freedom. While in the second mode of operation and in response to movement of the control device in the second degree of freedom, the method includes applying a first haptic force to the control device by the teleoperational control system.

[0008] In another embodiment, a haptic feedback method includes engaging an interface mode of a teleoperational control system. In response to movement of a control device of the teleoperational control system, the method also includes applying a first haptic force to the control device from a nominal position to urge the control device toward the nominal position. The method also includes determining that the control device has moved a first displacement distance from the nominal position to an engaged position in a first degree of freedom. The method also includes applying a second haptic force to the control device to provide a haptic sensation indicative of an engaged operating state. The method also includes applying a third haptic force to the control device while in the engaged operating state to urge the control device to return from the engaged position to the nominal position. BRIEF DESCRIPTION OF DRAWINGS

[0009] Aspects of the disclosure can best be understood with reference to the following detailed description when read with the accompanying drawings in which, according to common practice, the various features are not necessarily drawn to scale. In fact, the dimensions can be arbitrarily increased or decreased with the emphasis being upon illustrative rather than recreational aspects. Additionally, the disclosure can repeat reference numerals and / or letters in various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0010] Figure 1A is a schematic view of a teleoperational medical system according to embodiments of the present disclosure.

[0011] Figure 1B is a perspective view of a surgeon’s console for a teleoperational medical system according to many embodiments.

[0012] Figure 1C is a perspective view of a teleoperational medical system electronics cart according to many embodiments.

[0013] Figure 1D is a perspective view of a patient side cart according to one example of the principles described herein.

[0014] Figure 2 illustrates a graphical user interface including a selector icon.

[0015] Figure 3A selector icon in a nominal position. Figure 2

[0016] Figure 3B selector icon in an entry position. Figure 2

[0017] Figure 3C selector icon returning from an entry position to a nominal position. Figure 2

[0018] Figure 4 selector icon according to an alternative embodiment.

[0019] Figure 5 is a flowchart illustrating a method of providing haptic feedback by a control device of a remote operating system when using a graphical user interface menu.

[0020] Figure 6 is a torque curve for providing haptic feedback according to one embodiment of the present disclosure.

[0021] Figure 7 is a flowchart illustrating another method of providing haptic feedback by a control device of a remote operating system when using a graphical user interface menu.

[0022] Figure 8 is a torque curve for providing haptic feedback according to another embodiment of the present disclosure.

[0023] Figure 9 is a flowchart illustrating another method of providing haptic feedback by a control device of a remote operating system when using a graphical user interface menu. DETAILED DESCRIPTION

[0024] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe them. It will nevertheless be understood that no limitation of the scope of the disclosure is intended. In the following detailed description of the aspects of the application, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to one skilled in the art that embodiments of the present disclosure can be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present disclosure.

[0025] ​​​Any alterations and further modifications to the described devices, instruments, methods, and any further applications of the principles of the present application are contemplated and can be made by those skilled in the art without departing from the spirit and scope of the present application. Particularly, it is contemplated that the features, components, and / or steps described with respect to one embodiment can be combined with features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for particular examples and it is contemplated that different sizes, dimensions, and / or ratios can be utilized to implement the concepts of the present disclosure. To avoid needless descriptive repetition, one or more components or acts described according to one illustrative embodiment can be used or omitted according to the applicability of other illustrative embodiments. For the sake of brevity, many iterations of these combinations will not be described separately. For simplicity, in some instances, the same reference numbers are used throughout the drawings to refer to the same or like parts.

[0026] The following embodiments will describe various instruments and portions of instruments in terms of their states in three-dimensional space. As used herein, the term "position" refers to the location of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or portion of an object (three rotational degrees of freedom, e.g., roll, pitch, and yaw).

[0027] Referring to the drawings Figure 1A A teleoperated medical system for use in, for example, a medical procedure including a diagnostic, therapeutic, or surgical procedure is generally indicated by the reference numeral 10. As will be described, the teleoperated medical system of this disclosure is under the remote operator control of a surgeon. In alternative embodiments, the teleoperated medical system can be under partial control of a computer programmed to perform a procedure or sub-procedure. In other alternative embodiments, a fully automated medical system under the full control of a computer programmed to perform a procedure or sub-procedure can be used to perform the procedure or sub-procedure. As shown in Figure 1A The teleoperated medical system 10 generally includes a teleoperation assembly 12 mounted to or near an operating table O on which a patient P is positioned. The teleoperation assembly 12 can be referred to as a patient-side cart. A medical instrument system 14 and an endoscopic imaging system 15 are operatively coupled to the teleoperation assembly 12. An operator input system 16 allows a surgeon or other type of clinician S to view images of or representative of a surgical site and to control the operation of the medical instrument system 14 and / or the endoscopic imaging system 15.

[0028] The operator input system 16 can be located at a surgeon's console, which is typically in the same room as the operating table O. However, it should be understood that the surgeon S can be in a different room or a completely different building from the patient P. The operator input system 16 typically includes one or more control devices for controlling the medical instrument system 14. The one or more control devices can include one or more of any number of various input devices such as hand grips, joysticks, trackballs, data gloves, trigger guns, hand-operated controllers, voice recognition devices, touch screens, limb motion or presence sensors, etc. In some embodiments, the one or more control devices will be provided with the same degrees of freedom as the medical instruments of the teleoperational assembly to provide the surgeon with telepresence: the perception that the one or more control devices are an integral part of the instruments, such that the surgeon has a strong awareness of directly controlling the instruments, as if at the surgical site. In other embodiments, the one or more control devices can have more or fewer degrees of freedom than the associated medical instruments, and still provide the surgeon with telepresence. In some embodiments, the one or more control devices are hand input devices that move in six degrees of freedom and can also include actuatable handles for actuating the instruments (e.g., to close grasper jaws, apply an electrical potential to an electrode, deliver a therapeutic drug, etc.).

[0029] When surgeon S views the surgical site through console 16, teleoperational assembly 12 supports and manipulates medical instrument system 14. Images of the surgical site can be obtained through endoscopic imaging system 15, such as a stereoscopic endoscope, which can be manipulated by teleoperational assembly 12 to position endoscope 15. Electronic cart 18 can be used to process images of the surgical site for subsequent display to surgeon S through surgeon's console 16. The number of medical instrument systems 14 used at one time typically depends on the diagnostic or surgical procedure and space limitations within the operating room, among other factors. Teleoperational assembly 12 can include one or more kinematic structures of non-servo controlled links (e.g., one or more links that can be manually positioned and locked in place, often referred to as set-up structures) and teleoperational manipulators. Teleoperational assembly 12 includes a plurality of electric motors that drive inputs on medical instrument system 14. These electric motors move in response to commands from a control system (e.g., control system 20). The electric motors include drive systems that, when coupled to medical instrument system 14, can advance the medical instrument into a naturally or surgically created anatomical orifice. Other motorized drive systems can move the distal end of the medical instrument over a plurality of degrees of freedom including three linear motion angles (e.g., linear motion along the X, Y, Z axes of a Cartesian reference frame) and three rotational motion angles (e.g., rotation about the X, Y, Z Cartesian coordinate axes). Additionally, the electric motors can be used to actuate articulable end effectors of the instruments to grasp tissue in jaws of a biopsy device or the like. The electric motors can be controlled to produce commanded torques (or forces under linear motors).

[0030] Teleoperational medical system 10 also includes control system 20. Control system 20 includes at least one memory and at least one processor (not shown), and typically a plurality of processors, for effecting control between medical instrument system 14, operator input system 16, and electronic system 18. Control system 20 also includes programmed instructions (e.g., computer readable media storing instructions) to implement some or all of the methods described in accordance with aspects disclosed herein. Although control system 20 is shown as a single block in the simplified schematic of FIG. 1, the system can include two or more data processing circuits, with one portion of the processing optionally being performed on or near teleoperational assembly 12, another portion of the processing being performed at operator input system 16, etc. Any of a wide variety of centralized or distributed data processing architectures can be employed. Similarly, the programmed instructions can be implemented as a plurality of separate programs or subroutines, or they can be integrated into many other aspects of the teleoperational systems described herein. In one embodiment, control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry. Figure 1A Although control system 20 is shown as a single block in the simplified schematic of FIG. 1, the system can include two or more data processing circuits, with one portion of the processing optionally being performed on or near teleoperational assembly 12, another portion of the processing being performed at operator input system 16, etc. Any of a wide variety of centralized or distributed data processing architectures can be employed. Similarly, the programmed instructions can be implemented as a plurality of separate programs or subroutines, or they can be integrated into many other aspects of the teleoperational systems described herein. In one embodiment, control system 20 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0031] In some embodiments, the control system 20 can include one or more servo controllers that receive force and / or torque feedback from the medical instrument system 14. In response to the feedback, the servo controllers send signals to the operator input system 16. The servo controller(s) can also send signals to instruct the teleoperational assembly 12 to move the medical instrument system(s) 14 and / or endoscopic imaging system 15 that extend to an internal surgical site within a patient's body via an opening in the body. Any suitable conventional or specialized servo controllers can be used. The servo controllers can be separate from or integrated with the teleoperational assembly 12. In some embodiments, the servo controllers and teleoperational assembly are provided as part of a teleoperational arm cart positioned near the patient's body.

[0032] The teleoperational medical system 10 can also include optional operating and support systems (not shown), such as illumination systems, steering control systems, irrigation systems, and / or suction systems. In alternative embodiments, the teleoperational system can include more than one teleoperational assembly and / or more than one operator input system. The exact number of manipulator assemblies will depend on the surgical procedure and space limitations within the operating room, among other factors. The operator input systems can be collocated or they can be placed in separate locations. Multiple operator input systems allow more than one operator to control one or more manipulator assemblies in various combinations.

[0033] Figure 1Bis a perspective view of the surgeon's console 16. The surgeon's console 16 includes a left eye display 32 and a right eye display 34 for presenting to the surgeon S a coordinated stereo view of the surgical site that can be perceived in depth. The console 16 also includes one or more input control devices 36, 37 that the surgeon uses to perform functions of the system 10. The input control device 36 is a hand input control device that causes the teleoperational assembly 12 to manipulate one or more instruments or endoscopic imaging systems. The input control device 36 can provide the same degrees of freedom as the instrument 14 associated therewith to provide the surgeon S with telepresence, or the input control device 36 can be perceptually integrated with the instrument 14 so that the surgeon has a strong sense of directly controlling the instrument 14. To this end, position, force, and tactile feedback sensors (not shown) can be employed to transmit position, force, and tactile sensations from the instrument 14 back to the surgeon's hand through the input control device 36. The input control device 37 allows the system 10 to switch between operating modes. The device 37 can be a foot pedal operated by the surgeon, or can be other types of hand or foot switches that function as clutch devices to disengage a first operating mode and enter a second operating control mode. The operating modes of the teleoperational medical system 10 can include, for example, a surgical instrument control mode, a camera control mode, a menu control mode, etc. The surgical instrument control mode can allow the surgeon to control manipulation of the instrument 14 as described above. The camera control mode can allow the surgeon to use the input control device 36 to cause the teleoperational assembly 12 to manipulate the endoscopic imaging system 15. The menu control mode can allow the surgeon to use the input control device 36 to navigate through graphical user interface menus that are displayed to the surgeon via the left eye display 32 and the right eye display 34. For example, the control device 37 can be depressed to switch between the surgical instrument control mode and the menu mode.

[0034] Figure 1C is a perspective view of the electronics cart 18. The electronics cart 18 can be coupled with the endoscope 15 and can include a processor to process captured images for subsequent display to a surgeon, such as on the surgeon's console or on another suitable display located locally and / or remotely. For example, in the case of using a stereo endoscope, the electronics cart 18 can process captured images to present to the surgeon a coordinated stereo view of the surgical site. Such coordination can include alignment between the opposing images, and can include adjustment of the stereo working distance of the stereo endoscope. As another example, image processing can include compensating for imaging errors of the image capture device, such as optical aberrations, using previously determined camera calibration parameters. The electronics cart 18 can also include components of the display monitor and control system 20.

[0035] Figure 1Dis a perspective view of one embodiment of a teleoperational assembly 12 that can be referred to as a patient-side cart. The patient-side cart 12 shown provides manipulation of three surgical tools 26 (e.g., instrument systems 14) and an imaging device 28 (e.g., endoscopic imaging system 15), such as a stereoscopic endoscope for capturing images of a procedure site. The imaging device can transmit signals to the electronics cart 18 through a cable 56. Manipulation is provided by a teleoperational mechanism having a plurality of joints. The imaging device 28 and surgical tools 26 can be positioned and manipulated through an incision in the patient such that the kinematic remote center is maintained at the incision to minimize the size of the incision. Images of the surgical site can include images when the distal end of the surgical tool 26 is positioned within the field of view of the imaging device 28.

[0036] The patient-side cart 22 includes a drivable base 58. The drivable base 58 is connected to telescoping columns 57 that allow adjustment of the height of the arms 54. The arms 54 can include a rotary joint 55 that rotates and moves up and down. Each arm 54 can be connected to a dolly platform 53. The dolly platform 53 can be able to rotate 360 degrees. The patient-side cart 22 can also include a telescoping horizontal boom 52 that moves the dolly platform 53 in a horizontal direction.

[0037] In the present example, each arm 54 is connected to a manipulator arm 51. The manipulator arm 51 can be directly connected to a medical instrument 26. The manipulator arm 51 can be teleoperated. In some examples, the arms 54 connected to the dolly platform are not teleoperable. Rather, these arms 54 are positioned as needed prior to the surgeon 18 beginning operation with the teleoperational components.

[0038] Various configurations can be included to provide an endoscopic imaging system (e.g., system 15, 28), including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end to the proximal end of the endoscope. Flexible endoscopes transmit images using one or more flexible optical fibers. Endoscopes can be provided with different viewing angles, including a 0° viewing angle for forward axial viewing or a viewing angle between 0°-90° for forward tilted viewing. Digital image-based endoscopes have a “chip-on-the-tip” design, in which a distal digital sensor, such as one or more charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS) devices, stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images can provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a viewer with more accurate depth perception. Stereoscopic endoscopic instruments employ stereoscopic cameras to capture stereoscopic images of patient anatomy.

[0039] To expand the functionality of the teleoperated medical system 10 without adding additional structural accessories to the console 16 such as foot pedals, physical switches, dials, and buttons, a graphical user interface menu can be used to provide additional options and actions for operating the system 10. When providing additional functionality of the system 10 using a graphical user interface menu, it is helpful to provide haptic feedback to the operator at the console 16 to facilitate interaction with the graphical user interface menu elements and to provide confirmation that a command has been executed. Other feedback mechanisms, such as audible and visual feedback cues, can also provide confirmation that a command has been executed. The use of haptic feedback when using a graphical interface menu can provide a sense of immersion for the user.

[0040] Figure 2 A display 100 is illustrated that is visible through the left eye display 32 and the right eye display 34 of the surgeon's console 16. In this embodiment, the display includes a view of a surgical environment 102 that includes medical tools 104, 106, 108. In a "follow" mode of operation of the surgical instrument control or teleoperated system, the control devices 36 can be manipulated to control movement of the tools 104, 106, 108 in a limited or unlimited degrees of freedom. For example, in a surgical instrument control mode, the control devices can operate to move the instruments in three translational degrees of freedom in a Cartesian coordinate space. When a graphical user interface mode of the teleoperated system is invoked (e.g., by depressing a clutch pedal 37), the control devices 36 are decoupled from the tools 104, 106, 108, and can be used to select items from a graphical user interface menu. Figure 2 A graphical user interface menu is illustrated that includes a selector icon 110 and a plurality of menu items 112, 114, 116. As shown, the graphical user interface menu 113 can be superimposed on an image of the surgical environment 102, or alternatively, the graphical user interface menu can be displayed alone without the surgical environment. In the graphical user interface mode where the tools 104, 106, 108 are decoupled from the control devices 36, one or more of the control devices can be coupled to move the selector icon 110 between a nominal or neutral position (as shown) and the menu items 112, 114, 116. In one embodiment, the menu items 112 can be associated with an endoscope camera control mode in which the camera can be repositioned. Figure 2 Figure 2 As shown, the graphical user interface menu 113 can be superimposed on an image of the surgical environment 102, or alternatively, the graphical user interface menu can be displayed alone without the surgical environment. In the graphical user interface mode where the tools 104, 106, 108 are decoupled from the control devices 36, one or more of the control devices can be coupled to move the selector icon 110 between a nominal or neutral position (as shown) and the menu items 112, 114, 116. In one embodiment, the menu items 112 can be associated with an endoscope camera control mode in which the camera can be repositioned.

[0041] Figure 3A A graphical user interface menu is illustrated that includes a selector icon 110 and a plurality of menu items 112, 114, 116. As shown, the graphical user interface menu 113 can be superimposed on an image of the surgical environment 102, or alternatively, the graphical user interface menu can be displayed alone without the surgical environment. In the graphical user interface mode where the tools 104, 106, 108 are decoupled from the control devices 36, one or more of the control devices can be coupled to move the selector icon 110 between a nominal or neutral position (as shown) and the menu items 112, 114, 116. In one embodiment, the menu items 112 can be associated with an endoscope camera control mode in which the camera can be repositioned. Figure 2 ​selector icon 110 in a nominal position and having a clockwise direction of movement that corresponds to clockwise movement of the control device 36 about the control device's axis of rotation. The nominal position of the control device 36 can be in the space in which the control device is positioned and oriented when entering the graphical user interface mode. Thus, the position of the nominal position of the control device 36 in the space can be different each time the interface mode is entered.

[0042] The movement of the selector icon 110 can be coupled to a right or left hand control device. Different graphical user interface menus can be associated with each hand, such that the right hand makes selections from a first menu and the left hand makes selections from a second menu. Alternatively, the menus can be configured such that menu items are selected by counterclockwise rotation. When the system is in other operating modes such as a surgical instrument control mode, the rotational movement of the control device 36 used to move the selector icon 110 can be a degree of freedom that is not used to operate the tool. Thus, the user recognizes that the roll degree of freedom about the axis of the control device is used for menu selection rather than tool operation. In other embodiments, the same degree of freedom used to control the selector icon can be used to control movement of the tool in the surgical instrument control mode. In other embodiments, the control device can be moved in both clockwise and counterclockwise directions to select menu items. For example, clockwise rotation can point the selector icon to a menu item that "activates a camera control action" while counterclockwise rotation can point the selector icon to a menu item that "activates a migration" action. In other embodiments, the control device can be moved about other rotational axes or can be translated along an axis to produce the selection motion. In other words, any Cartesian translational or rotational motion can produce the selection motion.

[0043] Reference is now made to Figure 5 FIG. 15 is a flowchart 150 illustrating a method of providing haptic feedback by a control device of a teleoperational system (e.g., system 10) when using a graphical user interface menu (e.g., menu 113). Prior to initiating the process shown in flowchart 150, the system 10 can be in a surgical instrument control mode or another operating mode of the system. At process 152, the control system 20 determines whether an input control device 37 has been actuated to initiate a graphical user interface mode of the system 10. For example, this mode can be enabled by depressing a clutch pedal 37 of the surgeon's console to decouple from the surgical instrument control mode (including disengaging the surgical instrument from the control devices 36) and enter a graphical user interface control mode, including coupling at least one of the control devices to a graphical user interface menu selector icon. At process 154, the menu 113 is displayed with or without an image of the surgical environment. For example, if the menu 113 is associated with an instrument that is visible in the surgical environment, the menu can appear near the distal tip of the instrument.

[0044] At process 156, the system 10 receives and tracks a control signal from the control device 36 indicating that the control device is being moved in a clockwise direction about the axis of the shaft of the control device. Accordingly, the selector icon 110 is moved counterclockwise from the nominal position N (see Figure 3B ) to the menu item A as shown. Figure 3A

[0045] At process 158, a haptic force, such as a haptic torque, is provided to the control device when the instrument control device moves a threshold distance of displacement associated with the selection position corresponding to the menu item A 114. The haptic force can be superimposed on the normal torque generated by the controller. The haptic torque provides a haptic detent or feel to the user indicating that the controller has selected the menu item A. The normal torque can be a torque that is generated, for example, to reflect to the user a tracking error that the teleoperational assembly has with respect to the position / orientation of the control device.

[0046] At process 160, a command associated with the menu item A is executed in response to a user input (e.g., pressing a physical button on the control device). For example, the menu item A can initiate a camera control. Alternatively, and in particular, if there is a single menu item, the command can be executed when the control device reaches the selected position. Optionally, the selector icon 110 and the control device 36 can remain in the selected position, waiting for further movement of the control device to move the selector icon clockwise to the menu item B 116 or counterclockwise to the nominal position.

[0047] At process 162, optionally, after the command of the menu item A is executed at process 160, another haptic torque can be provided to return the control device 36 and the selector icon 110 to the nominal position (see Figure 3C ​). With this technique of re-centering the selector icon and control 36 to the nominal position, the counter module can be used to count each time the selector icon and control 36 returns from the selected position to the nominal position. In this embodiment, the count value of the counter module is incremented each time the selector icon and control 36 returns from the selected position to the nominal position. Each count increment of the counter module can be associated with a menu item, allowing the user to switch between items in the menu with repeated back-and-forth movement of the control 36. For example, a single click (e.g., a count of one movement into the selected position and one return into the nominal position) can correspond to a first menu item, and two clicks can correspond to a second menu item. The incremental menu items can be displayed to the user. Using the counter module requires the user to rotate the control only a limited distance (e.g., to menu item A) to switch through several menu items, rather than requiring the user to rotate the control away from the nominal position. This can prevent the user from twisting his hand into a large angular position, and allows the user's hand to remain close to the nominal position, where he can quickly change modes and resume control of the surgical instrument coupled to the control 36.

[0048] Figure 4 An alternative graphical user interface menu 120 is illustrated, in which the selector icon 122 moves in a translational direction. Other menu arrangements in which the control and selector icon move in other single degrees of freedom can be used.

[0049] In an alternative embodiment, clockwise motion of the control 36 or counterclockwise motion of the control can cause the selector icon 110 to move in a clockwise direction, or can cause the selector icon 122 to move in a single translational direction. In other words, motion of the control 36 in the left-right roll direction will cause the same single advancement of the selector icon. For each single action, the user can feel a tactile detent feature or "click" sound. This embodiment can be suitable for making the movement action easy in multiple postures of the control 36. For example, when the control 36 is pointing to the right, the user can more easily twist the roll axis of the control to the left than to the right to advance the selector icon. By making the movement action symmetrical (i.e., such that either control can advance the selector), the user has more choices for controlling the selector. This feature can be activated on both left and right controls simultaneously. The software can detect which control exceeds the δ1 threshold to determine which control is activating the tactile detent feature.

[0050] Figure 6 A tactile detent torque profile T S is illustrated, which is superimposed on a controller torque profile T S used to provide tactile feedback to the control 36. CThe above, where the applied motor torque simulates a spring-loaded button that rotates. In this embodiment, the tactile detent torque curve T S produces the tactile feel of a spring-loaded button. Figure 7 is a flowchart 180 that describes Figure 6 the torque curve. The torque curve is provided to one or more drive actuators in the control device 36 to provide force feedback that is felt by the surgeon's hand. The controller torque curve T C provides a centering or force that pushes the control device toward the nominal position. Greater torque is applied as the displacement δ (e.g., the angle or distance of rotation of the control device 36) increases, and as the displacement δ increases, the surgeon's hand on the control device feels increasing resistance of the control device. The nominal position is at the location where the displacement δ is equal to zero. The tactile detent torque curve T C is superimposed on the controller torque curve T S has multiple regions.

[0051] Referring to Figure 6 and Figure 7 , in process 182, when the control device has a displacement (relative to the nominal value) within the displacement region between zero and δ1, the torque curve T1 follows the controller torque curve T C , providing no additional resistive torque beyond the torque curve T C . In process 184, when the control device has a displacement in the displacement region between δ1 and δ2, the torque curve T2 has a steep negative slope away from the torque curve T C . That is, the resistance experienced by the surgeon suddenly decreases. In process 186, when the control device has a displacement in the displacement region between δ2 and δ3, the torque curve T3 has a torque sign reversal that provides a force that urges the control device away from the nominal position and toward the displacement distance δ2. The torque curve T3 provides a tactile feel of a strong push toward a button "click" sound or entry point that occurs at the displacement distance δ3. In process 188, once the control device has reached the displacement distance δ3, the torque curve T4 provides a centering (toward the nominal) force that is smaller than the torque curve T C at the same displacement distance. As Figure 6 shown, the torque curves T C and T S have symmetric curves that represent two "button" torque curves. The symmetric torque curves can provide the same tactile detent feel, but can occur when the rotational motion of the control device is in opposite directions. In alternative embodiments, the calculated torque curve can change at different displacements associated with the positions of different menu items, and the magnitude of the torque provided can increase or decrease depending on a variety of factors including user preferences, user grip actions associated with detent amplitudes, or expected times between successive actions.

[0052] Figure 8 The haptic detent torque curve T C is superimposed on the controller torque curve T E for providing haptic feedback to the control device 36 after the entry or "click" displacement distance δ3 is reached. In this embodiment, the haptic detent torque curve T E creates a haptic feel of a spring-loaded button entering or "clicking" and a haptic feel of a centered detent. Figure 9 is a flowchart 190 of the torque curve described Figure 8 Once the control device 36 has reached the displacement distance δ3, the torque curve T4 provides a small centering (toward nominal) force than the torque curve T C at the same displacement distance. Following the torque curve T4 until the control device reaches the displacement distance δ5 when returning to the nominal position. In process 192, when the control device reaches the displacement distance δ5, the torque curve T5 provides an increased torque to the level of the controller torque curve T C When the torque curve T5 is applied to the control device, the user experiences a sudden increase in resistance than T4. The sudden increase in resistance can cause the user to instinctively relax or release their grip, thereby allowing the control device to re-center to the nominal position in process 194 where δ is equal to zero.

[0053] In one alternative embodiment, a torque simulating a translating switch is superimposed on the normal control device torque, which provides ON and OFF switch positions at displacement distances along a line in space. In another alternative embodiment, a torque simulating a rotating switch is superimposed on the normal control device torque, which provides ON and OFF switch positions at angular displacement distances around an axis in space. This embodiment can be similar to the button embodiment described in detail above, except that a centering force is not provided to move the control device toward the nominal position. Rather, the control device remains in the selected position (i.e., "click state") until the actuation action is performed, at which time the detent is deactivated or reset. In another alternative embodiment, a torque simulating a spring-loaded translating button moving along a line in space is superimposed on the normal control device torque.

[0054] One or more elements of embodiments of the application can be implemented in software to execute on a computer system such as a processor of a control processing system. When implemented in software, the elements of embodiments of the application are essentially the code segments that perform the necessary tasks. The program or code segments can be stored in a processor readable storage medium or device, which can include any medium or device readable by a processor, including optical, semiconductor, and magnetic varieties. A processor readable storage device can include an electronic circuit; a semiconductor device, a semiconductor memory device, a read-only memory (ROM), a flash memory, an erasable programmable read-only memory (EPROM); a floppy diskette, a CD-ROM, an optical disk, a hard disk, or other storage device. The code segments can be downloaded via computer networks such as the Internet, Intranet, etc.

[0055] It is to be noted that the processes and displays presented can not inherently be related to any particular computer or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the operations described. The required structure for a variety of these systems will appear as elements in the claims. In addition, the embodiments of the application are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the application as described herein.

[0056] While certain example embodiments of the application have been described and shown herein, it is understood that such embodiments are merely illustrative of the many possible specific arrangements which can be present as specific constructions and configurations, and that all such specific arrangements and configurations are intended to be included within the scope of the present application.

Claims

1. A teleoperational control system comprising: one or more processors; and one or more memories having computer readable instructions stored thereon that, when executed by the one or more processors, cause the teleoperational control system to: enter an interface mode of the teleoperational control system; in response to movement of a control device of the teleoperational control system from a nominal position, apply a first haptic force to the control device to urge the control device toward the nominal position; determine that the control device has moved from the nominal position in a first degree of freedom by a first displacement distance to an entry position; apply a second haptic force to the control device to provide a haptic sensation indicative of an entry operational state; and while in the entry operational state, apply a third haptic force to the control device to urge the control device to return from the entry position to the nominal position.

2. The teleoperation control system of claim 1, wherein, the second haptic force provides a haptic sensation indicative of an initial phase of the entry operational state, and wherein the computer readable instructions, when executed by the one or more processors, further cause the teleoperational control system to: apply a fourth haptic force to the control device opposite the second haptic force to urge the control device to the entry position, the fourth haptic force providing a haptic sensation indicative of an entire phase of the entry operational state.

3. The teleoperation control system of claim 1, wherein, the computer readable instructions, when executed by the one or more processors, further cause the teleoperational control system to: increment a counter value upon the control device returning to the nominal position.

4. The teleoperation control system of claim 1, wherein, the computer readable instructions, when executed by the one or more processors, further cause the teleoperational control system to enter an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a plurality of degrees of freedom, wherein the plurality of degrees of freedom does not include the first degree of freedom.

5. The teleoperation control system of claim 1, wherein, the movement of the control device in the first degree of freedom is in a rotational degree of freedom about an axis of rotation of the control device.

6. The teleoperational control system of claim 1, further comprising a display graphical user interface, the graphical user interface comprising a selector icon, a nominal indicium, and a selection indicium, and wherein, the selector icon identifies the nominal indicium when the control device is in the nominal position and identifies the selection indicium when the control device is in the entry position.

7. The teleoperation control system of claim 1, wherein, the teleoperational control system includes an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a second degree of freedom different from the first degree of freedom.

8. The teleoperation control system of any one of claims 1 to 7, wherein, the teleoperational control system includes an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a plurality of degrees of freedom, wherein the plurality of degrees of freedom does not include the first degree of freedom.

9. A non-transitory computer readable medium comprising machine readable instructions that, when executed by one or more processors associated with a teleoperational control system, cause the teleoperational control system to: enter an interface mode of the teleoperational control system; in response to movement of a control device of the teleoperational control system from a nominal position, applying a first haptic force to the control device to urge the control device toward the nominal position; determining that the control device has moved from the nominal position in a first degree of freedom by a first displacement distance to an entry position; applying a second haptic force to the control device to provide a haptic sensation indicative of an entry operating state; and while in the entry operating state, applying a third haptic force to the control device to urge the control device to return from the entry position toward the nominal position.

10. The non-transitory computer-readable medium of claim 9, wherein, the second haptic force provides a haptic sensation indicative of an initial phase of the entry operating state, and wherein the machine-readable instructions, when executed by the one or more processors, further cause the teleoperational control system to: apply a fourth haptic force to the control device opposite the second haptic force to urge the control device to the entry position, the fourth haptic force providing a haptic sensation indicative of an entire phase of the entry operating state.

11. The non-transitory computer-readable medium of claim 9, wherein, the machine-readable instructions, when executed by the one or more processors, further cause the teleoperational control system to: increment a counter value upon return of the control device to the nominal position.

12. The non-transitory computer-readable medium of claim 9, wherein, the machine-readable instructions, when executed by the one or more processors, further cause the teleoperational control system to enter an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a plurality of degrees of freedom, wherein the plurality of degrees of freedom does not include the first degree of freedom.

13. The non-transitory computer-readable medium of claim 9, wherein movement of the control device in the first degree of freedom is in a rotational degree of freedom about an axis of rotation of the control device.

14. The non-transitory computer-readable medium of claim 9, further comprising: displaying a graphical user interface including a selector icon, a nominal indicium, and a selection indicium, and wherein the selector icon identifies the nominal indicium when the control device is in the nominal position and the selector icon identifies the selection indicium when the control device is in the entry position.

15. The non-transitory computer-readable medium of claim 9, wherein the teleoperational control system includes an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a second degree of freedom different from the first degree of freedom.

16. The non-transitory computer-readable medium of any one of claims 9 to 15, wherein, the teleoperational control system includes an instrument control mode for operating a teleoperational instrument in response to movement of the control device in a plurality of degrees of freedom, wherein the plurality of degrees of freedom does not include the first degree of freedom.

Images