Gradual force-feedback transitions between control states

A gradual transition of haptic feedback force vectors addresses the challenge of abrupt feedback changes in surgical systems, providing a seamless user experience during control state shifts.

JP7590072B2Active Publication Date: 2024-11-26INTUITIVE SURGICAL OPERATIONS INC
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2018533730
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2016-01-12
Filing Date
2017-01-12
Publication Date
2024-11-26
Estimated Expiration
2037-01-12

AI Technical Summary

Technical Problem

Remotely operated surgical systems face challenges in providing seamless haptic feedback transitions during control state changes, leading to unintuitive and disorienting experiences for surgeons due to abrupt force changes, especially when switching between different instruments or control modes.

Method used

Implementing a gradual transition of haptic feedback force vectors from one control state to another, allowing for a smooth shift in force magnitude and direction over time, maintaining consistency in tactile feedback during state changes.

Benefits of technology

Ensures an intuitive tactile experience for surgeons by reducing confusion and disorientation during control state transitions, enhancing user comfort and system usability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007590072000001
    Figure 0007590072000001
  • Figure 0007590072000002
    Figure 0007590072000002
  • Figure 0007590072000003
    Figure 0007590072000003
Patent Text Reader

Abstract

A surgical system is provided having an input device and a controller that receives control input from the input device and provides haptic feedback to the input device, the controller being configured to apply a gradual transition from a first haptic feedback profile on the input device to a second haptic feedback profile on the input device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Claiming priority This application claims the benefit of priority to U.S. Provisional Patent Application No. 62 / 277,820, filed January 12, 2016, which is incorporated herein by reference in its entirety.

[0002] The present invention relates to systems and methods for providing tactile feedback to an operator of a surgical system, and more particularly to systems and methods for maintaining an intuitive tactile profile for a user as control states of the system change. [Background technology]

[0003] Teleoperated surgical systems are often intended to increase a surgeon's precision and / or reduce patient trauma during a medical procedure. In such systems, the surgeon interacts with an input device (sometimes called a "master" or "master controller") to control surgical instruments actuated by drive mechanisms such as motors. Because the surgeon does not directly manipulate the surgical instruments, it may be beneficial to provide the input device with haptic feedback that indicates or replicates the forces felt by the surgical instruments. To provide a good user experience, the surgeon would ideally experience a seamless haptic experience with state changes and configuration changes of the system. However, this may be difficult to achieve, especially when a given input device is used in multiple different control states. For example, a single input device may be used to control multiple different instruments (requiring switching control between these different instruments), to control the overall positioning of the surgical system (e.g., changing the viewpoint of an endoscope looking into an anatomical region of interest), to change the settings of the surgical system, and / or may simply be isolated from any control effects.

[0004] Simply activating / deactivating / changing haptic feedback in response to such control state changes can result in jerky and unintuitive interactions. If force is being applied to the user in one state and not in the next, it can be confusing to the user to immediately change from full force to no force at all. For example, transitioning from controlling an instrument with haptic feedback to controlling an instrument without haptic feedback (e.g., an endoscope), or otherwise changing instruments (i.e., removing an instrument from a manipulator), exiting from following steps, etc. When transitioning into a control state without haptic feedback, such as ending a subsequent step (i.e., disabling control of a command by the input device) or arm swap (i.e., changing surgical instrument / arm / manipulator under control of the input device), the immediate loss of haptic feedback will feel like an unexpected loss of resistance to the user.

[0005] Users may also be confused as to whether force feedback is immediately enabled when going from an instrument rendering no (or low) force to an instrument with a (high) force. This can happen, for example, when going back from camera control to an instrument with force feedback. It can also happen when a user first takes control of an instrument (for example, going from one progression state to the next with the instrument).

[0006] Another problem occurs when a user changes control states to intermediate states that remain associated with a particular instrument (e.g., clutching, head-in UI, etc.) but where no control is applied. Turning force feedback off during the intermediate state and back on again when returning to direct control of the instrument is not ideal for the user. Most of the time, these intermediate control states are short-lived, so rapidly turning force on and off can be problematic.

[0007] Note that the force feedback presented to the user may be the sum of feedback from sensors, feedback from algorithms, feedback from user interface cues, feedback from collision detection, etc. Summary of the Invention

[0008] To mitigate the discomfort of abrupt force transitions when changing control states, a gradual transition can be provided from the original force exerted on the input device (and therefore on the user) to a level appropriate for the new control state.

[0009] In some embodiments, the haptic feedback force vector transitions (over some non-zero time, in some direction and / or magnitude) from force A to force B. The magnitude of force A may be greater or less than force B (or may be equivalent in some embodiments and the transition is in only one direction). The profile of the gradual transition from the magnitude of force A to the magnitude of force B may be any profile that occurs over time, including a linear transition or a curvilinear transition, and may include a discontinuity (e.g., a step) as part of the transition.

[0010] In some embodiments, for control state changes involving entry into an intermediate mode in which the user remains in a state associated with a particular instrument, the force feedback may remain at the last level at which the user stopped directly controlling the instrument. For example, when the control state transitions from instrument control (subsequent step) to non-instrument control (exiting subsequent step), the haptic feedback at the associated input device may be maintained at its current level for at least a period of time. In various embodiments, after such a period of stability (no fluctuations), a gradual transition to a reduced or no force state may subsequently be applied.

[0011] It is to be understood that both the foregoing summary and the following detailed description are exemplary and explanatory in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the disclosure, in that regard, additional aspects, features, and advantages of the present disclosure will become apparent to those skilled in the art from the following detailed description. [Brief description of the drawings]

[0012] [Figure 1A] 13A-13C illustrate a method for providing intuitive tactile feedback to a user of a surgical system during control state changes in accordance with various embodiments of the present invention. [Figure 1B] 13A-13C illustrate a method for providing intuitive tactile feedback to a user of a surgical system during control state changes in accordance with various embodiments of the present invention. [Figure 2A] 5A-5C illustrate exemplary haptic force vector transitions during control state changes in accordance with various embodiments of the present invention. [Figure 2B] 11A-11C illustrate exemplary haptic force feedback transition profiles, in accordance with various embodiments of the present invention. [Figure 3A] 1A-1C illustrate an exemplary surgical system that provides transitions in haptic force feedback in response to changes in control state, in accordance with various embodiments of the present invention. [Figure 3B] 1A-1C illustrate an exemplary surgical system that provides transitions in haptic force feedback in response to changes in control state, in accordance with various embodiments of the present invention. [Figure 3C] 1A-1C illustrate an exemplary surgical system that provides transitions in haptic force feedback in response to changes in control state, in accordance with various embodiments of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Aspects of the present disclosure are best understood from the following detailed description when read in conjunction with the accompanying drawings. It is emphasized that, according to common practice in the industry, various features are not drawn to scale. In fact, dimensions of various features may be appropriately increased or decreased for clarity of discussion. Further, the present disclosure may repeat reference numerals and / or letters in various instances. This repetition is for the purposes of brevity and clarity and does not in itself dictate a relationship between the various embodiments and / or configurations discussed.

[0014] In the following detailed description of aspects of the present invention, numerous specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present invention. Also, to avoid unnecessary repetition of description, one or more components or operations described in accordance with an exemplary embodiment may be used or omitted as applicable from other exemplary embodiments.

[0015] By providing a gradual transition between force feedback profiles when the control state of the surgical system changes, an intuitive haptic experience can be maintained for the user of the surgical system.

[0016] 1A illustrates an exemplary method for providing such a graduated haptic transition. In a haptic feedback providing step 110, the surgical system allows a user (e.g., a surgeon) to control a surgical instrument (and / or other elements of the surgical system such as a robotic arm, a setup structure, or a positioning element such as a boom or cart) via an input device(s) (lever(s), grasper(s), joystick(s), and / or any other structure capable of receiving user input) and then provides force feedback to that input device based on a desired haptic feedback profile (a set of one or more haptic feedback effects that at least partially replicate or re-present the physical experience of a real or virtual / modeled interaction). The haptic feedback profile may be based on any haptic input model, such as forces sensed on an instrument (e.g., interaction with tissue or other instruments) or on a robotic arm (e.g., collision of the arm with a structure or staff), user guidance (e.g., haptic detents, fences, other profiles to provide guidance to the user to move the input device(s) along a desired path or trajectory), and user interface (UI) elements (e.g., presenting a virtual handle or steering wheel to the user). The transformation applied to the desired haptic feedback profile to generate the haptic feedback on the input device may be from a direct reproduction of the haptic feedback profile, to a scaling of the haptic feedback profile, to applying a non-linear modification of the haptic feedback, or any other transformation (e.g., scaling of forces that vary depending on one or more other factors, such as instrument state / speed, view magnification, etc.). The haptic feedback in step 110 may also include forces calculated based on additional sensors of the surgical system, such as position sensors, buttons, or contact sensors.

[0017] Next, in a control mode change step 120, a change in control mode in the surgical system occurs (typically in response to a user command / action, but in other instances in response to an external command such as a system timer or alarm, or in response to a third party action / command) such that the input on the input device no longer has the same effect on the surgical instrument. For example, in a surgical system that includes both a surgical instrument and a camera (e.g., an endoscope), the control state may change from a state in which the input device is controlling the surgical instrument to a camera control mode in which the input device is controlling the camera's viewpoint / view of the surgical site. If the input device is providing haptic feedback to the user when such a change in control state occurs, immediately removing the haptic feedback in response to the change may result in an abrupt loss of force feedback on the input device, which may be confusing and disorienting to the user.

[0018] Thus, in apply gradual haptic feedback transition step 130, a gradual transition is performed from the original haptic feedback profile to the desired haptic feedback profile for the new control state. In other words, the transition from the original haptic feedback profile to the desired haptic feedback profile includes at least one intermediate stage that reduces the abrupt change in haptic feedback that would result from an immediate switch between the old and new haptic profiles corresponding to a switch between control states.

[0019] In most cases, this transition from an original haptic feedback profile to a desired (new) haptic feedback profile will involve adjustments to the force feedback vector expressed on the input device. A New power from F B 4 illustrates an exemplary haptic force feedback vector transition from force F A is the force F B It may be greater or less (or equivalent in some embodiments, the transition is in one direction only). A New power from F BThe gradual transition to may take any path, including a linear transition or a curvilinear transition, and may include discontinuities (e.g., steps) as part of the transition. For example, FIG. 2B shows several example gradual transitions between haptic feedback profiles that may occur in response to step 120, including straight linear scaling (#1), monotonic scaling (#2), sawtooth bidirectional scaling (#3), and curvilinear multidirectional scaling (#4). In some examples, the transition may move the force magnitude through a zero force state (e.g., #3 and #4). In various other embodiments, the force F A From force F B The transition to the intermediate direction vector is generated by a force F A and F B or adding a scaled version of the force F over a period of time. A and then force F B Several algorithms can be used, including keeping the .lambda. and .lambda. aligned.

[0020] Note that in some embodiments, "phantom" haptic feedback may be maintained when changing between a control state that has haptic feedback and a control state that normally does not produce haptic feedback. Figure 1B shows an exemplary method of providing this type of stable transition.

[0021] As discussed above with respect to FIG. 1A, in a provide haptic feedback step 110, the surgical system is in a control state that provides force feedback to an input device based on haptic model inputs, such as forces sensed at an instrument and / or other system structures, or a haptic feedback profile based on virtual or synthetic interactive elements (e.g., user interface elements, virtual fences, and / or virtual control features, among others). Also as discussed above, the haptic feedback in step 110 can also include forces calculated based on additional sensors of the surgical system, such as position sensors, buttons, or contact sensors. And, in a change control mode step 120, the control state of the surgical system can be changed to, among others, a system setup mode, a change arms mode (i.e., to switch control to a different instrument on the surgical system), or to exit from a subsequent step. The control state may change to one where haptic feedback is not relevant (or different from the original haptic feedback profile), such as a "following" mode (i.e., simply disabling the control link between the input device and the surgical instrument).

[0022] Next, in step 125 of maintaining haptic feedback, the haptic feedback profile remains unchanged even as the control state of the system changes. Such consistency of haptic feedback can sometimes give the user a more intuitive experience compared to changing or eliminating the feedback altogether. For example, when changing from an instrument control state (with haptic feedback) to a control state in which an input device is used to change a system parameter (e.g., select a menu option) and thus does not have any logical haptic feedback relationship to the force of the surgical instrument, it may be beneficial to simply maintain the feedback control profile from the instrument control state even during the system setting control state, so that when the control state returns to the instrument control state, the user is not surprised by the sudden appearance of force feedback on the input device. Thus, although the haptic feedback profile of the new state technically does not match the control action performed on the input device for the new state, that haptic mismatch can actually give a more consistent user experience.

[0023] It should be noted that in some embodiments, step 125 may include overlaying or adding additional haptic feedback associated with the new control state to the original haptic feedback profile. In the system setting example described above, certain haptic feedback effects, such as haptic "clicks" or "bumps," may be associated with the system setting control state when the setting is changed or selected. In some embodiments, such haptic effects may be provided along with the original force feedback from the previous control state.

[0024] Further, note that in some embodiments, maintaining the haptic feedback in step 125 may be followed by applying a gradual haptic feedback transition in step 130, as described above with respect to Figure 1 A. In such a case, the haptic feedback profile would initially not be changed after the change of control state in step 120, but would eventually undergo a gradual transition to the new haptic feedback profile.

[0025] 3A shows a block diagram of a surgical system 300 incorporating haptic feedback and a means for providing a user with an intuitive haptic experience as described above with respect to FIG. 1A and / or FIG. 1B. The surgical system includes an instrument 310 (e.g., forceps, cutter, retractor, vessel sealer, needle driver, catheter, etc.) for performing a surgical task, an input device 330 (e.g., lever(s), grasper(s), joystick(s), or any other structure capable of receiving user input, etc.) for receiving input from a user (e.g., surgeon), and a controller 320 that receives input commands from the input device 330 and accordingly controls the operation of the instrument 310 via a steering structure 313 and provides commands to a haptic feedback actuator 340 to provide haptic feedback to the input device 330 according to a desired haptic feedback profile. In various embodiments, the steering structure 313 can include any number of systems and structures for manipulating, positioning, actuating, or otherwise controlling the behavior of the instrument 330, including robotic arm(s) / manipulator(s), setup structure(s), and / or positioning elements such as boom(s) or cart(s), among others. The controller 320 can include any combination of hardware, software, firmware, and other modalities for generating, managing, controlling, and executing the operations described herein. In various embodiments, the controller 320 may be integrated with the instrument 310, the input device 330, and / or separate control hardware (e.g., a stand-alone processing device or computer platform).

[0026] For illustrative purposes, Figure 3A shows the surgical system 300 in control state A, with the input device 330 controlling the instrument 310. As shown in Figure 3A, the surgeon uses the input device 330 to grasp and retract a portion of tissue 390 with an end effector 311 on the shaft 312 of the instrument 310. This results in a downward force Fs on the end effector 311, in response to which the controller 320 generates a haptic feedback force Fs (e.g., as described with respect to step 110 above). A to the input device 330. In this manner, the surgeon will "feel" the resistance provided by the tissue 390 as it is being retracted.

[0027] It should be noted that this description of control state A is for illustrative purposes only, as this initial control state A could be any state of the surgical system 300. For example, the haptic feedback profile force FA is represented as derived from a force Fs sensed at the end effector 311 of the instrument 310 for illustrative purposes, and in various other embodiments, a force FS, such as an interaction at the shaft 312 or any other element of the manipulation structure 313 (e.g., pressure of an arm against a structure or staff), could be sensed anywhere where corresponding haptic feedback in the input device 330 would be beneficial.

[0028] In various other embodiments, the force FS can be defined according to non-physical parameters, such as the synthetic interactive elements described above. For example, in some embodiments, the surgical system 300 can include an optional display 350 (e.g., monitor(s), head-in-viewer(s), projector, video glasses / helmet(s), and / or any other graphical presentation element). In various embodiments, the display 350 can present virtual or synthetic elements 361 that can be interacted with via the input device 330. In some embodiments, the synthetic elements 361 can be used as an auxiliary interface to interact with the physical elements of the surgical system 300. For example, as shown in FIG. 3A, the synthetic elements 361 can be virtual handles or knobs that can be "grabbed" and dragged with the input device 330 to reposition the instrument 310 at the surgical site. In other embodiments, the synthetic elements 361 can provide purely virtual interaction elements, such as dials, toggles, levers, or any other structures to control the surgical system 300. In either case, by generating a haptic feedback profile based on a model force FS1 associated with interaction with the synthetic element 361 (e.g., the radially outward resistive force that occurs when gripping a round knob), the controller 320 can then attempt to impart an appropriate haptic feedback profile force FA to the input device 330.

[0029] In various other embodiments, the surgical system 300 can provide guidance to the user regarding the movement of the instrument 310 and / or the input device 330. For example, a desired movement of the instrument 310 (e.g., a targeted or safe dissection path, a desired retreat movement, or any other beneficial articulation movement) can optionally be defined as a trajectory 362. By generating a haptic feedback profile based on a model force FS2 associated with maintaining the position of the instrument 310 along the trajectory 362 (e.g., an inward desired force generated upon deviation from the trajectory 362), the controller 320 can then generate an appropriate haptic feedback profile force F Ato the input device 330.

[0030] Next, the exemplary control state shifts control by the input device 330 from its existing element (e.g., the original instrument 310, such as a retractor) to a new element (e.g., instrument 310(2), such as a needle driver for suturing tissue 390) by changing to a new control state B shown in FIG. 3B (e.g., as described with respect to step 120 above). The change in control state shifts control by the input device 330 from its existing element (e.g., the original instrument 310, such as a retractor) to a new element (e.g., instrument 310(2), such as a needle driver for suturing tissue 390). The change in control state is reflected by a force Fs (tactile feedback Fs) on the original element (e.g., instrument 310) (e.g., as described with respect to step 120 above). B Thus, over time (t2-t1), the haptic force feedback F A From F B A gradual transition to the force feedback F is performed as described with respect to step 130 above. As previously described, for a gradual transition, the user of the input device 330 may receive a gradual transition to the force feedback F, which occurs when a force feedback transformation is applied to the sensed force switched simultaneously with a change in control state. A From F B For example, a gradual transition ensures that haptic feedback representing retracting supporting tissue does not disappear immediately, or the sensation of holding a virtual knob or handle does not disappear immediately.

[0031] In another embodiment, another exemplary control state change from control state A in FIG. 3A to new control state C shown in FIG. 3C shifts control of the input device 330 from the instrument 310 (or any other control state with haptic feedback) to control of the system settings selection interface 321 (or any other control state without haptic feedback (or different haptic feedback)). Here, the haptic feedback profile is maintained even though the control state is changed, so that the feedback force on the input device 330 is the force F even though the settings selection menu is not a force generating environment. AAs a result, the user is not surprised by the sudden disappearance of force feedback on the input device (or the sudden reappearance of force feedback after a selection menu is exited).

[0032] As mentioned above, in some embodiments, the retained haptic force F A is overlaid with additional haptic feedback associated with the new control state, such as a haptic "click" or "bump," when a setting is changed or selected. Further, as described above, the maintained haptic feedback can be followed by a next gradual transition to a new haptic feedback profile.

[0033] While certain illustrative embodiments of the invention have been described and illustrated in the accompanying drawings, such embodiments are merely illustrative of the invention and are not intended to limit the broad invention, and the embodiments of the invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art.

Claims

1. A surgical system, comprising: An input device; a controller for receiving control input from the input device and providing haptic feedback to the input device; the controller is configured to apply a gradual transition from the first non-zero haptic force feedback profile on the input device to the second non-zero haptic force feedback profile on the input device when the surgical system transitions from a first control state associated with a first non-zero haptic force feedback profile of a first instrument to a second control state associated with a second non-zero haptic force feedback profile of a second instrument; the gradual transition is time dependent and includes a zero haptic force state between the first non-zero haptic force feedback profile and the second non-zero haptic force feedback profile, one of the first and second non-zero haptic force feedback profiles exhibiting a decreasing function around the zero haptic force and the other of the first and second non-zero haptic force feedback profiles exhibiting an increasing function around the zero haptic force. Surgery system.

2. The surgical system of claim 1 , wherein the gradual transition comprises at least one of a linear scaling, a curvilinear scaling, and a directional correction of the first non-zero haptic force feedback profile.

3. The surgical system of claim 1 or 2, wherein the gradual transition comprises scaling a force magnitude of the first non-zero haptic force feedback profile.

4. The surgical system of claim 1 or 2, wherein the gradual transition comprises changing a direction of a force vector of the first non-zero haptic force feedback profile.

5. The surgical system of claim 1 , further comprising an instrument, and the first control state includes an instrument control mode of the instrument.

6. The surgical system of claim 1 or 5, wherein the first and second control states include different modes among an instrument control mode, a clutch mode, an instrument exchange mode, a camera control mode, a system setting mode, an arm exchange mode, a subsequent step completion mode, and an interactive virtual element manipulation mode.

7. The surgical system of claim 1 or 2, wherein the gradual transition comprises maintaining the first non-zero haptic force feedback profile on the input device for a predetermined period of time.

8. The surgical system of claim 1 or 2, wherein the controller is further configured to superimpose at least one haptic effect on the gradual transition.

9. The surgical system of claim 1 or 2, wherein the controller is further configured to restore the first non-zero haptic force feedback profile in the input device when the surgical system returns from the second control state to the first control state.

10. 1. A method, comprising: generating a first non-zero haptic force feedback profile on an input device of a surgical system during a first control state of a first instrument of the surgical system; generating a second non-zero haptic force feedback profile at the input device of the surgical system during a second control state of a second instrument of the surgical system; applying a gradual transition from the first non-zero haptic force feedback profile to the second non-zero haptic force feedback profile at the input device of the surgical system upon detecting a change from the first control state of the surgical system to the second control state of the surgical system; the gradual transition is time dependent and includes a zero haptic force state between the first non-zero haptic force feedback profile and the second non-zero haptic force feedback profile, one of the first and second non-zero haptic force feedback profiles exhibiting a decreasing function around the zero haptic force and the other of the first and second non-zero haptic force feedback profiles exhibiting an increasing function around the zero haptic force. method.

11. The method comprises: controlling a surgical instrument according to input received at the input device while the surgical system is in the first control state; receiving sensed force data from the surgical instrument while the surgical system is in the first control state; The method of claim 10 , wherein the first non-zero haptic force feedback profile is based on the sensed force data.

12. The method of claim 10 , wherein the first non-zero haptic force feedback profile is based on an interactive virtual element generated by a controller.

13. The method of claim 10 , wherein the gradual transition comprises scaling a force magnitude of the first non-zero haptic force feedback profile.

14. The method of claim 10 , wherein the gradual transition comprises changing a direction of a force vector of the first non-zero haptic force feedback profile.

Citation Information

Patent Citations

  • Haptic interaction method in augmented reality and system thereof

    JP2009187550A

  • Frame mapping and force feedback methods, apparatus, and systems

    JP2010524548A

  • Ultrasonic surgical system and method

    US20070239028A1

  • System and method for automated minimally invasive therapy using radiometry

    US20110295247A1

  • Surgical robot and surgical robot control method

    US20140142592A1