User interface interactive elements with associated degrees of freedom of motion

By using haptic plane constraints and dynamic cursor position determination techniques in robot-assisted medical systems, the problem of low communication efficiency between operators and multi-degree-of-freedom interaction elements is solved, and more efficient and accurate interaction operations are achieved.

CN118946325APending Publication Date: 2024-11-12INTUITIVE SURGICAL OPERATIONS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380030189.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-01
Filing Date
2023-03-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In the robot-assisted medical process, it is difficult for operators to efficiently communicate effectively with graphical user interface interactive elements with different degrees of freedom of movement, resulting in limited operational efficiency and accuracy.

Method used

By displaying the interactive panel on the display device and using the haptic plane to constrain the movement of the operator interface device, determine the cursor position and element type, dynamically add or remove tactile constraints to accommodate different types of interactive elements, allowing the operator to be natural and non-existent Move the cursor in a slit.

Benefits of technology

It improves the communication efficiency and accuracy of the operator's interactive elements with the graphical user interface, enhances the ability to control multi-dimensional interactive elements, and reduces operational complexity and risk of misoperation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946325A_ABST
    Figure CN118946325A_ABST
Patent Text Reader

Abstract

A method may include displaying an interaction panel on a display device. The interaction panel has a display reference frame and includes a plurality of interaction elements. The method may also include constraining motion of the operator interface device to translational motion in a haptic plane with a haptic plane constraint while displaying the interaction panel, and determining a cursor position in the display frame of reference corresponding to a position of a selected portion of the operator interface device in the operator frame of reference. The method may also include determining an element type of a first interaction element of the plurality of interaction elements in which the cursor position is located, and removing haptic plane constraints on the operator interface device if the element type is a three-dimensional rotation type.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-referenced applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 326,652, filed on April 1, 2022, and entitled “User Interface Interaction Elements With Associated Degrees of Freedom of Motion,” which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for presenting graphical user interface interactive elements that correspond to degrees of freedom of motion available to an input interface device. Background Art

[0004] During a medical procedure that may be performed using a robot-assisted minimally invasive medical tool, a clinician or other operator may be provided with a graphical user interface. The graphical user interface may include an image of a three-dimensional field of view of a patient's anatomy, and may include interactive elements (e.g., menus, buttons, switches, or other elements that allow a user to access information or control system functions). Improved systems and methods are needed to present graphical user interface elements that enable an operator to efficiently and effectively interface with interactive elements having different degrees of freedom of motion. Summary of the invention

[0005] Embodiments of the invention are best summarized by the claims accompanying the description.

[0006] In one example embodiment, a method may include displaying an interaction panel on a display device. The interaction panel has a display reference frame and includes a plurality of interactive elements. The method may also include constraining movement of an operator interface device to a translational movement in a tactile plane using a tactile plane constraint when displaying the interaction panel, and determining a cursor position in the display reference frame, the cursor position corresponding to a position of a selection portion of the operator interface device in the operator reference frame. The method may also include determining an element type of a first interactive element of the plurality of interactive elements where the cursor position is located, and if the element type is a three-dimensional rotation type, removing the tactile plane constraint on the operator interface device.

[0007] In another example, a medical device system may include a control system, a display device, and an operator interface device in communication with the control system. The control system may include a processor and a memory, the memory including machine-readable instructions, which, when executed by the processor, cause the control system to display an interactive panel on the display device. The interactive panel may have a display reference system and include multiple interactive elements. The instructions may also cause the control system to use a tactile plane constraint to constrain the movement of the operator interface device to a translational movement in the tactile plane when displaying the interactive panel. The instructions may also cause the control system to determine a cursor position in the display reference system, the cursor position corresponding to the position of a selected portion of the operator interface device in the operator reference system, and determine the element type of a first interactive element among the multiple interactive elements where the cursor position is located. If the element type is a three-dimensional rotation type, the instructions may also cause the control system to remove the tactile plane constraint on the operator interface device.

[0008] It should be understood that the above general description and the following detailed description are exemplary and illustrative in nature, and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In this regard, additional aspects, features and advantages of the present disclosure will be apparent to those skilled in the art based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An operator of a medical system is shown in an operator environment.

[0010] Figure 2 A graphical user interface including interactive elements according to some examples is shown.

[0011] Figure 3 is a flow chart illustrating a method for changing a constraint on an operator interface device based on a displayed interactive element, according to some examples.

[0012] Figure 4 is a flow chart illustrating a method for manipulating a three-dimensional interactive element according to some examples.

[0013] Figure 5 is a flow chart illustrating a method for manipulating a one-dimensional rotational interactive element according to some examples.

[0014] Figure 6 is a flow chart illustrating a method for manipulating a two-dimensional translation interactive element according to some examples.

[0015] Figure 7 is a flow chart illustrating a method for manipulating a one-dimensional translation interactive element according to some examples.

[0016] Figure 8Schematic diagrams of medical systems are shown according to some examples.

[0017] Fig. 9 Based on some examples Figure 8 A perspective view of a manipulator assembly of a medical system.

[0018] Fig.10 is a front view of an operator console in a robotic-assisted medical system according to some examples.

[0019] Fig.11 Based on some examples Fig.10 A perspective view of an exemplary gimbaled device of an operator console for controlling one or more coupled arms and / or tools of a robotic-assisted medical system.

[0020] Embodiments of the present disclosure and their advantages are best understood by reference to the following detailed description. It should be understood that like reference numerals are used to identify like elements shown in one or more of the accompanying drawings, wherein the illustrations therein are for the purpose of illustrating embodiments of the present disclosure, rather than for the purpose of limiting embodiments of the present disclosure. DETAILED DESCRIPTION

[0021] In a robot-assisted medical procedure, an operator performing the medical procedure can use a graphical interactive panel including interactive elements (e.g., a menu or other virtual tools) to access information or control a medical system. The operator can use an operator interface device to control the robot-assisted instrument used in the medical procedure and communicate with the interactive elements on the graphical interactive panel. In some examples, the operator interface device can have six degrees of freedom (e.g., three translational degrees of freedom and three rotational degrees of freedom) for controlling the robot-assisted instrument. When using the operator interface device to interact with the graphical interactive panel, the operator interface device can be constrained to limit the movement to a translational movement in a tactile plane. However, some interactive elements on the graphical interactive panel can be manipulated on more than two or less than two translational degrees of freedom. In order to allow the operator to move naturally and seamlessly between different types of interactive elements, tactile constraints can be added or removed based on the determination of the cursor position and the element type associated with each interactive element at the cursor position.

[0022] Figure 1 A clinician or operator O is shown in an operator environment having an operator reference frame X. O , Y O , Z O. Operator O can exercise control over a right-hand operator interface device 10 and a left-hand interface device 12 to control tools or virtual interactive elements visible on the display system 14. In some examples, the interface devices 10, 12 can be part of a robotic-assisted medical system (e.g., interface device 336), such as the medical system 310 described in more detail below. In some examples, the interface devices 10, 12 can communicate wirelessly with the control system. In other examples, the interface device can be grounded and directly coupled to an operator console, a control system, or other components in the operator's reference frame. The selection portion 16 of the interface device 10 can correspond to the location where the tip portion of the operator's index finger is located. In other examples, the selection portion can correspond to other portions of the operator interface device. In some examples, when the robotic-assisted medical system is in an instrument control mode or an instrument "follow" mode, the interface devices 10, 12 can be movable in six degrees of freedom to provide corresponding movement to a specified tool in the anatomical environment. In some examples, the robotic-assisted medical system can exit the follow mode and enter an assist mode in which one or both of the interface devices 10, 12 can be used to interact with virtual elements (e.g., menus, anatomical models, or system controls) displayed on the display system. The interface device(s) can include one or more of any number of various input devices (e.g., hand grips, joysticks, trackballs, data gloves, trigger guns, manual controllers, touch screens, body motion or presence sensors, and other types of input devices). A viewing cone 17 around a central viewing axis 18 can intersect the display system.

[0023] Figure 2 A graphical user interface 50 is shown displayed on a display system 52 (e.g., display system 14). In this example, the graphical user interface 50 may include an image 54 of a field of view provided by an imaging device (e.g., an endoscopic imaging system 315 described below) within an anatomical environment in a patient's anatomy. In this example, the image 54 may be a three-dimensional stereoscopic image, but in other examples, the image may be a two-dimensional image. The graphical user interface 50 may have an image reference frame X. I , Y I , Z I . Operator reference frame X O , Y O , Z O Can be registered to the image reference system X I , Y I , Z I. This alignment of the reference frame allows the operator's hand movements in each cardinal direction to correspond to movements within the image. In order to access additional information about anatomical structures, system functions and / or system information, the medical system operator can exit the follow mode of the medical system and enter the assist mode. In the assist mode, the graphical user interface 50 can display a cursor 58 and an interactive panel 56 displayed above the field of view image 54. The cursor 58 can track the movement of a selection portion of the operator interface device (e.g., the selection portion 16 of the device 10 where the user's index finger is positioned). The interactive panel 56 may include movable or manipulable interactive elements, which can each be characterized by a predetermined element type based on, for example, the type of movement that the interactive element can perform. For example, the interactive element 60 can be a three-dimensional (3D) rotation type interactive element that is rotatable in three rotational directions. In this example, the interactive element can be a 3D global model of an anatomical organ (e.g., a kidney). The 3D model interactive element can be rotatable in up to three directions around the center of mass 62. The interactive element 64 can be a one-dimensional translation type interactive element, such as a single linear direction (e.g., X I Interactive element 66 may be a one-dimensional rotational type interactive element, such as a dial that is manipulable in a single rotational direction. Interactive element 68 may be a two-dimensional translational type interactive element, such as a two-dimensional menu that is manipulable in two translational directions. In other examples, the interactive elements may be movable in other combinations of translational and / or rotational directions.

[0024] In an assistive mode of the medical system displaying the interactive panel 56, the movement of an operator interface device (e.g., the operator interface device 10) may be mapped to the movement of the cursor 58. When in the assistive mode, tactile plane constraints may be placed on the operator interface device 10 to constrain the six-degree-of-freedom operator interface device to degrees of freedom of motion corresponding to translational motion along the tactile plane (e.g., degrees of freedom corresponding to X and Y, respectively). I and Y I The direction corresponding to X O and Y O direction). Constraints may be used as force feedback on the operator interface device 10 to limit the degrees of freedom of motion that are not used for motion along the tactile plane (e.g., the three rotational degrees of freedom and the Z-direction translational direction). For example and with reference to Fig.11, the gimbal orientation degrees of freedom of the operator interface device can be locked to prevent movement in unused degrees of freedom. The tactile plane can be used as a "virtual mouse pad" that promotes hand stability and intuitive correspondence between two-dimensional input movement and resulting cursor 58 movement. In some examples, the tactile plane can be adjustable to remain generally perpendicular to the central viewing axis 18, so that if the viewing angle between the operator and the display device changes (e.g., based on ergonomic adjustments), the tactile plane can also change to maintain the same orientation.

[0025] Figure 3 1 is a flow chart showing a method 100 for changing constraints on an operator interface device based on displayed interactive elements. The methods described herein are shown as a set of operations or processes and continue to be described with reference to the attached figures. Not all of the processes shown may be performed in all embodiments of the method. Additionally, one or more processes not explicitly shown may be included before, after, between, or as part of the processes shown. In some embodiments, one or more processes may be implemented at least in part in the form of executable code stored on a non-transitory, tangible, machine-readable medium, which, when run by one or more processors (e.g., a processor of a control system), may cause the one or more processors to perform one or more processes. In one or more embodiments, these processes may be performed by a control system.

[0026] refer to Figure 3 At process 102, an interaction panel (e.g., interaction panel 56) may be displayed on a display device (e.g., display system 52). At process 104, the movement of the operator interface device may be constrained to the tactile plane. Process 104 may include process 105 of constraining the movement of the operator interface device to translational movement in the tactile plane. For example, the movement of the operator interface device 10 may be constrained by force feedback to allow movement in a virtual plane in the operator reference frame in a direction X. O , Y O The translation movement on the virtual plane corresponds to the image reference system X I , Y I , Z I The operator interface device 10 is moved in the translation direction Z OThe movement on the operator interface device 10 may be tactilely restricted to prevent the operator from perceiving that the operator interface device 10 is piercing or lifting off the tactile plane corresponding to the interactive panel 56. Tracking movement of a selection portion of the operator interface device (e.g., the portion of the selection portion 16 that engages the operator's index finger) may be displayed as corresponding cursor movement on the interactive panel. Process 104 may also include an optional process 106 in which rotational movement of the operator interface device is constrained. For example, movement of the operator interface device 10 may be constrained by force feedback to prohibit rotational movement. Implementing the constraint may include locking the gimbal degrees of freedom of the operator interface device 10 to prevent rotational movement from exceeding the tactile plane. Thus, all movement in the tactile plane may be constrained to X O , Y O Translational movement in direction.

[0027] At process 108, a cursor position corresponding to a selection portion of the operator interface device may be determined. For example, a cursor 58 position corresponding to the selection portion 16 of the operator interface device 10 may be determined relative to the interaction panel 56. The cursor position may be associated with the interactive element at which the cursor is positioned. Figure 2 , for example, cursor 58 is positioned over and corresponds to interactive element 60 .

[0028] In process 110, determine the element type of the interactive element where the cursor position is located. For example, in the case where the cursor 58 is positioned above the interactive element 60, the interactive element 60 is identified by the cursor. The element type associated with the interactive element 60 can be identified. The element type can be predetermined based on the degree of freedom of motion associated with the interactive element. The interactive element 60 can have an element type of a three-dimensional (3D) rotation type, because the interactive element 60 is an anatomical model that can be manipulated on three rotational degrees of freedom around the center of mass 62 based on the movement of the operator interface device 10. The interactive element 64 can be identified as a one-dimensional translation type interactive element that can be manipulated in a single linear direction. The interactive element 66 can be identified as a one-dimensional rotation type interactive element that can be manipulated in a single rotation direction. The interactive element 68 can be identified as a two-dimensional translation type interactive element that can be manipulated in two translation directions. In other examples, the interactive element can be identified using other element types based on the motion or other interaction parameters of the interactive element.

[0029] If the interactive elements are layered in the display on the display device, the element type of the topmost interactive element at which the cursor is positioned can be determined. In some examples, multiple layered tactile planes are provided. The user can be allowed to access and control interactive elements on lower layers through the top tactile plane.

[0030] At process 112 , constraints on the operator interface device may be changed based on the identified element type. Figure 4-Figure 7 Methods 112A-112D are described, respectively, illustrating alternative examples for changing constraints on an operator interface device based on an identified element type.

[0031] refer to Figure 4 In method 112A, at optional process 150, if the element type is a 3D rotation type, the rotation constraints on the operator device (e.g., the constraints of process 106) can be removed. For example, in the case where the interactive element 60 is identified as a 3D rotation type interactive element, the rotation constraints on the operator interface device 10 can be removed to allow the operator to rotate the operator interface device about an axis perpendicular to the tactile plane (e.g., wrist rotation about the roll axis), while the tactile plane continues to be constrained to Z. O Removing the rotational constraints may include removing or relaxing a locking force on a gimbal of operator interface device 10 .

[0032] At process 152, a determination may be made as to whether a rotation signal has been received. For example, the rotation signal may indicate an operator's intent to manipulate the 3D rotatable interactive element 60. The rotation signal may be a selection input at the operator interface device 10, such as a pinch gesture, a button press, or any other gesture, prompt, or input at the operator interface device that indicates an operator's intent to rotate the interactive element. In some examples, the cursor may disappear from the display when the rotation signal is received.

[0033] At process 154, if a rotation signal was received at process 152, some or all of the haptic plane constraints may be removed. For example, the haptic planes that constrain motion on the operator interface device 10 may be removed, allowing the operator interface device 10 to move freely in six degrees of freedom (3 translations, 3 rotations). The motion of the interactive element 60 may directly track the motion of the operator interface device 10, thereby allowing the interactive element 60 (e.g., the anatomical model) to be moved in X-ray form. I , Y I , Z I The interactive element 60 is rotatable by the operator, and the operator can feel the sudden removal of the tactile plane constraint. The operator can feel the tactile feedback cue, suggesting to the operator that the underlying interface element is an interface element that can be manipulated using three-dimensional motion. In some examples, a visual cue such as an outline, highlight, glow effect, or other cue can be provided to indicate that the interactive element is rotatable.

[0034] At process 156, a determination may be made as to whether a rotational motion input has been received from the operator interface device. For example, a rotation of the operator interface device 10 may generate a rotational motion input. At process 158, in the event that a rotational motion input is received from the operator interface device, a corresponding rotational motion of the identified interactive element may be generated. For example, receiving a rotational motion input from the operator interface device 10 in the operator reference frame may generate a corresponding rotational motion of the interactive element 60 in the image reference frame. The interactive element 60 may be free to rotate and translate in six degrees of freedom according to the tracking motion of the operator interface device 10. In some examples, the rotational signal (e.g., a pinch gesture by the operator at the operator interface device) may be maintained for the entire duration of the rotational motion input.

[0035] When the rotation signal is terminated, the tactile plane constraint can be restored to the operator interface device. For example, terminating the pinch gesture can terminate the rotation signal and restore the tactile plane constraint. Unpinching the operator interface device can be used as a clutching gesture that allows the operator to adjust to a more comfortable hand position before pinching the operator interface device again to continue the rotational movement of the operator interface device and the corresponding rotational movement of the interactive element. The rotation constraint of process 106 can also be restored when the rotation signal is terminated and the cursor position moves away from the 3D rotation type interactive element.

[0036] refer to Figure 5 In method 112B, at process 170, if the element type is a one-dimensional ("1D") rotation type (e.g., a dial interface element), translational motion on the tactile plane may be limited. For example, in the case where the interactive element 66 is identified as a 1D rotation type interactive element, the operator interface device 10 may be limited to move in the X O , Y O , Z O Translational movement in direction.

[0037] At process 172, if the element type is a 1D rotation type, the rotation constraints on the operator device (e.g., the constraints of process 106) may be removed along the axis corresponding to the rotation axis of the interactive element. For example, where interactive element 66 is identified as a 1D rotation type interactive element, the rotation constraints on operator interface 10 may be removed to allow the operator to rotate the operator interface device about an axis perpendicular to the tactile plane (e.g., wrist rotation about the roll axis) while the X O , Y O , Z O Translational motion in the direction continues to be restricted, and rotational motion about axes along the tactile plane (eg, pitch and yaw directions about axes lying in the tactile plane) continues to be restricted.

[0038] At process 174, a determination may be made as to whether a rotational motion input has been received from the operator interface device. For example, rotation of the operator interface device 10 may generate the rotational motion input. At process 176, in the event that a rotational motion input is received from the operator interface device, a corresponding rotational motion of the identified interactive element may be generated. For example, receiving a rotational motion input from the operator interface device 10 in the operator reference frame may generate a corresponding rotational motion of the interactive element 66 in the image reference frame. The interactive element 66 may be freely rotated in a single degree of freedom according to the tracking motion of the operator interface device 10. When the rotational motion input is terminated and the cursor position moves away from the 1D rotation type interactive element, the rotational constraint of process 106 may be restored.

[0039] refer to Figure 6 In method 112C, at process 180, if the element type is a two-dimensional rotation type (e.g., a 2D menu interface element), the movement of the operator interface device may be constrained to allow only translational movement along the tactile plane. For example, in the case where the interactive element 68 is identified as a 2D translation type interactive element, the movement of the operator interface device 10 may be constrained to allow only translational movement along the tactile plane. O , Y O Movement in direction.

[0040] At process 182, a determination may be made as to whether a translational motion input has been received from the operator interface device. O , Y O In process 184, in the event that a translational motion input is received from the operator interface device, a corresponding translational motion of the identified interactive element may be generated. For example, receiving a motion input signal from the operator interface device 10 in the operator reference frame may generate a corresponding translational motion of the interactive element 68 in the image reference frame. The interactive element 68 may include menu columns and rows that are based on the tracking motion of the operator interface device 10 in the direction X. I , Y I becomes visible due to translational movement on the

[0041] refer to Figure 7In method 112D, at process 190, a determination may be made as to whether a selection signal has been received. For example, the selection signal may indicate an operator's intent to manipulate the 1D slider interactive element 60. The selection signal may be an input at the operator interface device 10, such as a pinch gesture, a button press, a mouse click, or any other gesture, prompt, or input at the operator interface device that signals an operator's intent to translate the interactive element. In some examples, the cursor may disappear from the display when the selection signal is received.

[0042] At process 191, if the element type is a one-dimensional translation type (e.g., a 1D slider interface element) and a selection signal is received, the movement of the operator interface device may be constrained to allow translational movement in a single direction along the tactile plane. For example, where the interactive element 64 is identified as a 1D translation type interactive element, the movement of the operator interface device 10 may be constrained to be in the X direction. O Optionally, one or more rotational constraints on the operator interface device (e.g., constraints on a universal joint of the operator interface device) may be removed, thereby allowing the operator to more naturally rotate while controlling the slider element.

[0043] At process 192, a determination may be made as to whether a translational motion input has been received from the operator interface device. O In process 184, in the event that a translational motion input is received from the operator interface device, a corresponding translational motion of the identified interactive element may be generated. For example, receiving a motion input from the operator interface device 10 in the operator reference frame may generate an X-axis translation of the interactive element 64 in the image reference frame. I The corresponding translational movement in the direction.

[0044] Figure 8-Figure 11 Together, an overview of a medical system 310 is provided that can be used, for example, for a medical procedure, including a diagnostic, therapeutic, or surgical procedure. The examples provided above of using an operator interface device to interact with an interactive element can be used in the context of a medical system 310. The medical system 310 is located in a medical environment 311. The medical environment 311 is located in a medical environment. Figure 8 311 is depicted as an operating room. In other embodiments, medical environment 311 may be an emergency room, a medical training environment, a medical laboratory, or some other type of environment in which any number of medical procedures or medical training procedures may be performed. In still other embodiments, medical environment 311 may include an operating room and a control area located outside of the operating room.

[0045] In one or more embodiments, the medical system 310 can be a robotically assisted medical system under the remote operator control of an operator (e.g., a surgeon, clinician, physician, etc.). In alternative embodiments, the medical system 310 can be under partial control of a computer programmed to perform a medical procedure or subroutine. In still other alternative embodiments, the medical system 310 can be a fully automated medical system that is under the full control of a computer programmed to perform a medical procedure or subroutine in conjunction with the medical system 310. An example of a medical system 310 that can be used to implement the systems and techniques described in the present disclosure is the daVevo Surgical manufactured by Intuitive Surgical, Inc. of Sunnyvale, California. Surgical systems.

[0046] like Figure 8 As shown, medical system 310 generally includes component 312, which can be mounted to or positioned near an operating table T where a patient P is positioned. Component 312 can be referred to as a patient side cart, a surgical cart, or a surgical robot. In one or more embodiments, component 312 can be a remotely operated component. The remotely operated component can be referred to as, for example, a remotely operated arm cart. Medical instrument system 314 and endoscopic imaging system 315 are operably coupled to component 312. Operator input system 316 allows operator O or other types of clinicians to view images of the surgical site or images representing the surgical site and control the operation of medical instrument system 314 and / or endoscopic imaging system 315.

[0047] The medical device system 314 may include one or more medical devices. In embodiments where the medical device system 314 includes multiple medical devices, the multiple medical devices may include multiple identical medical devices and / or multiple different medical devices. Similarly, the endoscopic imaging system 315 may include one or more endoscopes.

[0048] The operator input system 316 may be located at an operator's console, which may be located in the same room as the operating table T. In some embodiments, the operator O and the operator input system 316 may be located in a different room than the patient P or in an entirely different building. The operator input system 316 typically includes one or more interface devices (e.g., the operator interface device 10) for controlling the medical instrument system 314. The interface device(s) may include one or more of any number of various input devices (e.g., hand grips, joysticks, trackballs, data gloves, trigger guns, foot pedals, manual controllers, voice recognition devices, touch screens, body motion or presence sensors, and other types of input devices).

[0049] In some embodiments, the interface device(s) will be provided with the same degrees of freedom as the medical device(s) of the medical device system 314 to provide telepresence (which is the perception that the interface device(s) are integral with the instrument) to the operator so that the operator has a strong sense of direct control of the instrument (as if present at the surgical site). In other embodiments, the interface device(s) may have more or fewer degrees of freedom than the associated medical instrument and still provide telepresence to the operator. In some embodiments, the interface device(s) are manual input devices capable of movement in six degrees of freedom, and may also include an actuatable handle for actuating the instrument (e.g., for closing a grasping jaw end effector, applying electrical potentials to electrodes, delivering drug therapy, and actuating other types of instruments).

[0050] When operator O views the surgical site through operator input system 316, component 312 supports and manipulates medical instrument system 314. Images of the surgical site can be obtained by endoscopic imaging system 315, which can be manipulated by component 312. Component 312 can include endoscopic imaging system 315, and can similarly also include multiple medical instrument systems 314. The number of medical instrument systems 314 used at one time will generally depend on the diagnostic or surgical procedure to be performed and the space constraints in the operating room and other factors. Component 312 can include one or more non-servo control links (e.g., one or more links that can be manually positioned and locked in place, generally referred to as setting structures) and the kinematic structure of the manipulator. When the manipulator takes the form of a remotely operated manipulator, component 312 is a remotely operated component. Component 312 includes multiple motors that drive inputs on medical instrument system 314. In an embodiment, these motors move in response to commands from a control system (e.g., control system 320). The motor includes a drive system that, when coupled to the medical device system 314, can advance the medical device into a natural or surgically created anatomical hole. Other motorized drive systems can move the distal end of the medical device in multiple degrees of freedom, which can include three degrees of linear motion (e.g., linear motion along the X, Y, Z Cartesian axes) and three degrees of rotational motion (e.g., rotation around the X, Y, Z Cartesian axes). Additionally, the motor can be used to actuate an articulated end effector of the medical device to grasp tissue in the jaws of a biopsy device, etc. The medical device of the medical device system 314 may include an end effector having a single working member, such as a scalpel, a blunt blade, an optical fiber, or an electrode. Other end effectors may include, for example, tweezers, a grasper, scissors, or a clip applier.

[0051] The medical system 310 also includes a control system 320. The control system 320 includes at least one memory 324 and at least one processor 322 for implementing control between the medical device system 314, the operator input system 316, and other auxiliary systems 326, which may include, for example, imaging systems, audio systems, fluid delivery systems, display systems, lighting systems, steering control systems, irrigation systems, and / or suction systems. The clinician can circulate within the medical environment 311 and can access, for example, the components 312 during a setup procedure or view the display of the auxiliary system 326 from the patient's bedside.

[0052] Despite Figure 8 312, in some embodiments, the control system 320 may be completely contained within the component 312. The control system 320 also includes programming instructions (e.g., stored on a non-transitory computer readable medium) to implement some or all of the methods described in accordance with aspects disclosed herein. Figure 8 Although shown as a single block in the simplified schematic of , control system 320 may include two or more data processing circuits, with some processing optionally being performed on or near component 312 , some processing being performed at operator input system 316 , and so on.

[0053] Any of a variety of centralized or distributed data processing architectures may be employed. Similarly, the programming instructions may be implemented as a plurality of separate programs or subroutines, or they may be integrated into a variety of other aspects of the systems described herein (including remote operating systems). In one embodiment, the control system 320 supports wireless communication protocols such as Bluetooth, IrDA, HomeRF, IEEE 802.11, DECT, and wireless telemetry.

[0054] In some embodiments, the control system 320 may include one or more servo controllers that receive force and / or torque feedback from the medical device system 314. In response to the feedback, the servo controller transmits a signal to the operator input system 316. The servo controller (one or more) may also transmit a signal that instructs the assembly 312 to move the medical device system 314 (one or more) and / or the endoscopic imaging system 315, which extend to an internal surgical site within the patient's body via an opening in the body. Any suitable conventional or dedicated servo controller may be used. The servo controller may be separate or integrated with the assembly 312. In some embodiments, the servo controller and assembly 312 are provided as part of a teleoperated arm cart positioned near the patient's body.

[0055] The control system 320 can be coupled to the endoscopic imaging system 315 and can include a processor to process the captured images for subsequent display, such as on an operator's console or on another suitable display located locally and / or remotely, to the operator. For example, where a stereoscopic endoscope is used, the control system 320 can process the captured images to present a coordinated stereoscopic image of the surgical site to the operator. Such coordination can include alignment between opposing images and can include adjusting the stereo working distance of the stereoscopic endoscope.

[0056] In alternative embodiments, the medical system 310 may include more than one component 312 and / or more than one operator input system 316. The exact number of components 312 will depend on the surgical procedure and the space constraints within the operating room, among other factors. The operator input systems 316 may be collocated, or they may be positioned in separate locations. Multiple operator input systems 316 allow more than one operator to control one or more components 312 in various combinations. The medical system 310 may also be used for training and rehearsing medical procedures.

[0057] Fig. 9 312 is a perspective view of an embodiment of an assembly 312, which may be referred to as a patient side cart, a surgical cart, a teleoperated arm cart, a manipulator assembly, or a surgical robot. The assembly 312 shown provides manipulation of three surgical tools 330a, 330b, and 330c (e.g., a medical instrument system 314) and an imaging device 328 (e.g., an endoscopic imaging system 315) such as a stereoscopic endoscope for capturing images of a procedural site. The imaging device may transmit a signal to a control system 320 via a cable 356. Manipulation is provided by a teleoperation mechanism having a plurality of joints. The imaging device 328 and the surgical tools 330a-330c may be positioned and manipulated through an incision in the patient's body so that a kinematic remote center is maintained at the incision to minimize the size of the incision. When the distal ends of the surgical tools 330a-330c are positioned within the field of view of the imaging device 328, the image of the surgical site may include an image of the distal ends of the surgical tools 330a-330c.

[0058] Assembly 312 includes a drivable base 358. Drivable base 358 is connected to a telescopic column 357, which allows the height of arm 354 to be adjusted. Arm 354 may include a swivel joint 355 that both rotates and moves up and down. Each arm 354 may be connected to an orientation platform 353. Arm 354 may be marked to facilitate troubleshooting. For example, each arm 354 may be decorated with different numbers, letters, symbols, other identifiers or combinations thereof. Orientation platform 353 may be capable of 360 degree rotation. Assembly 312 may also include a telescopic horizontal cantilever 352 for moving orientation platform 353 in a horizontal direction.

[0059] In this example, each arm 354 is connected to a manipulator arm 351. The manipulator arm 351 can be directly connected to a medical instrument (e.g., one of the surgical tools 330a-330c). The manipulator arm 351 can be remotely operable or robot-assisted. In some examples, the arms 354 connected to the orientation platform 353 may not be remotely operable or robot-assisted. Instead, these arms 354 can be positioned as desired before the operator O begins to operate using the remotely operated components.

[0060] Endoscopic imaging systems (e.g., endoscopic imaging system 315 and imaging device 328) can be provided in various configurations, including rigid or flexible endoscopes. Rigid endoscopes include a rigid tube that houses a relay lens system for transmitting images from the distal end of the endoscope to the proximal end of the endoscope. Flexible endoscopes use one or more flexible optical fibers to transmit images. Digital image-based endoscopes have a "chip on the tip" design in which a distal digital sensor (e.g., one or more charge-coupled devices (CCDs) or complementary metal oxide semiconductor (CMOS) devices) stores image data. Endoscopic imaging systems can provide two-dimensional or three-dimensional images to a viewer. Two-dimensional images may provide limited depth perception. Three-dimensional stereoscopic endoscopic images can provide a more accurate depth perception for the viewer. Stereoscopic endoscopic instruments use stereo cameras to capture stereoscopic images of patient anatomical structures. Endoscopic instruments can be fully sterilizable assemblies in which the endoscope cable, handle, and shaft are rigidly coupled and hermetically sealed.

[0061] Fig.103 is a perspective view of an embodiment of an operator input system 316 at an operator's console. The operator input system 316 includes a head mounted display system 335 (e.g., display system 14) having a left eye display 332 and a right eye display 334 for presenting a coordinated stereoscopic view of the surgical environment enabling depth perception to the operator O. In other embodiments, the display system 335 may include one or more other types of displays. The display system 335 may present, for example, images captured by the imaging system 315 to display the endoscopic field of view to the operator. The endoscopic field of view may be enhanced by virtual or synthetic menus, indicators, and / or other graphical or textual information to provide additional information to the viewer.

[0062] The operator input system 316 also includes one or more interface devices 336, which in turn cause the component 312 to manipulate one or more instruments of the endoscopic imaging system 315 and / or the medical instrument system 314. As described above, in the auxiliary mode, the interface device can also control the movement of the interactive elements of the displayed interactive panel. The interface device 336 can provide the same degree of freedom as the instrument associated with it to provide remote presentation to the operator O, or provide a sense that the interface device 336 is integrated with the instrument so that the operator has a strong sense of direct control of the instrument. To this end, position, force and tactile feedback sensors (not shown) can be used to transmit the position, force and tactile sensations from the medical instrument (e.g., surgical tools 330a-330c or imaging device 328) back to the operator's hand through the input interface device 336. The interface device 339 is a foot pedal that receives input from the user's foot. Aspects of the operator input system 316, the component 312 and the auxiliary system 326 can be adjustable and customizable to meet the physiological needs, skill level or preferences of the operator O.

[0063] Reference now Fig.11 , which shows a perspective view of one embodiment of an interface device 336 having a finger assembly 337. The finger assembly 337 may correspond to the selection portion 16 of the interface device 10. Fig.11 The illustrated embodiment of interface device 336 may be used as interface devices 10, 12. Interface device 336 is a gimbal device that pivotably supports assembly 337, which may include a touch-sensitive handle to generate control signals for controlling assembly 302, robotic medical tools, and / or interactive elements.

[0064] The depicted interface device 336 includes first, second and third gimbal members 402, 404 and 406. The above-mentioned restraining force can be applied to constrain the movement of the operator interface device to prevent movement at one or more gimbal members. The touch-sensitive handle provided by the finger assembly 337 includes a tubular support structure 412, a first grip 414A and a second grip 414B. The first grip 414A and the second grip 414B are supported at one end by the structure 412. In some embodiments, the grip 414 may include a ring of material that helps to fix the physician's fingers in the appropriate position relative to the structure of the grip. Additionally, some embodiments may include more than two grips connected to the support structure 412 or two grips 414 and another control mechanism (e.g., a button, switch, track pad, or scroll wheel). For example, the interface device 336 may include a button 418 that can be activated by the physician to switch control modes or perform specific actions. As shown, button 418 is mounted at the proximal end of support structure 412 (disposed between grip 414) so ​​that button 418 can be actuated when a hand is gripping support structure 412. Button 418 may include redundant buttons or similar but non-redundant buttons disposed on opposite sides of support structure 412. However, in other embodiments, one or more similar buttons may be located elsewhere. Finger assembly 337 may be positioned around Fig.11 The handles 414A and 414B can be squeezed or pinched together around the tubular structure 412. The "pinch" or gripping freedom in the handles is indicated by arrows Ha and Hb. These or other movements of the handles 414 relative to the support structure 412 can provide commands for manipulating the tools 330a-330c.

[0065] Finger assembly 337 is rotatably supported by first gimbal member 402 by means of rotational joint 416A. First gimbal member 402, in turn, is rotatably supported by second gimbal member 404 about axis B by means of rotational joint 416B. Similarly, second gimbal member 404 is rotatably supported by third gimbal member 406 about axis C using rotational joint 416C. In this manner, interface device 336 allows finger assembly 337 to be moved and oriented in a workspace using three degrees of freedom.

[0066] The movement in the universal joint of the main interface device 336 for reorienting the finger assembly 337 in space can be converted into a control signal to control the arm / tool ​​combination. For example, the rotational movement of the finger assembly 337 around the axis A in Figure 15 can be used to roll the instrument 330a around its shaft axis. Alternatively or additionally, for example, the squeezing movement of the grip 414A, 414B on its freedom of movement indicated by the arrows Ha and Hb can be used to command the grasping movement with forceps, or the cutting movement with scissors, or control the flow of fluid of the suction / irrigator robot medical tool positioned at the intervention site. The grip 414 can be passively biased to spring open, providing a restoring force to release forceps, open scissors, etc. In order to sense the movement in the touch-sensitive handle and generate control signals for the instrument 330, sensors can be installed in the universal joint components of the finger assembly 337 and the interface device 336. Exemplary sensors can include Hall effect transducers, potentiometers, encoders, etc.

[0067] Elements described in detail with reference to one embodiment, implementation or application may optionally be included in other embodiments, implementations or applications in which they are not specifically shown or described (as long as feasible). For example, if an element is described in detail with reference to one embodiment, but not described with reference to a second embodiment, the element may still be required to be included in the second embodiment. Therefore, in order to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, implementation or application may be incorporated into other embodiments, implementations or aspects, unless otherwise specifically described, unless the one or more elements will make the embodiment or implementation inoperative, or unless two or more elements provide conflicting functions.

[0068] Any changes and further modifications to the described devices, systems, apparatuses, methods, and any further applications of the principles of the present disclosure are fully contemplated as would be normally expected by a person skilled in the art of the present disclosure. In particular, it is fully contemplated that the features, components, and / or steps described with respect to one embodiment may be combined with the features, components, and / or steps described with respect to other embodiments of the present disclosure. In addition, the dimensions provided herein are for specific examples, and it is contemplated that different sizes, dimensions, and / or ratios may be utilized to implement the concepts of the present disclosure. In order to avoid unnecessary descriptive repetitions, one or more components or actions described according to an illustrative embodiment may be used or omitted when applicable according to other illustrative embodiments. For the sake of brevity, multiple iterations of these combinations will not be described separately.

[0069] Various systems and portions of systems have been described in terms of their state in three-dimensional space. As used herein, the term "position" refers to the position of an object or portion of an object in three-dimensional space (e.g., three translational degrees of freedom along Cartesian X, Y, Z coordinates). As used herein, the term "orientation" refers to the rotational placement of an object or portion of an object (three rotational degrees of freedom, such as roll, pitch, and yaw). As used herein, the term "pose" refers to the position of an object or portion of an object in at least one translational degree of freedom, as well as the orientation of the object or portion of the object in at least one rotational degree of freedom (up to six total degrees of freedom).

[0070] Although some of the examples described herein relate to surgical procedures or instruments, or medical procedures and medical instruments, the disclosed technology is optionally applicable to non-medical procedures and non-medical instruments. For example, the instruments, systems and methods described herein can be used for non-medical purposes, including industrial uses, general robotic uses, and sensing or manipulating non-tissue artifacts. Other example applications relate to decorative improvements, imaging of human or animal anatomical structures, collecting data from human or animal anatomical structures, and training medical or non-medical personnel. Additional example applications include procedures for tissues removed from human or animal anatomical structures (not returned to human or animal anatomical structures), and procedures performed on human or animal corpses. In addition, these technologies can also be used in surgical and non-surgical medical treatment or diagnostic procedures.

[0071] A computer is a machine that follows programmed instructions to perform mathematical or logical functions on input information to produce processed output information. A computer includes logic units that perform mathematical or logical functions, and memory that stores programmed instructions, input information, and output information. The term "computer" and similar terms (such as "processor" or "controller" or "control system") are synonymous.

[0072] Although certain exemplary embodiments of the present invention have been described and shown in the accompanying drawings, it should be understood that these embodiments are merely illustrative of the broad invention rather than limiting thereof, and that the embodiments of the present invention are not limited to the specific constructions and arrangements shown and described, since various other modifications may occur to those skilled in the art.

Claims

1. A method comprising: Displaying an interactive panel on a display device, the interactive panel having a display reference frame and including a plurality of interactive elements; constraining movement of the operator interface device to translational movement in a tactile plane using a tactile plane constraint when displaying the interactive panel; determining a cursor position in the display reference frame, the cursor position corresponding to a position of a selected portion of the operator interface device in the operator reference frame; Determining an element type of a first interactive element among the plurality of interactive elements where the cursor is located; and If the element type is a three-dimensional rotation type, the haptic plane constraint on the operator interface device is removed.

2. The method according to claim 1, further comprising: A rotational constraint is utilized to constrain rotational movement of the operator interface device when the interactive panel is displayed.

3. The method according to claim 2, further comprising: If the element type is a three-dimensional rotation type, the rotation constraint on the operator interface device is removed.

4. The method according to claim 3, further comprising: A determination is made as to whether the cursor position has moved away from the first interactive element, and if so, restoring the tactile plane constraint and the rotation constraint on the operator interface device.

5. The method according to claim 1, further comprising: A rotational motion input is received from the operator interface device in the operator reference frame.

6. The method according to claim 5, further comprising: A rotational motion is generated for the first interactive element in the display reference frame corresponding to the rotational motion input from the operator interface device.

7. The method according to claim 1, wherein: Removing the haptic plane constraint on the operator interface device is further based on receiving a rotation signal from the operator interface device. The method of claim 7 , wherein the rotation signal comprises a pinch gesture on the operator interface device.

9. The method according to claim 7, wherein: When the rotation signal is received, the mark of the cursor position disappears.

10. The method of claim 1, wherein the operator interface device has a plurality of translational and rotational degrees of freedom.

11. The method of claim 1 , wherein the interactive panel includes menu elements that are responsive to translational movement of an operator interface device in the tactile plane.

12. The method according to claim 1, wherein: The haptic plane constraint is generated by a constraint force provided to the operator interface device.

13. The method according to claim 1, wherein: A cursor marking can be displayed at the cursor position on the display device.

14. The method of claim 1, wherein the selected portion of the operator interface device is a tip portion of the operator interface device.

15. The method of claim 1, wherein the element type is a three-dimensional rotation type.

16. The method according to claim 1, wherein the element type is a two-dimensional translation type, the method further comprising: If the element type is a two-dimensional rotation type, the movement of the operator interface device is constrained to movement in the haptic plane.

17. The method according to claim 1, wherein the element type is a single-dimensional translation type, the method further comprising: If the element type is a single-dimensional translation type, the movement of the operator interface device is constrained to a single translation direction in the haptic plane.

18. The method according to claim 1, wherein the element type is a single-dimensional rotation type, the method further comprising: If the element type is a one-dimensional rotational type, the movement of the operator interface device is constrained to be a rotational movement in the haptic plane.

19. The method according to claim 1, wherein: When the control system is in the auxiliary mode, the interactive panel is displayed on the display device.

20. A medical device system, comprising: Control systems; Display devices; and an operator interface device in communication with said control system, The control system comprises: Processor; and a memory comprising machine-readable instructions that, when executed by the processor, cause the control system to: Displaying an interactive panel on the display device, the interactive panel having a display reference system and including a plurality of interactive elements; constraining movement of the operator interface device to translational movement in a tactile plane using a tactile plane constraint when displaying the interactive panel; determining a cursor position in the display reference frame, the cursor position corresponding to a position of a selected portion of the operator interface device in the operator reference frame; Determining an element type of a first interactive element among the plurality of interactive elements at which the cursor is located; and If the element type is a three-dimensional rotation type, the haptic plane constraint on the operator interface device is removed.

21. The medical instrument system of claim 20, wherein the instructions further cause the control system to utilize a rotational constraint to constrain rotational movement of the operator interface device when displaying the interactive panel.

22. The medical device system according to claim 21, wherein: The instructions further cause the control system to remove the rotation constraint on the operator interface device if the element type is a three-dimensional rotation type.

23. The medical instrument system of claim 22, wherein the instructions further cause the control system to determine whether the cursor position has moved away from the first interactive element, and if so, to restore the tactile plane constraint and the rotation constraint on the operator interface device.

24. The medical device system of claim 20, wherein the instructions further cause the control system: wherein the instructions further cause the control system to receive rotational motion input from the operator interface device in the operator reference frame.

25. The medical instrument system of claim 24, wherein the instructions further cause the control system to generate a rotational motion for the first interactive element in the display reference frame corresponding to the rotational motion input from the operator interface device.

26. The medical instrument system of claim 20, wherein the instructions further cause the control system: wherein removing the haptic plane constraint on the operator interface device is further based on receiving a rotation signal from the operator interface device.

27. The medical instrument system of claim 26, wherein the rotation signal comprises a pinch gesture on the operator interface device.

28. The medical device system of claim 26, wherein: When the rotation signal is received, the mark of the cursor position disappears.

29. The medical instrument system of claim 20, wherein the operator interface device has multiple translational and rotational degrees of freedom.

30. The medical instrument system of claim 20, wherein the interaction panel includes menu elements that are responsive to translational movement of an operator interface device in the tactile plane.

31. The medical device system of claim 20, wherein the tactile plane constraint is generated by a constraint force provided to the operator interface device.

32. The medical device system of claim 20, wherein: A cursor marking can be displayed at the cursor position on the display device.

33. The medical instrument system of claim 20, wherein the selected portion of the operator interface device is a tip portion of the operator interface device.

34. The medical device system of claim 20, wherein the element type is a three-dimensional rotation type.

35. The medical device system of claim 20, wherein the element type is a two-dimensional translation type, and wherein if the element type is a two-dimensional rotation type, the instructions further cause the control system to constrain movement of the operator interface device to movement in the tactile plane.

36. The medical device system of claim 20, wherein the element type is a single-dimensional translation type, and wherein if the element type is a single-dimensional translation type, the instructions further cause the control system to constrain movement of the operator interface device to a single translation direction in the tactile plane.

37. The medical device system of claim 20, wherein the element type is a one-dimensional rotational type, and wherein if the element type is a one-dimensional rotational type, the instructions further cause the control system to constrain movement of the operator interface device to rotational movement in the tactile plane.

38. The medical device system of claim 20, wherein: When the control system is in the auxiliary mode, the interactive panel is displayed on the display device.