Controlling virtual user interface elements

Abstract

The present disclosure describes a computer system and method for manipulating models. The computer system includes a display, a first user input device, a second user input device, a memory, and a processor communicatively coupled to the memory. The processor presents, on the display, a model of an object and rotates the model based on rotational movement of at least one of the first user input device or the second user input device within a virtual shape. The processor resists, at an edge of the virtual shape, the rotational movement of at least one of the first user input device or the second user input and translates the model and the virtual shape based on translational movement of at least one of the first user input device or the second user input device.

Description

CONTROLLING VIRTUAL USER INTERFACE ELEMENTSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of co-pending United States provisional patent application Serial No. 63 / 610,470 filed December 15, 2023. The aforementioned related patent application is herein incorporated by reference in its entirety.TECHNICAL FIELD

[0002] The present disclosure relates generally to medical systems (e.g., surgical systems). Specifically, the present disclosure relates to a medical system that provides different control modes for manipulating virtual elements in a user interface.BACKGROUND

[0003] Doctors use computer assisted medical systems to perform operations on patients, even remotely. These medical systems provide the doctors various views of surgical sites during the operations. The medical systems may also provide a user interface with which the doctors may interact during the operations. To interact with the user interface, the doctor may use the same input devices that the doctor uses to control the medical instruments. For the safety of the patient, the medical system may not move the medical instruments when the doctor moves the input devices to interact with the user interface. This movement of the input devices, however, may cause the input devices to be misaligned with the medical instruments after the doctor finishes interacting with the user interface. The doctor may experience difficulty accurately and precisely moving the medical instruments due to this misalignment.SUMMARY

[0004] The present disclosure describes a computer system and method for manipulating models. According to an embodiment, the computer system includes a display, a first user input device, a second user input device, a memory, and a processor communicatively coupled to the memory. The processor presents, on the display, a model of an object and rotates, within a virtual shape, the model based on rotational movement of at least one of the first user input device or the second user input device. The processor resists, at an edge of the virtual shape, the rotationalmovement of at least one of the first user input device or the second user input and translates the model and the virtual shape based on translational movement of at least one of the first user input device or the second user input device.

[0005] According to another embodiment, a method includes presenting, on a display, a model of an object and rotating, within a virtual shape, the model based on rotational movement of at least one of a first user input device or a second user input device. The method also includes resisting, at an edge of the virtual shape, the rotational movement of at least one of the first user input device or the second user input and translating the model and the virtual shape based on translational movement of at least one of the first user input device or the second user input device. Other embodiments includes a non-transitory machine-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method.

[0006] The foregoing general description and the following detailed description are exemplary and explanatory in nature and are intended to provide an understanding of the present disclosure without limiting the scope of the present disclosure. In that regard, additional aspects, features, and advantages of the present disclosure will be apparent to one skilled in the art from the following detailed description.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 illustrates an example medical system.

[0008] Figures 2A and 2B illustrate example components of the medical system of Figure 1.

[0009] Figure 3 illustrates an example operation for changing modes in the medical system of Figure 1 .

[0010] Figure 4 illustrates an example operation for moving a medical instrument in the medical system of Figure 1 .

[0011] Figures 5A, 5B, 5C, 5D, 5E, 5F, 5G, and 5H illustrate an example operation for interacting with a user interface in the medical system of Figure 1 .

[0012] Figures 6A, 6B, 6C, 6D, and 6E illustrate an example operation for interacting with a user interface in the medical system of Figure 1.

[0013] Figure 7 illustrates an example operation for interacting with a user interface in the medical system of Figure 1 .

[0014] Figure 8 is a flowchart of an example method for changing modes and performed by the medical system of Figure 1.

[0015] Figure 9 is a flowchart of an example method for interacting with a user interface and performed by the medical system of Figure 1.DETAILED DESCRIPTION

[0016] Doctors use computer assisted medical systems (e.g., surgical systems) to perform operations on patients, even remotely. These medical systems provide the doctors various views of surgical sites during the operations. To perform the operations, the doctors may move user input devices to control medical instruments (e.g., surgical instruments) at the surgical sites.

[0017] A medical system may also provide a user interface with which a doctor may interact during an operation. To interact with the user interface, the doctor may provide input (e.g., press a button, press or depress a foot pedal, provide a verbal command, etc.) that causes the medical system to enter a virtual control mode. The medical system may then display the user interface and allow the doctor to interact with the user interface. The user interface may include menus, virtual buttons, models, or other elements with which the doctor can interact.

[0018] To interact with the user interface, the doctor may use the same input devices that the doctor uses to control the medical instruments. For the safety of the patient, the medical system may not move the medical instruments when the doctor moves the input devices during the virtual control mode. In conventional systems, the movement of the input devices during the virtual control mode may cause the input devices to be misaligned with the medical instruments after the doctor finishes interacting with the user interface. As a result, when the medical system returns to a manipulation input mode in which the doctor can control the medical instruments, the doctor may experience difficulty accurately and precisely moving the medical instruments due to the misalignment with the input devices.

[0019] The present disclosure describes a medical system that provides various features that help the medical instruments remain more aligned with the user input devices during the virtual control mode. For example, the medical system may track the rotational movement of the user input devices during the virtual control mode and resist the rotation of the user input devices if rotation would cause the user input devices to become too misaligned with the medical instruments. The medical system may track the rotational movement of the user input devices using a virtual shape. As a user input device rotates and approaches an edge of the virtual shape, the medical system may cause a force to be applied to the user input device that resists the rotation of the user input device closer towards the edge of the virtual shape. The magnitude of the force may increase as the user input device rotates closer towards the edge of the virtual shape.

[0020] As another example, the medical system may apply an offset to the movement of the user input device after transitioning back to the manipulation input mode. The offset may account for misalignment that may have occurred during the virtual control mode (e.g., misalignment that occurs even though the user input devices stayed within the virtual shape). The medical system may gradually bring the user input device back into alignment with the medical instrument as the doctor moves the user input device to move the medical instrument. The medical system may gradually reduce the offset as the user input device becomes more aligned with the medical instrument.

[0021] In certain embodiments, the medical system provides several technical advantages. For example, the medical system may help prevent a user input device from becoming too misaligned with a medical instrument during the virtual control mode. As another example, the medical system may gradually bring the user input device back into alignment with the medical instrument (as opposed to suddenly bringing the user input device into alignment) when transitioning back to the manipulation input mode. In this manner, the medical system improves the intuitive feel of the user input device for the doctor and improves safety for the patient.

[0022] In some examples, one or more components of a medical system may be implemented as a computer-assisted surgical system. It is understood, however, that the medical system may be implemented in any type of medical system (e.g., digitalfiducial systems, anatomy detection systems, and clinical guidance systems). Figure 1 shows an example computer-assisted surgical system 100 that implements some of the features described herein.

[0023] The surgical system 100 includes a manipulator assembly 102, a user control apparatus 104, and an auxiliary apparatus 106, all of which are communicatively coupled to each other. The surgical system 100 is utilized by a medical team to perform a computer-assisted medical procedure or other similar operation on a body of a patient 108 or on any other body as may serve a particular implementation. The medical team includes a first user 110-1 (such as a surgeon for a surgical procedure), a second user 110-2 (such as a patient-side assistant), a third user 110-3 (such as another assistant, a nurse, a trainee, etc.), and a fourth user 110- 4 (such as an anesthesiologist for a surgical procedure), all of whom are collectively referred to as users 110, and each of whom may control, interact with, or otherwise be a user of the surgical system 100. More, fewer, or alternative users may be present during a medical procedure as may serve a particular implementation. For example, team composition for different medical procedures, or for non-medical procedures, may differ and include users with different roles.

[0024] Although Figure 1 illustrates an ongoing minimally invasive medical procedure such as a minimally invasive surgical procedure, it will be understood that the surgical system 100 may similarly be used to perform open medical procedures or other types of operations. For example, operations such as exploratory imaging operations, mock medical procedures used for training purposes, and / or other operations may also be performed.

[0025] The manipulator assembly 102 includes one or more manipulator arms 112 (e.g., manipulator arms 112-1 through 112-4) to which one or more instruments may be coupled. The instruments are used for a computer-assisted surgical procedure on the patient 108 (e.g., by being at least partially inserted into the patient 108 and manipulated within the patient 108). While the manipulator assembly 102 is depicted and described herein as including four manipulator arms 112, the manipulator assembly 102 may include a single manipulator arm 112 or any other number of manipulator arms as may serve a particular implementation. Although the example of Figure 1 illustrates the manipulator arms 112 as robotic manipulator arms, one or moreinstruments may be partially or entirely manually controlled, such as by being handheld and controlled manually by a person. These partially or entirely manually controlled instruments are used in conjunction with, or as an alternative to, computer- assisted instrumentation that is coupled to the manipulator arms 112.

[0026] During the medical operation, the user control apparatus 104 facilitates teleoperational control by the user 110-1 of the manipulator arms 112 and instruments attached to the manipulator arms 112. To this end, the user control apparatus 104 provides the user 110-1 with imagery of an operational area associated with the patient 108 as captured by an imaging device. The manipulator arms 112 or any instruments coupled to the manipulator arms 112 mimic the dexterity of the hand, wrist, and fingers of the user 110-1 across multiple degrees of freedom of motion. In this manner, the user 110-1 intuitively performs a procedure (e.g., an incision procedure, a suturing procedure, etc.) using one or more of the manipulator arms 112 or any instruments coupled to the manipulator arms 112.

[0027] The auxiliary apparatus 106 includes one or more computing devices that perform auxiliary functions in support of the procedure, such as providing insufflation, electrocautery energy, illumination or other energy for imaging devices, image processing, or coordinating components of the surgical system 100. The auxiliary apparatus 106 includes a display monitor 114 that displays one or more user interfaces, or graphical or textual information in support of the procedure. In some instances, the display monitor 114 is a touchscreen display that provides user input functionality. Augmented content provided by a region-based augmentation system may be similar to, or differ from, content associated with the display monitor 114 or one or more display devices in the operation area (not shown).

[0028] The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 are communicatively coupled one to another in any suitable manner. The manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may be communicatively coupled by way of control lines 116, which represent any wired or wireless communication link as may serve a particular implementation. To this end, the manipulator assembly 102, user control apparatus 104, and auxiliary apparatus 106 may each include one or more wired or wireless communicationinterfaces, such as one or more local area network interfaces, Wi-Fi network interfaces, cellular interfaces, and so forth.

[0029] Figure 2A illustrates an example manipulator assembly 102. As seen in Figure 2A, the manipulator assembly 102 includes a base 118, a manipulator arm 112- 1 , a manipulator arm 112-2, a manipulator arm 112-3, and a manipulator arm 112-4. Each manipulator arm 112-1 , 112-2, 112-3, and 112-4 is pivotably coupled to the base 118. Although the base 118 may include casters to allow ease of mobility, in some embodiments, the manipulator assembly 102 is fixedly mounted to a floor, ceiling, operating table, structural framework, or the like.

[0030] In a typical procedure, two of the manipulator arms 112-1 , 112-2, 112-3, or 112-4 hold surgical instruments and a third holds a stereo endoscope. The remaining manipulator arms are available so that other instruments may be introduced at the work site. Alternatively, the remaining manipulator arms may be used for introducing another endoscope or another image capturing device, such as an ultrasound transducer, to the work site.

[0031] Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 are formed of links that are coupled together and manipulated through actuatable joints. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include a setup arm and a device manipulator. The setup arm positions its held device so that a pivot point occurs at its entry aperture into the patient. The device manipulator may then manipulate its held device so that the held device may be pivoted about the pivot point, inserted into and retracted out of the entry aperture, and rotated about its shaft axis. Each of the manipulator arms 112-1 , 112-2, 112-3, and 112-4 may include sensors (e.g., kinematics sensors, position sensors, accelerometers, etc.) that detect or track movement of the manipulator arms 112-1 , 112-2, 112-3, and 112-4. For example, these sensors may detect how far or how quickly a manipulator arm 112-1 , 112-2, 112-3, or 112-4 moves in a certain direction.

[0032] Figure 2B illustrates an example user control apparatus 104. The user control apparatus 104 includes a stereo vision display 120 so that the user may view the surgical work site in stereo vision from images captured by the stereoscopic camera of the manipulator assembly 102. Left and right eyepieces 122 and 124 areprovided in the stereo vision display 120 so that the user may view left and right display screens inside the display 120 respectively with the user's left and right eyes. While viewing typically an image of the surgical site on a suitable viewer or display, the surgeon performs the surgical procedures on the patient by manipulating control input devices, which in turn control the motion of robotic instruments.

[0033] The user control apparatus 104 also includes left and right input devices 126 and 128 that the user grasps respectively with his / her left and right hands to manipulate devices (e.g., surgical instruments) being held by the manipulator arms 112-1 , 112-2, 112-3, and 112-3 of the manipulator assembly 102 in preferably six or more degrees of freedom (“DOF”). Foot pedals 130 with toe and heel controls are provided on the user control apparatus 104 so the user may control movement and / or actuation of devices associated with the foot pedals.

[0034] A processing device 132 is provided in the user control apparatus 104 for control and other purposes. The processing device 132 performs various functions in the surgical system 100. One function performed by processing device 132 is to translate and transfer the mechanical motion of input devices 126 and 128 to actuate their corresponding joints in their associated manipulator arms 112-1 , 112-2, 112-3, and 112-4 so that the surgeon can effectively manipulate devices, such as the surgical instruments. Another function of the processing device 132 is to implement the methods, crosscoupling control logic, and controllers or processors described herein. The auxiliary apparatus 106 includes a processing device 132 that performs the functions or actions described herein. The processing device 132 may include a processor and a memory that perform the functions described herein.

[0035] The processor may include any electronic circuitry, including, but not limited to one or a combination of microprocessors, microcontrollers, application specific integrated circuits (ASIC), application specific instruction set processor (ASIP), and / or state machines, that communicatively couples to a memory and controls the operation of the user control apparatus 104 and / or the auxiliary apparatus 106. The processor may be 8-bit, 16-bit, 32-bit, 64-bit or of any other suitable architecture. The processor may include an arithmetic logic unit (ALU) for performing arithmetic and logic operations, processor registers that supply operands to the ALU and store the results of ALU operations, and a control unit that fetches instructions from memory andexecutes them by directing the coordinated operations of the ALU, registers and other components. The processor may include other hardware that operates software to control and process information. The processor executes software stored on a memory to perform any of the functions described herein. The processor controls the operation and administration of the user control apparatus 104 or the auxiliary apparatus 106 by processing information (e.g., information received from the user control apparatus 104, the manipulator assembly 102, the auxiliary apparatus 106, and / or a memory). The processor is not limited to a single processing device and may encompass multiple processing devices contained in the same device or computer or distributed across multiple devices or computers. The processor is considered to perform a set of functions or actions if the multiple processing devices collectively perform the set of functions or actions, even if different processing devices perform different functions or actions in the set.

[0036] Figures 3 through 7 illustrate example operations performed by a computer system in a medical system (e.g., the surgical system 100 of Figure 1 ). Generally, the computer system (which may be implemented in the user control apparatus 104 and / or the auxiliary apparatus 106 of the surgical system 100 using the processing device 132) provides different control modes in which an operator (e.g., a doctor, surgeon, or other medical professional) of the computer system may move a medical instrument or interact with certain user interface elements.

[0037] Figure 3 illustrates an example operation 300 performed by the computer system. Generally, the computer system provides for different types of movements during different control modes. As seen in Figure 3, the computer system receives an input 302 that indicates a mode 304. The input 302 may be provided by an operator of the computer system using any input mechanism or modality (e.g., the input devices 126 or 128, the foot pedals 130, the operator’s voice, etc.). The input 302 indicates the mode 304 desired by the operator. For example, the input 302 may indicate that the operator wants to enter a manipulation input mode in which the operator controls medical instruments. As another example, the input 302 may indicate that the operator desires to enter a virtual control mode in which the operator interacts with various elements of a user interface rather than controlling the medical instruments. Thecomputer system sets or switches the mode 304 according to the input 302 provided by the operator.

[0038] The computer system provides for different type of movement during the different modes 304. During the mode 304, the computer system receives input 306. The input 306 may be provided by an operator of the computer system using any type of controller (e.g., the input devices 126 or 128 or the foot pedals 130). The input 306 indicates one or more different types of movement. For example, the input 306 may indicate translational movement 308 and / or rotational movement 310. Generally, the translational movement 308 is indicated by a movement that shifts or changes coordinates in a three-dimensional space. On the other hand, the rotational movement 310 is indicated by a rotation in three-dimensional space. As an example, the operator may use the input device 126 or 128 to provide the input 306. The operator may move the input device 126 or 128 to different positions or locations in three-dimensional space to provide the translational movement 308. The operator may rotate the input device 126 or 128 to provide the rotational movement 310.

[0039] The computer system processes the translational movement 308 and the rotational movement 310 differently depending on the mode 304. For example, during a manipulation input mode, the computer system processes the translational movement 308 and the rotational movement 310 into corresponding movements of a medical instrument. The translational movement 308 shifts or changes the position of the medical instrument in three-dimensional space. The rotational movement 310 rotates the medical instrument. As another example, during a virtual control mode, the computer system processes the translational movement 308 and the rotational movement 310 into corresponding interactions with certain user interface elements. The translational movement 308 moves a displayed model of an anatomical structure to a different position on a display. The rotational movement 310 rotates the model on the display. As another example, the translational movement 308 may move a cursor to different parts of the display. The rotational movement 310 may navigate a menu on the display.

[0040] Figure 4 illustrates an example operation 400 performed by the computer system. Generally, Figure 4 shows the computer system during a manipulation input mode 402. When the computer system is in the manipulation input mode 402, thecomputer system allows an operator of the computer system to control a medical instrument 404 (e.g., a surgical instrument).

[0041] As seen in Figure 4, the computer system is in the manipulation input mode 402. The computer system may have entered the manipulation input mode 402 in response to input 302 that indicates the operator of the computer system desires to enter the manipulation input mode 402. The operator of the computer system may have indicated this desire by operating the input devices 126 or 128 or the foot pedals 130. When the computer system receives the input 302, the computer system enters the manipulation input mode 402.

[0042] During the manipulation input mode 402, the computer system allows the operator of the computer system to move the medical instrument 404. The computer system receives the input 306 from the operator. The operator may provide the input 306 by moving a controller (e.g., the input devices 126 or 128). As seen in Figure 4, the input 306 indicates translational movement 308 of the controller and / or rotational movement 310 of the controller. For example, if the operator were operating the input devices 126 or 128, the operator may indicate translational movement 308 by moving the input devices 126 or 128 to different positions in three-dimensional space. The operator may indicate rotational movement 310 by rotating the input devices 126 or 128.

[0043] The computer system processes the translational movement 308 and / or the rotational movement 310 to make corresponding movements of the medical instrument 404. For example, the computer system may process the translational movement 308 to move the instrument 404 from one position to another in three- dimensional space. The computer system may move the instrument 404 in the same direction as the translational movement 308. Additionally, the computer system may move or translate the instrument 404 by an amount (e.g., by a distance) indicated by the translational movement 308. As another example, the computer system may rotate the instrument 404 according to the rotational movement 310. The computer system may rotate the instrument 404 in the same direction as the rotational movement 310. Additionally, the computer system may rotate the instrument 404 by an amount (e.g., by a number of degrees) indicated by the rotational movement 310.In this manner, the computer system operates the instrument 404 according to the input 306 provided by the operator during the manipulation input mode 402.

[0044] Figures 5A, 5B, 5C, and 5D illustrate an example operation 500 performed by the computer system. Generally, Figures 5A, 5B, 5C, and 5D show the computer system during a virtual control mode 502. During the virtual control mode 502, the computer system allows an operator of the computer system to interact with user interface elements. The computer system may prevent the operator from operating the medical instrument 404 during the virtual control mode 502.

[0045] The computer system transitions to the virtual control mode 502 in response to input 302 from the operator of the computer system. The operator may provide the input 302 by operating the input devices 126 or 128 or pressing or depressing the foot pedals 130. The input 302 may indicate that the operator desires to enter the virtual control mode 502. In response, the computer system transitions to the virtual control mode 502.

[0046] As seen in Figure 5A, the virtual control mode 502 further indicates that a controller should be operated separately from other controllers. For example, if the operator of the computer system were operating the input devices 126 or 128, the computer system allows the input devices 126 and 128 to be operated separately from each other. In some embodiments, the computer system determines that the controller should be operated separately based on a preference of the operator set in the computer system. When the computer system transitions to the virtual control mode 502, the computer system determines the saved preference for separate operation and then automatically set the controllers to be operated separately.

[0047] The controller receives the input 306 during the virtual control mode 502. The operator may provide the input 306 by operating a controller (e.g., the input devices 126 or 128). The input 306 may indicate translational movement 308 of the controller. For example, the translational movement 308 may indicate that the controller has been moved from one position to another in three-dimensional space.

[0048] The computer system processes the translational movement 308 to move one or more user interface elements. In Figure 5A, the computer system moves a displayed model 504 to different parts of a display according to the translationalmovement 308. The model 504 may be a model 504 of an anatomical structure at a surgical site. The computer system displays the model 504 during the virtual control mode 502. The computer system may move the model 504 in the same direction as the translational movement 308. Additionally, the computer system may move or translate the model 504 by an amount (e.g., across a distance) indicated by the translational movement 308. In this manner, the computer system may move or translate the model 504 to different parts of the display.

[0049] The computer system implements a virtual shape 506 that the computer system virtually attaches to the controller. The virtual shape 506 is a three- dimensional shape that the computer system uses to discourage input that would cause the controller from becoming too misaligned with a medical instrument during the virtual control mode 502. The virtual shape 506 may be any geometric shape, including but not limited to a cone, a pyramid, a cylinder, a sphere, and / or a blob. A tip or surface of the virtual shape 506 may have the same coordinate in three- dimensional space as a portion of the controller. The virtual shape 506 may not be displayed and may not be visible to the operator of the computer system. The computer system may move or translate the virtual shape 506 in the same way that the computer system moves or translates the model 504 according to the translational movement 308 such that the virtual shape 506 moves with the controller. For example, the computer system may move the virtual shape 506 in the same direction as the translational movement 308. Additionally, the computer system may move or translate the virtual shape 506 by an amount indicated by the translational movement 308. As a result, the virtual shape 506 may be moved the same distance and in the same direction as the controller in response to the translational movement 308.

[0050] Figure 5B shows the computer system processing rotational movement 310 during the virtual control mode 502. During the virtual control mode 502, the computer system resists rotational movement 310 near the edges of the virtual shape 506. For example, the computer system may apply a force on a controller that opposes the operator’s rotation of the controller when the rotational movement 310 brings the controller near the edges of the virtual shape 506. In this manner, the computer system indicates to the operator that rotation beyond the edges of the virtual shape 506 is not desirable or optimal.

[0051] Rotational movement 310 beyond the edges of the virtual shape 506 may cause the controller to become too misaligned with the medical instrument 404 (which is not being controlled or moved during the virtual control mode 502). This misalignment may cause the operator to experience difficulty manipulating the medical instrument 404 when the computer system transitions back to the manipulation input mode 402 from the virtual control mode 502. By opposing or restricting the rotation near the edges of the virtual shape 506, the computer system encourages the controller to stay aligned or remain within a threshold alignment with the medical instrument 404. In this manner, the computer system provides for more accurate movements of the medical instrument 404 when the computer system transitions from the virtual control mode 502 to the manipulation input mode 402, which may improve the health and safety of a patient.

[0052] As seen in Figure 5B, the computer system is in the virtual control mode 502. As discussed previously, during the virtual control mode 502, the operator of the computer system operates a controller (e.g., the input devices 126 or 128) to interact with user interface elements, such as the model 504 of an anatomical structure at the surgical site.

[0053] The computer system receives the input 306 indicating the rotational movement 310. For example, the operator of the computer system may rotate the controller to indicate the rotational movement 310. The computer system rotates the model 504 according to the rotational movement 310. For example, the computer system may rotate the model 504 in the same direction as the rotational movement 310 and by an amount indicated by the rotational movement 310.

[0054] The rotational movement 310 of the controller occurs within the virtual shape 506. As discussed previously, the virtual shape 506 may not be displayed and may not be visible to the operator of the computer system. Rotation of the controller may cause the controller to rotate closer to the edges of the virtual shape 506. When the controller approaches the edge of the virtual shape 506, the computer system resists or opposes the further rotation of the controller. For example, the computer system may apply an opposing force to the controller that indicates to the operator that further rotation of the controller is not be desirable or optimal. Specifically, further rotation of the controller may cause the controller to become too misaligned with the medicalinstrument 404. In response, the operator of the computer system may stop rotating the controller, which keeps the controller aligned or within a threshold alignment with the medical instrument 404.

[0055] Figure 5C shows an implementation of the virtual shape 506. In the example of Figure 5C, the virtual shape 506 is a cone. The computer system processes the rotational movement 310 of the controller within the virtual shape 506. When the controller approaches the edge of the virtual shape 506, the computer system begins applying resistance to the rotational movement 310. For example, when the operator of the computer system rotates the controller such that the controller approaches closer to the edge of the virtual shape 506, the computer system applies a resistive force to the controller that opposes the rotation. The operator of the computer system may feel or experience the resistive force and understand that further rotation of the controller is not desired or optimal because further rotation may cause the controller to become too misaligned with the medical instrument 404. In response, the operator of the computer system may stop the rotational movement 310 to keep the controller within the virtual shape 506.

[0056] In some embodiments, the computer system scales the effects of the rotational movement 310 by different amounts depending on how close the controller is to the edges of the virtual shape 506. When the controller is far from the edge of the virtual shape 506 (e.g., near a central axis of the virtual cone), the computer system does not scale or minimally scales the effects of the rotational movement 310. The computer system increases the scale the closer the controller is to the edges of the virtual shape 506. As a result, the interaction with the user interface element is scaled by a larger value the closer the controller is to the edges of the virtual shape 506. For example, if the computer system rotates the model 504 according to the rotational movement 310, then the computer system increases the scale, which increases the amount by which the model 504 rotates, when the controller has rotated close to the edge of the virtual shape 506. As a result, the computer system applies the resistive force at the edge of the virtual shape 506, which reduces the amount of rotational movement 310 that occurs, but the computer system increases the amount by which the model 504 rotates according to the rotational movement 310. In this manner, thecomputer system maintains the rotation of the model 504, while resisting the rotation of the controller at the edge of the virtual shape 506.

[0057] Figure 5D illustrates an example input device 126 or 128 and a virtual shape 506. In the example of Figure 5D, the virtual shape 506 is a cone. The computer system positions the virtual shape 506 such that the virtual shape 506 extends from the input device 126 or 128. The tip of the virtual shape 506 is positioned on or near a coordinate of the input device 126 or 128 in three-dimensional space. As discussed previously, translational movement 308 moves the input device 126 or 128 and the virtual shape 506 in three-dimensional space. Additionally, rotational movement 310 of the input device 126 or 128 occurs within the virtual shape 506. In some embodiments, the computer system does not move the virtual shape 506 according to rotational movement 310 of the input device 126 or 128.

[0058] The input device 126 or 128 includes multiple arms 508 that each rotate about an axis in three-dimensional space. As a result, the rotational movement 310 may cause the input device 126 or 128 to rotate about three different axes. As discussed previously, the rotation of the input device 126 or 128 occurs within the virtual shape 506. The computer system applies a resistive force to the input device 126 or 128 that opposes the rotational movement 310 of the input device 126 or 128 when the input device 126 or 128 has rotated near an edge of the virtual shape 506. The operator of the computer system may feel or experience the resistive force when operating the input device 126 or 128 and understand that rotation of the input device 126 or 128 should be stopped so that the input device 126 or 128 does not become too misaligned with the medical instrument 404.

[0059] The computer system may adjust the size or shape of the virtual shape 506 based on movement of the input device 126 or 128. Figure 5E shows an operation for changing the size or shape of the virtual shape 506. In the example of Figure 5E, the virtual shape 506 is a cone. The computer system may expand or shrink the cone by adjusting an apex angle of the cone in response to certain movement of the input device 126 or 128. For example, the rotational position of the input device 126 or 128 may be indicated using a roll angle, a yaw angle, and a pitch angle. The computer system may adjust the shape or size of the cone based on one or more of these angles. For example, the computer system may adjust the shape or size of the conebased on the roll angle. When the roll angle is at zero degrees, the computer system may set the apex angle of the cone at a maximum angle such that the cone is wide and large. As the operator rolls the input device 126 or 128, the computer system adjusts the apex angle of the cone so that the cone narrows and becomes smaller. When the operator pitches or yaws the input device 126 or 128, the computer system may apply a resistive force to the input device 126 or 128 the closer the input device 126 or 128 is pitched or yawed to the edge of the cone.

[0060] Although several of the examples show the virtual shape 506 being a cone, it is understood that the virtual shape 506 may be any geometric shape. For example, the virtual shape 506 may be a pyramid, cylinder, sphere, and / or blob. When rotation of the input device 126 or 128 causes the input device 126 or 128 to approach an edge of the virtual shape 506 in three-dimensional space, the computer system may apply a force that resists further rotation of the input device 126 or 128 to signal to an operator that further rotation may cause the input device 126 or 128 to become too misaligned with a medical instrument. For example, when the virtual shape 506 is a pyramid, the computer system may adjust the apex angle of the pyramid based on the roll angle of the input device 126 or 128. As another example, when the virtual shape 506 is a cylinder, the computer system may adjust may adjust the radius or diameter of the cylinder based on the roll angle of the input device 126 or 128. When the operator pitches or yaws the input device 126 or 128, the computer system may apply a resistive force to the input device 126 or 128 the closer the input device 126 or 128 is pitched or yawed to the edge of the virtual shape 506.

[0061] Figure 5F illustrates an example feature of the computer system. The computer system allows for or enables the rotation of the model 504 in response to selection of a widget 510. The computer system displays the widget 510. The operator of the computer system operates a controller to select the widget 510. Selection of the widget 510 enables interaction with a user interface element through rotational movement 310. For example, selection of the widget 510 may allow the operator of the computer system to rotate the model 504 through the rotational movement 310. Before selection of the widget 510, the computer system may not rotate the model 504 even though the operator of the computer system had performed rotational movement 310 of the controller. Figure 5G shows a view of the model 504and the widget 510. The widget 510 is displayed with the model 504. The operator selects the widget 510 by operating a controller (e.g., the input device 126 or 128) to move a cursor on the display and to select the widget 510. When the widget 510 is selected, the computer system allows rotation of the model 504. For example, after the operator selects the widget 510, the computer system may allow the model 504 to rotate about a vertical axis 516. The axis 516 may or may not be visibly displayed with the model 504 and the widget 510.

[0062] In some embodiments, the computer system assists the rotational movement 310. For example, the computer system may apply a force 512 to the controller in the direction of the rotational movement 310. The magnitude of the force 512 changes depending on a position of the controller during the rotational movement 310. The computer system selects a coordinate 514 in three-dimensional space. As the controller moves closer to that coordinate 514, the computer system increases the magnitude of the force 512. When the controller is moved further away from that coordinate 514, the computer system decreases the magnitude of the force 512. In this manner, the computer system implements a gravity well type feature that assists the rotational movement 310 of the controller at certain positions in three-dimensional space. Figure 5H shows a view of an input device 126 or 128. The computer system implements a virtual gravity well 518 that is positioned at a coordinate 514 in three- dimensional space. When the operator rotates the input device 126 or 128 towards the coordinate 514, the computer system applies a force 520 in the direction of the rotation to assist the rotation. The closer the input device 126 or 128 is rotated towards the coordinate 514, the greater the magnitude of the force 512.

[0063] Figure 6A illustrates an example operation 600 performed by the computer system. Generally, Figure 6A shows the operation of the computer system during a virtual control mode 602. During the virtual control mode 602, the computer system allows an operator of the computer system to interact with certain user interface elements, such as the model 504. During the virtual control mode 602, the operator operates controllers (e.g., the input devices 126 and 128) in a tethered configuration. In the tethered configuration, the computer system maintains a relative distance between the controllers that mimics the operation of a handlebar. The controllers move together as if the controllers are tethered together. The computer system mayset the controllers to operate in the tethered configuration (as opposed to the separate configuration of the virtual control mode 502) based on preferences of the operator. These preferences may be saved, and when the operator provides the input 302 that instructs the computer system to transition to the virtual control mode, the computer system references the saved preferences to determine to transition to the virtual control mode 602 in which the controllers are in a tethered configuration.

[0064] During the virtual control mode 602, the computer system receives the input 306. The operator of the computer system provides the input 306 by operating the controllers. The computer system locks or maintains a relative distance 606 between the controllers. As a result, translational movement and / or rotational movement of the controllers may not change the relative distance 606 between the controllers. For example, when one controller is translated, the computer system may cause the other controller to similarly translate to maintain the relative distance 606 between the controllers.

[0065] The computer system also tracks positions 604 of the controllers during the translational or rotational movement. When the position 604 of a controller changes, the computer system causes the position 604 of the other controller to change to maintain the relative distance 606 between the controllers. In some embodiments, the computer system applies a force 608 to one or more of the controllers to maintain the relative distance 606 between the controllers. For example, the computer system may apply a force 608 such that a controller moves along with the movement of the other controller. As another example, the computer system may apply a force 608 that prevents movement that would cause the relative distance 606 between the controllers to change. As a result, the controllers move as if a handlebar is coupled or attached to both controllers.

[0066] The computer system may track the movement of the controllers based on a midpoint 610 between the controllers. The midpoint 610 is a point on the middle of the virtual handlebar between the controllers that maintains the relative distance 606 between the controllers. The computer system determines the midpoint 610 by determining the coordinate that is half the relative distance 606 between the controllers. When the controllers translate, the midpoint 610 translates. When the controllers rotate, the midpoint 610 rotates. The computer system translates or rotatesthe model 504 based on the movement of the midpoint 610 rather than the movement of the controllers themselves.

[0067] Figures 6B and 6C illustrate example movements of the input devices 126 and 128 during the virtual control mode 602. The computer system implements a virtual handlebar 612 between the input devices 126 and 128 by locking or maintaining a relative distance 606 between the input devices 126 and 128. In Figure 6B, when an input device 126 or 128 translates, the computer system translates the other input device 128 or 126 in the same direction to maintain the relative distance 606 between the input devices 126 and 128 (e.g., by applying the force 608). The computer system also maintains the relative orientation between the input devices 126 and 128 so that the virtual handlebar 612 does not rotate. The computer system tracks the position and movement of the input devices 126 and 128 based on the midpoint 610 on the virtual handlebar 612.

[0068] The translation of the input devices 126 and 128 may occur across multiple orthogonal axes in three-dimensional space. In Figure 6C, the input devices 126 and 128 translate in at least two different orthogonal axes. This translation causes rotation of the virtual handlebar 612. The computer system applies forces 608 that maintain the relative distance 606 between the input devices 126 and 128 during the translation. For example, the force may prevent the input device 126 or 128 from translating too far away from the other input device 126 or 128. As another example, the force may push or pull an input device 126 or 128 along with the translation of the other input device 126 or 128.

[0069] Figures 6D and 6E illustrate example movements of the input devices 126 and 128 during the virtual control mode 602. Similar to Figure 6B, in Figure 6D, when the input device 126 or 128 translates, the computer system also translates the other input device 126 or 128 in the same direction (e.g., by applying the force 608) to maintain the relative distance 606 and orientation between the input devices 126 and 128. Additionally, the computer system allows the input devices 126 and 128 to roll. In Figure 6D, the input device 128 may roll without changing the relative distance 606 and orientation between the input devices 126 and 128. The computer system may not cause a corresponding roll in the input device 126 as a result of rolling the input device 128, and vice versa.

[0070] Similar to Figure 6C, in Figure 6E, when the input device 126 or 128 translates in at least two different orthogonal axes in three-dimensional space, the computer system applies forces 608 that maintain the relative distance 606 between the input devices 126 and 128 during the translation. The relative orientation between the input devices 126 and 128 may change, which results in a rotation of the virtual handlebar 612. Additionally, the computer system allows the input devices 126 and 128 to roll. In Figure 6E, the input device 128 may roll without changing the relative distance 606 between the input devices 126 and 128. The computer system may not cause a corresponding roll in the input device 126 as a result of rolling the input device 128, and vice versa.

[0071] In some embodiments, the computer system uses the virtual shape 506 to determine when to limit rotation of the controllers when the controllers are tethered together (e.g., in Figures 6B, 6C, 6D, and 6E). For example, the virtual shape 506 may extend from either of the controllers or from the midpoint 610 of the controllers. When the computer system detects that the controllers or the midpoint 610 have rotated close to the edge of the virtual shape 506, the computer system applies a force to one or more of the controllers to resist further rotation. The operator of the computer system may feel this force and understand that further rotation is not desired or optimal. In this manner, the computer system encourages the controllers to stay aligned or remain within a threshold alignment with the medical instrument(s).

[0072] Figure 7 illustrates an example operation 700 performed by the computer system. Generally, Figure 7 shows the computer system transitioning back to the manipulation input mode 402 from the virtual control mode 502 or 602. When the computer system transitions back to the manipulation input mode 402, the computer system implements certain features to correct or resolve misalignment between a controller (e.g., the input devices 126 or 128) and a medical instrument 404.

[0073] As discussed previously, the operator of the system provides the input 302 to indicate a desire to transition back to the manipulation input mode 402. During the manipulation input mode 402, the computer system allows the operator to move the medical instrument 404 by operating the controller. Some misalignment may exist between the controller and the medical instrument 404 when the computer system first transitions back to the manipulation input mode 402.

[0074] The computer system first detects a position 702 of the controller. The computer system compares the position 702 of the controller with the position of the medical instrument 404 to determine whether the controller is misaligned with the medical instrument 404. The computer system may also determine a magnitude of the misalignment. The computer system applies a corrective offset 704 based on the magnitude of the misalignment between the controller and the medical instrument 404. The offset 704 may be applied to the translational movement 308 and / or the rotational movement 310 indicated by the input 306. The offset 704 corrects or reduces the effects of the misalignment between the controller and the medical instrument 404. When the offset 704 is applied to the translational movement 308 and / or the rotational movement 310, the offset 704 creates the impression that the controller is not as misaligned with the medical instrument 404. Thus, the offset 704 makes control of the medical instrument 404 feel more intuitive to the operator relative to if the offset 704 is not used. As a result, the computer system improves the health and safety of the patient as the medical instrument 404 is moved.

[0075] The computer system gradually brings the controller back into alignment with the medical instrument 404 as the controller is moved during the manipulation input mode 402. For example, the computer system may move the medical instrument 404 by amounts less than the amounts indicated by the translational movement 308 or the rotational movement 310 to bring the medical instrument 404 closer to alignment with the controller. When the medical instrument 404 is aligned with the controller, the computer system begins moving the medical instrument 404 by the amounts indicated by the translational movement 308 or the rotational movement 310. During this process, the computer system gradually reduces the offset 704 as the controller is moved closer to alignment with the medical instrument 404. When the computer system has brought the controller back into alignment with the medical instrument 404, the computer system stops using or applying an offset 704.

[0076] In some embodiments, the computer system applies a corrective force 706 to the controller while transitioning to the manipulation input mode 402. For example, the computer system may determine from the position 702 of the controller that the controller had rotated out of the virtual shape 506 during the virtual control mode 502 or 602. In response, the computer system applies the force 706 to the controller torotate the controller back into the virtual shape 506 before allowing the operator to provide the input 306. As a result, the computer system brings the controller back closer to aligning with the medical instrument 404 before allowing the operator to provide the input 306. There may still be some misalignment between the controller and the medical instrument 404, but the misalignment may be reduced by applying the force 706. In this manner, the computer system improves the safety for the patient by reducing the misalignment between the controller and the medical instrument 404.

[0077] In some embodiments, the computer system implements a firewall 708 that prevents the input 306, the translational movement 308, and / or the rotational movement 310 from being communicated to the medical instrument 404 during the manipulation input mode 402 until the computer system has moved the controller closer to alignment with the medical instrument 404. For example, when the computer system applies the force 706 to the controller to rotate the controller back into the virtual shape 506, the computer system activates the firewall 708 until the controller has been rotated back into the virtual shape 506. As a result, if the operator of the computer system operates the controller before the controller has rotated back into the virtual shape 506, the firewall 708 will block the signals from the controller from reaching the medical instrument 404. As a result, the medical instrument 404 may not be controlled or moved until the controller has been rotated back into the virtual shape 506, which improves the safety of the patient.

[0078] Figure 8 is a flowchart of an example method 800 performed in a medical system. In particular embodiments, the computer system performs the method 800. By performing the method 800, the computer system transitions between different control or input modes.

[0079] In block 802, the computer system enters a manipulation input mode 402. The computer system receives the input 302 from an operator of the computer system using a controller (e.g., input devices 126 or 128). During the manipulation input mode 402, the computer system allows the operator of the computer system to move or operate a medical instrument 404 by operating the controller.

[0080] In block 804, the computer system transitions to a virtual control mode 502 or 602. The computer system receives the input 302 from the operator of the computersystem that indicates a desire to transition to the virtual control mode 502 or 602. During the virtual control mode 502 or 602, the computer system allows the operator of the computer system to interact with or manipulate user interface elements. Additionally, during the virtual control mode 502 or 602, the computer system prevents the operator from moving or operating the medical instrument 404. For example, during the virtual control mode 502 or 602, the operator of the computer system may move a controller to move or manipulate a model 504 of an anatomical structure without moving or operating the medical instrument 404.

[0081] During the virtual control mode 502 or 602, the computer system limits or resists rotational movement 310 that would cause the controller to rotate out of a virtual shape 506. For example, the computer system may apply a resistive force to the controller that opposes the rotational movement 310 of the controller when the controller has rotated near an edge of the virtual shape 506. In this manner, the computer system encourages the controller to remain aligned, or remain within a threshold alignment with the medical instrument 404.

[0082] In block 806, the computer system transitions back to the manipulation input mode 402. The operator of the computer system provides the input 302 that indicates a desire to transition back to the manipulation input mode of 402. The computer system then transitions back to the manipulation input mode 402 to allow the operator of the computer system to move or operate the medical instrument 404.

[0083] In some embodiments, before allowing the operator of the computer system to move or operate the medical instrument 404 during the manipulation input mode 402, the computer system corrects or adjusts for some of the misalignment between the controller and the medical instrument 404. For example, the computer system may apply a corrective offset 704 to the movement of the controller during the manipulation input mode 402. The offset 704 reduces the effects of the misalignment between the controller and the medical instrument 404. The computer system may also bring the controller closer or back to alignment with the medical instrument 404 as the controller is moved during the manipulation input mode 402. For example, the computer system may apply the force 706 that rotates the controller back into the virtual shape 506.

[0084] Figure 9 is a flowchart of an example method 900 performed in the medical system. In particular embodiments, the computer system performs the method 900. By performing the method 900, the computer system maintains an alignment between a controller and a medical instrument 404 or maintains the controller within a threshold alignment of the medical instrument 404.

[0085] In block 902, the computer system displays the model 504 of an anatomical structure. The computer system may display the model 504 during the virtual control mode 502 or 602. In block 904, the computer system rotates the model 504. The computer system may rotate the model 504 according to rotational movement 310 of the controller. The rotation of the model 504 may be in the same direction as the rotational movement 310 and by an amount indicated by the rotational movement 310.

[0086] In block 906, the computer system resists the rotational movement 310 of the controller. Specifically, when the controller has rotated close to an edge of the virtual shape 506, the computer system applies a resistive force to the controller that opposes the rotational movement 310. In this manner, the computer system discourages the controller from further being rotated out of the virtual shape 506, which helps keep the controller within a threshold alignment of the medical instrument 404.

[0087] In block 908, the computer system translates the model 504. The computer system may translate the model 504 according to translational movement 308 of the controller. The computer system may translate the model 504 in the same direction as the translational movement 308 and by an amount indicated by the translational movement 308. Additionally, the computer system may translate the virtual shape 506 according to the translational movement 308. The computer system may translate the virtual shape 506 in the same direction as the translational movement 308 and by an amount that keeps the virtual shape 506 in the same relative position with the controller. In this manner, the virtual shape 506 moves or translates with the controller during translational movement 308.

[0088] In certain embodiments, the computer system provides other virtual control modes in which the virtual shape that limits or restricts the rotation of the controller is disabled. When the virtual shape is disabled, the controller rotates freely, even if the rotation would cause the controller to become misaligned with the medical instrument.In the tethered configuration, the controllers rotate freely while maintaining the relative distance between the controllers, even if the rotation would cause the controllers to become misaligned with medical instruments. When the controllers rotate or translate, the computer system manipulates user interface elements according to the rotation or translation.

[0089] In summary, the medical system 100 provides various features that help a medical instrument remain more aligned with a user input device during a virtual control mode. For example, the medical system 100 may track the rotational movement of the user input device during the virtual control mode and resist the rotation of the user input device if rotation would cause the user input device to become too misaligned with the medical instrument. The medical system 100 may track the rotational movement of the user input device using a virtual shape. As the user input device rotates and approaches an edge of the virtual shape, the medical system 100 may cause a force to be applied to the user input device that resists the rotation of the user input device closer towards the edge of the virtual shape. The magnitude of the force may increase as the user input device rotates closer towards the edge of the virtual shape.

[0090] This description and the accompanying drawings that illustrate aspects, embodiments, or modules should not be taken as limiting. Various mechanical, compositional, structural, electrical, and operational changes may be made without departing from the spirit and scope of this description and the claims. In some instances, well-known circuits, structures, or techniques have not been shown or described in detail in order not to obscure other features. Like numbers in two or more figures represent the same or similar elements.

[0091] In this description, specific details are set forth describing some embodiments consistent with the present disclosure. Numerous specific details are set forth in order to provide a thorough understanding of the embodiments. It will be apparent, however, to one skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are meant to be illustrative but not limiting. One skilled in the art may realize other elements that, although not specifically described here, are within the scope and the spirit of this disclosure. In addition, to avoid unnecessary repetition, one or morefeatures shown and described in association with one embodiment may be incorporated into other embodiments unless specifically described otherwise or if the one or more features would make an embodiment non-functional.

[0092] Further, the terminology in this description is not intended to be limiting. For example, spatially relative terms-such as “beneath”, “below”, “lower”, “above”, “upper”, “proximal”, “distal”, and the like may be used to describe one element’s or feature’s relationship to another element or feature as illustrated in the figures. These spatially relative terms are intended to encompass different positions (i.e., locations) and orientations (i.e., rotational placements) of the elements or their operation in addition to the position and orientation shown in the figures. For example, if the content of one of the figures is turned over, elements described as “below” or “beneath” other elements or features would then be “above” or “over” the other elements or features. Thus, the exemplary term “below” can encompass both positions and orientations of above and below. A device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly. Likewise, descriptions of movement along and around various axes include various special element positions and orientations. In addition, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, unless the context indicates otherwise. And, the terms “comprises”, “comprising”, “includes”, and the like specify the presence of stated features, steps, operations, elements, and / or components but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups. Components described as coupled may be electrically or mechanically directly coupled, or they may be indirectly coupled via one or more intermediate components.

[0093] Elements described in detail with reference to one embodiment, or module may, whenever practical, be included in other embodiments, or modules in which they are not specifically shown or described. For example, if an element is described in detail with reference to one embodiment and is not described with reference to a second embodiment, the element may nevertheless be claimed as included in the second embodiment. Thus, to avoid unnecessary repetition in the following description, one or more elements shown and described in association with one embodiment, or application may be incorporated into other embodiments, or aspectsunless specifically described otherwise, unless the one or more elements would make an embodiment or embodiments non-functional, or unless two or more of the elements provide conflicting functions.

[0094] In some instances, well known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0095] This disclosure describes various devices, elements, and portions of computer-assisted devices and elements in terms of their state in three-dimensional space. As used herein, the term “position” refers to the location of an element or a portion of an element in a three-dimensional space (e.g., three degrees of translational freedom along Cartesian x-, y-, and z-coordinates). As used herein, the term “orientation” refers to the rotational placement of an element or a portion of an element (three degrees of rotational freedom - e.g., roll, pitch, and yaw). As used herein, the term “shape” refers to a set positions or orientations measured along an element. As used herein, and for a device with repositionable arms, the term “proximal” refers to a direction toward the base of the computer-assisted device along its kinematic chain and “distal” refers to a direction away from the base along the kinematic chain.

[0096] Aspects of this disclosure are described in reference to computer-assisted systems and devices, which may include systems and devices that are teleoperated, remote-controlled, autonomous, semiautonomous, robotic, and / or the like. Further, aspects of this disclosure are described in terms of an embodiment using a medical system, such as the DA VINCI SURGICAL SYSTEM or ION SYSTEM commercialized by Intuitive Surgical, Inc. of Sunnyvale, California. Knowledgeable persons will understand, however, that aspects disclosed herein may be embodied and implemented in various ways, including robotic and, if applicable, non-robotic embodiments. Techniques described with reference to surgical instruments and surgical methods may be used in other contexts. Thus, the instruments, systems, and methods described herein may be used for humans, animals, portions of human or animal anatomy, industrial systems, general robotic, or teleoperational systems. As further examples, the instruments, systems, and methods described herein may be used for non-medical purposes including industrial uses, general robotic uses, sensing or manipulating non-tissue work pieces, cosmetic improvements, imaging of human oranimal anatomy, gathering data from human or animal anatomy, setting up or taking down systems, training medical or non-medical personnel, and / or the like. Additional example applications include use for procedures on tissue removed from human or animal anatomies (with or without return to a human or animal anatomy) and for procedures on human or animal cadavers. Further, these techniques can also be used for medical treatment or diagnosis procedures that include, or do not include, surgical aspects.

[0097] Although illustrative embodiments have been shown and described, a wide range of modification, change and substitution is contemplated in the foregoing disclosure and in some instances, some features of the embodiments may be employed without a corresponding use of other features. One of ordinary skill in the art would recognize many variations, alternatives, and modifications. Thus, the scope of the disclosure should be limited only by the following claims, and it is appropriate that the claims be construed broadly and, in a manner, consistent with the scope of the embodiments disclosed herein.

Claims

WHAT IS CLAIMED IS:

1. A computer system for manipulating models, the computer system comprising: a display; a first user input device; a second user input device; a memory; and a processor communicatively coupled to the memory, the processor configured to: present, on the display, a model of an object; rotate the model based on rotational movement of at least one of the first user input device or the second user input device within a virtual shape; resist, at an edge of the virtual shape, the rotational movement of at least one of the first user input device or the second user input device; and translate the model and the virtual shape based on translational movement of at least one of the first user input device or the second user input device.

2. The computer system of Claim 1 , wherein the processor is further configured to tether the first user input device and the second user input device together to maintain a relative distance between the first user input device and the second user input device during the translational movement.

3. The computer system of Claim 2, wherein translation of the model and the virtual shape is based on movement of a midpoint between the first user input device relative to the second user input device.

4. The computer system of Claim 2, wherein tethering the first user input device and the second user input device together prevents movement of the first user input device and the second user input device that would change the relative distance during the translational movement.

5. The computer system of Claim 1 , wherein the processor is further configured to, before rotation of the model, allow rotation of the model in response to selection,through user input on the first user input device or the second user input device, of a widget presented on the display.

6. The computer system of Claim 5, wherein the processor is further configured to assist the rotational movement of at least one of the first user input device or the second user input device.

7. The computer system of Claim 6, wherein assistance of the rotation comprises applying a force to at least one of the first user input device or the second user input device, wherein the force is in a direction of the rotation.

8. The computer system of Claim 7, wherein a magnitude of the force is based on a position of at least one of the first user input device or the second user input device.

9. The computer system of Claim 1 , wherein the processor is further configured to scale rotation of the model higher at the edge of the virtual shape.

10. The computer system of Claim 1 , wherein the processor is further configured to transition to a manipulation input mode in which the rotational movement and translational movement of at least one of the first user input device or the second user input device controls a surgical instrument.

11. The computer system of Claim 10, wherein the processor is further configured to apply an offset to the rotational movement and the translational movement of at least one of the first user input device or the second user input device after transitioning to the manipulation input mode.

12. The computer system of Claim 11 , wherein the offset is based on an alignment of at least one of the first user input device or the second user input device with the surgical instrument.

13. The computer system of Claim 11 , wherein the processor is further configured to reduce the offset over time after transitioning to the manipulation input mode.

14. The computer system of Claim 10, wherein the processor is further configured to detect that at least one of the first user input device or the second user input device rotated out of the virtual shape.

15. The computer system of Claim 14, wherein the processor is further configured to rotate at least one of the first user input device or the second user input device to offset the rotation out of the virtual shape before transitioning to the manipulation input mode.

16. The computer system of Claim 15, wherein a firewall prevents a control signal from reaching the surgical instrument until at least one of the first user input device or the second user input device has been rotated to offset the rotation out of the virtual shape.

17. The computer system of Claim 1 , wherein the processor is further configured to cause a force to be applied to at least one of the first user input device or the second user input device to resist the rotational movement.

18. The computer system of Claim 1 , wherein the model of the object is presented over a video of the object.

19. The computer system of Claim 1 , wherein the processor is further configured to refrain from moving a surgical instrument based on the translational movement.

20. The computer system of Claim 1 , wherein the virtual shape is at least one of a cone, a pyramid, a cylinder, or a sphere.

21. The computer system of Claim 1 , wherein the processor is further configured to adjust a size of the virtual shape based on a roll angle of at least one of the first user input device or the second user input device.

22. The computer system of Claim 21 , wherein the rotational movement adjusts a pitch angle or a yaw angle of at least one of the first user input device or the second user input device.

23. A method for manipulating models, the method comprising:presenting, on a display, a model of an object; rotating the model based on rotational movement of at least one of a first user input device or a second user input device within a virtual shape; resisting, at an edge of the virtual shape, the rotational movement of at least one of the first user input device or the second user input device; and translating the model and the virtual shape based on translational movement of at least one of the first user input device or the second user input device.

24. The method of Claim 23, further comprising tethering the first user input device and the second user input device together to maintain a relative distance between the first user input device and the second user input device during the translational movement.

25. The method of Claim 24, wherein translation of the model and the virtual shape is based on movement of a midpoint between the first user input device relative to the second user input device.

26. The method of Claim 24, wherein tethering the first user input device and the second user input device together prevents movement of the first user input device and the second user input device that would change the relative distance during the translational movement.

27. The method of Claim 23, further comprising, before rotation of the model, allowing rotation of the model in response to selection, through user input on the first user input device or the second user input device, of a widget presented on the display.

28. The method of Claim 27, further comprising assisting the rotational movement of at least one of the first user input device or the second user input device.

29. The method of Claim 28, wherein assisting the rotation comprises applying a force to at least one of the first user input device or the second user input device, wherein the force is in a direction of the rotation.

30. The method of Claim 29, wherein a magnitude of the force is based on a position of at least one of the first user input device or the second user input device.

31. The method of Claim 23, further comprising scaling rotation of the model higher at the edge of the virtual shape.

32. The method of Claim 23, further comprising transitioning to a manipulation input mode in which the rotational movement and translational movement of at least one of the first user input device or the second user input device controls a surgical instrument.

33. The method of Claim 32, further comprising applying an offset to the rotational movement and the translational movement of at least one of the first user input device or the second user input device after transitioning to the manipulation input mode.

34. The method of Claim 33, wherein the offset is based on an alignment of at least one of the first user input device or the second user input device with the surgical instrument.

35. The method of Claim 33, further comprising reducing the offset over time after transitioning to the manipulation input mode.

36. The method of Claim 32, further comprising detecting that at least one of the first user input device or the second user input device rotated out of the virtual shape.

37. The method of Claim 36, further comprising rotating at least one of the first user input device or the second user input device to offset the rotation out of the virtual shape before transitioning to the manipulation input mode.

38. The method of Claim 37, wherein a firewall prevents a control signal from reaching the surgical instrument until at least one of the first user input device or the second user input device has been rotated to offset the rotation out of the virtual shape.

39. The method of Claim 23, further comprising causing a force to be applied to at least one of the first user input device or the second user input device to resist the rotational movement.

40. The method of Claim 23, wherein the model of the object is presented over a video of the object.

41. The method of Claim 23, further comprising refraining from moving a surgical instrument based on the translational movement.

42. The method of Claim 23, wherein the virtual shape is at least one of a cone, a pyramid, a cylinder, or a sphere.

43. The method of Claim 23, further comprising adjusting a size of the virtual shape based on a roll angle of at least one of the first user input device or the second user input device.

44. The method of Claim 43, wherein the rotational movement adjusts a pitch angle or a yaw angle of at least one of the first user input device or the second user input device.

45. A non-transitory machine-readable medium storing instructions for adjusting models of anatomical objects that, when executed by a processor, cause the processor to: perform the method of any of Claims 23 through 44.

Images