Systems and methods for controlling a virtual cursor using a robotic tracking unit

Abstract

Provided systems for controlling a virtual cursor include: a robotic arm; a robotic tracking unit attached to the robotic arm, the robotic tracking unit supporting and orienting an end effector; sensor(s) positioned on the robotic tracking unit and sensing a force applied to the robotic tracking unit; a display displaying a virtual cursor movable within a Graphical User Interface; processor(s); and memory(ies) storing instructions for execution by the processor(s) that, when executed, cause the processor(s) to: receive sensor data from the sensor(s), which corresponds to the force applied to the robotic tracking unit and a duration of time that the force is applied to the robotic tracking unit, determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit, and cause the virtual cursor to move in the determined virtual cursor direction for the duration of time.

Description

A0011933W001SYSTEMS AND METHODS FOR CONTROLLING A VIRTUAL CURSOR USING A ROBOTIC TRACKING UNITCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application Serial No. 63 / 699,562, filed 26 September 2024, the entire content of which is incorporated herein by reference.BACKGROUND

[0002] The present disclosure is generally directed to controlling a virtual cursor, and relates more particularly to controlling a virtual cursor using a robotic tracking unit of a robotic system.

[0003] Surgical robots may assist a surgeon or other medical provider in carrying out a surgical procedure, or may complete one or more surgical procedures autonomously. Providing controllable linked articulating members allows a surgical robot to reach areas of a patient anatomy during various medical procedures.BRIEF SUMMARY

[0004] Example aspects of the present disclosure include:

[0005] A system for controlling a virtual cursor according to at least one embodiment of the present disclosure comprises: a robotic arm; a robotic tracking unit attached to the robotic arm, the robotic tracking unit configured to support and orient an end effector; at least one sensor positioned on the robotic tracking unit, the at least one sensor configured to sense a force applied to the robotic tracking unit; a display configured to display a virtual cursor movable within a Graphical User Interface (GUI); at least one processor; and at least one memory storing instructions for execution by the at least one processor that, when executed, cause the at least one processor to: receive sensor data from the sensor, the sensor data corresponding to the force applied to the robotic tracking unit and a duration of time that the force is applied to the robotic tracking unit, determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit, and cause the virtual cursor to move in the determined virtual cursor direction for the duration of time.

[0006] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a first pattern of forces received by the sensorA0011933W001 data, determine that at least one of the first pattern of forces and a voice command correspond to entering a planning mode, and cause the display to enter the planning mode and enable control of the virtual cursor via the robotic tracking unit.

[0007] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a second pattern of forces received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display to exit the planning mode and disable control of the virtual cursor via the robotic tracking unit.

[0008] Any of the aspects herein, wherein the first pattern and the second pattern are the same.

[0009] Any of the aspects herein, wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

[0010] Any of the aspects herein, wherein the robotic tracking unit includes a joystick in communication with the at least one sensor, wherein the force is applied to the joystick.

[0011] Any of the aspects herein, wherein the robotic tracking unit includes at least one button in communication with the at least one sensor, wherein the force is applied to the at least one button.

[0012] Any of the aspects herein, wherein the robotic tracking unit includes a touchpad in communication with the at least one sensor, wherein the force is applied to the touchpad.

[0013] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor to select an object in the GUI.

[0014] A system for controlling a virtual cursor according to at least one embodiment of the present disclosure comprises: a robotic arm; a robotic tracking unit attached to the robotic arm, the robotic tracking unit configured to support and orient an end effector; at least one sensor positioned on the robotic tracking unit, the at least one sensor configured to sense a force applied to the robotic tracking unit; a display configured to display a virtual cursor movable within a Graphical User Interface (GUI); at least one processor; and at least one memory storing instructions for execution by the at least one processor that, when executed, cause the at least one processor to: receive sensor data from the sensor comprising a first pattern of forces received by the sensor data, determine thatA0011933W001 the first pattern of forces correspond to entering a planning mode, and cause the display to enter the planning mode and enable control of the virtual cursor via the robotic tracking unit.

[0015] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising at least one of a second pattern of forces and a voice command received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display to exit the planning mode and disable control of the virtual cursor via the robotic tracking unit.

[0016] Any of the aspects herein, wherein the first pattern and the second pattern are the same.

[0017] Any of the aspects herein, wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

[0018] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor to select an object in the GUI.

[0019] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor, the sensor data corresponding to the force applied to the robotic tracking unit and a duration of time that the force is applied to the robotic tracking unit, determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit, and cause the virtual cursor to move in the determined virtual cursor direction for the duration of time.

[0020] A system for controlling a virtual cursor according to at least one embodiment of the present disclosure comprises: a robotic arm; a robotic tracking unit attached to the robotic arm, the robotic tracking unit configured to support and orient an end effector; at least one sensor positioned on the robotic tracking unit, the at least one sensor configured to sense a force applied to the robotic tracking unit; a display configured to display a virtual cursor movable within a Graphical User Interface (GUI); at least one processor; and at least one memory storing instructions for execution by the at least one processor that, when executed, cause the at least one processor to: receive sensor data from the sensor comprising a force received for less than a predetermined duration of time,A0011933W001 determine that the force corresponds to a selection command, and cause the virtual cursor to select an object in the GUI.

[0021] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a first pattern of forces received by the sensor data, determine that the first pattern of forces correspond to entering a planning mode, and cause the display to enter the planning mode and enable control of the virtual cursor via the robotic tracking unit.

[0022] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor comprising a second pattern of forces received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display to exit the planning mode and disable control of the virtual cursor via the robotic tracking unit.

[0023] Any of the aspects herein, wherein the first pattern and the second pattern are the same.

[0024] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: receive sensor data from the sensor, the sensor data corresponding to the force applied to the robotic tracking unit and a duration of time that the force is applied to the robotic tracking unit, determine at least one of a virtual cursor direction and a virtual cursor speed corresponding to the force applied to the robotic tracking unit, and cause the virtual cursor to move in the determined virtual cursor direction for the duration of time.

[0025] Any of the aspects herein, wherein the memory stores additional instructions for execution by the at least one processor that, when executed, further cause the at least one processor to: cause, upon entering the planning mode, at least one of the robotic arm and the robotic tracking unit to move into a predetermined pose associated with the planning mode.

[0026] Any aspect in combination with any one or more other aspects.

[0027] Any one or more of the features disclosed herein.

[0028] Any one or more of the features as substantially disclosed herein.

[0029] Any one or more of the features as substantially disclosed herein in combination with any one or more other features as substantially disclosed herein.A0011933W001

[0030] Any one of the aspects / features / embodiments in combination with any one or more other aspects / features / embodiments.

[0031] Use of any one or more of the aspects or features as disclosed herein.

[0032] It is to be appreciated that any feature described herein can be claimed in combination with any other feature(s) as described herein, regardless of whether the features come from the same described embodiment.

[0033] The details of one or more aspects of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques described in this disclosure will be apparent from the description and drawings, and from the claims.

[0034] The phrases “at least one”, “one or more”, and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together. When each one of A, B, and C in the above expressions refers to an element, such as X, Y, and Z, or class of elements, such as XI -Xn, Yl-Ym, and Zl- Zo, the phrase is intended to refer to a single element selected from X, Y, and Z, a combination of elements selected from the same class (e.g., XI and X2) as well as a combination of elements selected from two or more classes (e.g., Y 1 and Zo).

[0035] The term “a” or “an” entity refers to one or more of that entity. As such, the terms “a” (or “an”), “one or more” and “at least one” can be used interchangeably herein. It is also to be noted that the terms “comprising”, “including”, and “having” can be used interchangeably.

[0036] The preceding is a simplified summary of the disclosure to provide an understanding of some aspects of the disclosure. This summary is neither an extensive nor exhaustive overview of the disclosure and its various aspects, embodiments, and configurations. It is intended neither to identify key or critical elements of the disclosure nor to delineate the scope of the disclosure but to present selected concepts of the disclosure in a simplified form as an introduction to the more detailed description presented below. As will be appreciated, other aspects, embodiments, and configurations of the disclosure are possible utilizing, alone or in combination, one or more of the features set forth above or described in detail below.A0011933W001

[0037] Numerous additional features and advantages of the present disclosure will become apparent to those skilled in the art upon consideration of the embodiment descriptions provided hereinbelow.BRIEF DESCRIPTION OF SEVERAL VIEWS OF THE DRAWINGS

[0038] The accompanying drawings are incorporated into and form a part of the specification to illustrate several examples of the present disclosure. These drawings, together with the description, explain the principles of the disclosure. The drawings simply illustrate preferred and alternative examples of how the disclosure can be made and used and are not to be construed as limiting the disclosure to only the illustrated and described examples. Further features and advantages will become apparent from the following, more detailed, description of the various aspects, embodiments, and configurations of the disclosure, as illustrated by the drawings referenced below.

[0039] Fig. 1A shows aspects of a system according to at least one embodiment of the present disclosure;

[0040] Fig. IB shows additional aspects of the system according to at least one embodiment of the present disclosure;

[0041] Fig. 1C shows aspects of a robotic tracking unit according to at least one embodiment of the present disclosure;

[0042] Fig. ID shows additional aspects of the system according to at least one embodiment of the present disclosure;

[0043] Fig. IE shows aspects of the robotic tracking unit and a display according to at least one embodiment of the present disclosure; and

[0044] Fig. 2 is a flowchart according to at least one embodiment of the present disclosure.

[0045] Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of examples, aspects, and features illustrated.

[0046] In some instances, the apparatus and method components have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the of various embodiments, examples, aspects, and features so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.A0011933W001DETAILED DESCRIPTION

[0047] It should be understood that various aspects disclosed herein may be combined in different combinations than the combinations specifically presented in the description and accompanying drawings. It should also be understood that, depending on the example or embodiment, certain acts or events of any of the processes or methods described herein may be performed in a different sequence, and / or may be added, merged, or left out altogether (e.g., all described acts or events may not be necessary to carry out the disclosed techniques according to different embodiments of the present disclosure). In addition, while certain aspects of this disclosure are described as being performed by a single module or unit for purposes of clarity, it should be understood that the techniques of this disclosure may be performed by a combination of units or modules associated with, for example, a computing device and / or a medical device.

[0048] In one or more examples, the described methods, processes, and techniques may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Alternatively or additionally, functions may be implemented using machine learning models, neural networks, artificial neural networks, or combinations thereof (alone or in combination with instructions). Computer-readable media may include non-transitory computer-readable media, which corresponds to a tangible medium such as data storage media (e.g., RAM, ROM, EEPROM, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer).

[0049] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors (e.g., Intel Core i3, i5, i7, or i9 processors; Intel Celeron processors; Intel Xeon processors; Intel Pentium processors; AMD Ryzen processors; AMD Athlon processors; AMD Phenom processors; Apple A10 or 10X Fusion processors; Apple Al 1, A 12, A12X, A12Z, or A13 Bionic processors; or any other general purpose microprocessors), graphics processing units (e.g., Nvidia GeForce RTX 2000-series processors, Nvidia GeForce RTX 3000-series processors, AMD Radeon RX 5000-series processors, AMD Radeon RX 6000-series processors, or any other graphics processing units), application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term “processor” as used herein may referA0011933W001 to any of the foregoing structure or any other physical structure suitable for implementation of the described techniques. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0050] Before any embodiments of the disclosure are explained in detail, it is to be understood that the disclosure is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The disclosure is capable of other embodiments and of being practiced or of being carried out in various ways. Also, it is to be understood that the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including,” “comprising,” or “having” and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items. Further, the present disclosure may use examples to illustrate one or more aspects thereof. Unless explicitly stated otherwise, the use or listing of one or more examples (which may be denoted by “for example,” “by way of example,” “e.g.,” “such as,” or similar language) is not intended to and does not limit the scope of the present disclosure.

[0051] The terms proximal and distal are used in this disclosure with their conventional medical meanings, proximal being closer to the operator or user of the system, and further from the region of surgical interest in or on the patient, and distal being closer to the region of surgical interest in or on the patient, and further from the operator or user of the system.

[0052] During a surgical operation or procedure users such as, for example, surgeons may wish to adjust or create a surgical or procedure plan during the surgical operation or procedure. However, access to a main workstation using a mouse or touchscreen to adjust or create the surgical or procedure plan is via a non-sterile environment which the surgeon cannot access during the surgical procedure as the surgeon is in a sterile environment.

[0053] Thus, systems and methods according to the present disclosure enable the use of a tracked portion or a robotic tracking unit as a computer mouse in the sterile environment. More specifically, the robotic tracking unit includes a sensor such as, for example, a 6-axis force sensor or a forcetorque sensor that can be used to enter and exit a planning mode and to control a virtual cursor within a surgical planning program. The robotic tracking unit may include one or more buttons, a touchpad, and / or a joy stick that can be used to receive the user input. In some embodiments, the robotic tracking unit may have a predefined position in which the robotic tracking unit is used to control the virtual cursor and / or the surgical planning program. The systems and methods canA0011933W001 improve the human-machine interface by allowing surgeons more functions in real-time, during a procedure, and within a sterile environment to improve patient care.

[0054] Embodiments of the present disclosure provide technical solutions to one or more of the problems of (1) adjusting, updating, or creating a surgical plan or a procedure plan within a sterile environment, (2) controlling a virtual cursor via a robotic tracking unit of a robotic system in a sterile environment, and / or (3) improving patient and the surgical team’s safety.

[0055] With reference to Figs. 1A-1C, a block diagram of a system 100 according to at least one embodiment of the present disclosure are shown. The system 100 may be used to control a virtual cursor 150 using, for example, a Robot Tracking Unit (RTU) 136 of a robot 114 and / or to carry out one or more other aspects of one or more of the methods disclosed herein. As shown in Fig. 1A, the system 100 comprises a computing device 102, one or more imaging devices 112, the robot 114, a navigation system 118, a database 130, and / or a cloud or other network 134. Systems according to other embodiments of the present disclosure may comprise more or fewer components than the system 100. For example, the system 100 may not include the imaging device 112, the robot 114, the navigation system 118, one or more components of the computing device 102, the database 130, and / or the cloud 134.

[0056] The computing device 102 comprises a processor 104, a memory 106, a communication interface 108, and a user interface 110. Computing devices according to other embodiments of the present disclosure may comprise more or fewer components than the computing device 102.

[0057] The processor 104 of the computing device 102 may be any processor described herein or any similar processor. The processor 104 may be configured to execute instructions stored in the memory 106, which instructions may cause the processor 104 to carry out one or more computing steps utilizing or based on data received from the imaging device 112, the robot 114, the navigation system 118, the database 130, and / or the cloud 134.

[0058] The memory 106 may be or comprise RAM, DRAM, SDRAM, other solid-state memory, any memory described herein, or any other tangible, non-transitory memory for storing computer- readable data and / or instructions. The memory 106 may store information or data useful for completing, for example, any step of the method 200 described herein, or of any other methods. The memory 106 may store, for example, instructions and / or machine learning models that support one or more functions of the robot 114. For instance, the memory 106 may store content (e.g.,A0011933W001 instructions and / or machine learning models) that, when executed by the processor 104, enable a sensor processing 120.

[0059] The sensor processing 120 enables the processor 104 to process sensor data received from, for example, a sensor 142. The sensor data may be processed to obtain, for example, a load exerted onto an object such as a robotic arm 116, a RTU (Robotic Transfer Unit) 136, a tool changer 140, an end effector 146, or any other portion of the robot 114. The load may be a force and / or torque exerted onto the object. In other embodiments, the sensor data may be processed to obtain, for example, a pose of the object such as, for example, the robotic arm 116, the RTU 136, the tool changer 140, the end effector 146, or any other portion of the robot 114. Alternatively or additionally, pose information may be obtained from, for example, the navigation system 118. The sensor data may also be processed to obtain, for example, a virtual cursor direction, a command to enter / exit a planning mode, and / or any other command to execute within the planning mode, as will be discussed in detail in Figs. ID, IE, and 2.

[0060] Such content, if provided as instructions, may, in some embodiments, be organized into one or more applications, modules, packages, layers, or engines. Alternatively or additionally, the memory 106 may store other types of content or data (e.g., machine learning models, artificial neural networks, deep neural networks, etc.) that can be processed by the processor 104 to carry out the various methods and features described herein. Thus, although various contents of memory 106 may be described as instructions, it should be appreciated that functionality described herein can be achieved through use of instructions, algorithms, and / or machine learning models. The data, algorithms, and / or instructions may cause the processor 104 to manipulate data stored in the memory 106 and / or received from or via the imaging device 112, the robot 114, the database 130, and / or the cloud 134.

[0061] The computing device 102 may also comprise a communication interface 108. The communication interface 108 may be used for receiving image data or other information from an external source (such as the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud 134, and / or any other system or component not part of the system 100), and / or for transmitting instructions, images, or other information to an external system or device (e.g., another computing device 102, the imaging device 112, the robot 114, the navigation system 118, the database 130, the cloud 134, and / or any other system or component not part of the system 100). The communication interface 108 may comprise one or more wired interfaces (e.g., a USBA0011933W001 port, an Ethernet port, a Firewire port) and / or one or more wireless transceivers or interfaces (configured, for example, to transmit and / or receive information via one or more wireless communication protocols such as 802.1 la / b / g / n, Bluetooth, NFC, ZigBee, and so forth). In some embodiments, the communication interface 108 may be useful for enabling the device 102 to communicate with one or more other processors 104 or computing devices 102, whether to reduce the time needed to accomplish a computing-intensive task or for any other reason.

[0062] The computing device 102 may also comprise one or more user interfaces 110. The user interface 110 may be or comprise a keyboard, mouse, trackball, monitor, television, screen, touchscreen, and / or any other device for receiving information from a user and / or for providing information to a user. The user interface 110 may be used, for example, to receive a user selection or other user input regarding any step of any method described herein. Notwithstanding the foregoing, any required input for any step of any method described herein may be generated automatically by the system 100 (e.g., by the processor 104 or another component of the system 100) or received by the system 100 from a source external to the system 100. In some embodiments, the user interface 110 may be useful to allow a surgeon or other user to modify instructions to be executed by the processor 104 according to one or more embodiments of the present disclosure, and / or to modify or adjust a setting of other information displayed on the user interface 110 or corresponding thereto.

[0063] Although the user interface 110 is shown as part of the computing device 102, in some embodiments, the computing device 102 may utilize a user interface 110 that is housed separately from one or more remaining components of the computing device 102. In some embodiments, the user interface 110 may be located proximate one or more other components of the computing device 102, while in other embodiments, the user interface 110 may be located remotely from one or more other components of the computing device 102. As will be discussed in more detail in Figs. ID and IE, the user interface 110 may also include a display 152 for displaying a Graphical User Interface (GUI) 154 and the virtual cursor 150.

[0064] The imaging device 112 may be operable to image anatomical feature(s) (e.g., a bone, veins, tissue, etc.) and / or other aspects of patient anatomy to yield image data (e.g., image data depicting or corresponding to a bone, veins, tissue, etc.). “Image data” as used herein refers to the data generated or captured by an imaging device 112, including in a machine-readable form, a graphical / visual form, and in any other form. In various examples, the image data may compriseA0011933W001 data corresponding to an anatomical feature of a patient, or to a portion thereof. The image data may be or comprise a preoperative image, an intraoperative image, a postoperative image, or an image taken independently of any surgical procedure. In some embodiments, a first imaging device 112 may be used to obtain first image data (e.g., a first image) at a first time, and a second imaging device 112 may be used to obtain second image data (e.g., a second image) at a second time after the first time. The imaging device 112 may be capable of taking a 2D image or a 3D image to yield the image data. The imaging device 112 may be or comprise, for example, an ultrasound scanner (which may comprise, for example, a physically separate transducer and receiver, or a single ultrasound transceiver), an O-arm, a C-arm, a G-arm, or any other device utilizing X-ray-based imaging (e.g., a fluoroscope, a CT scanner, or other X-ray machine), a magnetic resonance imaging (MRI) scanner, an optical coherence tomography (OCT) scanner, an endoscope, a microscope, an optical camera, a thermographic camera (e.g., an infrared camera), a radar system (which may comprise, for example, a transmitter, a receiver, a processor, and one or more antennae), or any other imaging device 112 suitable for obtaining images of an anatomical feature of a patient. The imaging device 112 may be contained entirely within a single housing, or may comprise a transmitter / emitter and a receiver / detector that are in separate housings or are otherwise physically separated.

[0065] In some embodiments, the imaging device 112 may comprise more than one imaging device 112. For example, a first imaging device may provide first image data and / or a first image, and a second imaging device may provide second image data and / or a second image. In still other embodiments, the same imaging device may be used to provide both the first image data and the second image data, and / or any other image data described herein. The imaging device 112 may be operable to generate a stream of image data. For example, the imaging device 112 may be configured to operate with an open shutter, or with a shutter that continuously alternates between open and shut so as to capture successive images. For purposes of the present disclosure, unless specified otherwise, image data may be considered to be continuous and / or provided as an image data stream if the image data represents two or more frames per second.

[0066] The robot 114 may be any surgical robot or surgical robotic system. The robot 114 may be or comprise, for example, the Mazor X™ Stealth Edition robotic guidance system. The robot 114 may be configured to position any component or object such as, for example, the imaging device 112 at one or more precise position(s) and orientation / s), and / or to return the imaging deviceA0011933W001112 to the same position(s) and orientation(s) at a later point in time. The robot 114 may additionally or alternatively be configured to manipulate a surgical tool (whether based on guidance from the navigation system 118 or not) to accomplish or to assist with a surgical task. In some embodiments, the robot 114 may be configured to hold and / or manipulate an anatomical element during or in connection with a surgical procedure. The robot 114 may comprise one or more robotic arms 116. In some embodiments, the robotic arm 116 may comprise a first robotic arm and a second robotic arm, though the robot 114 may comprise more than two robotic arms. In some embodiments, one or more of the robotic arms 116 may be used to hold and / or maneuver the component or object. In embodiments where the imaging device 112 comprises two or more physically separate components (e.g., a transmitter and receiver), one robotic arm 116 may hold one such component, and another robotic arm 116 may hold another such component. Each robotic arm 116 may be positionable independently of the other robotic arm. The robotic arms 116 may be controlled in a single, shared coordinate space, or in separate coordinate spaces.

[0067] The robot 114, together with the robotic arm 116, may have, for example, one, two, three, four, five, six, seven, or more degrees of freedom. Further, the robotic arm 116 may be positioned or positionable in any pose, plane, and / or focal point. The pose includes a position and an orientation. As a result, an imaging device 112, surgical tool, or other object held by the robot 114 (or, more specifically, by the robotic arm 116) may be precisely positionable in one or more needed and specific positions and orientations. The robotic arm(s) 116 may comprise one or more sensors such as, for example, encoders that enable the processor 104 (or a processor of the robot 114) to determine a precise pose in space of the robotic arm (as well as any object or element held by or secured to the robotic arm).

[0068] In some embodiments, the robot 114 may be mechanically coupled with (e.g., affixed to, attached to, mounted to, etc.) a patient bed or table. In other embodiments, the robot 114 may be disposed on a robot cart 144, as shown in Fig. IB. The robot cart 144 may be or comprise a mobile platform that enables the robot 114 and / or components thereof to be positioned relative to the patient and / or the bed or table on which the patient is positioned. In some embodiments, the robot cart 144 may comprise wheels that enable the robot cart 144 to roll or move relative to the patient. The robot cart 144 may be detachable from the wheels or the wheels may lockable such that, once the robot cart 144 is positioned in a desired location relative to the patient, the robot cart 144 will remain fixed in the desired location. In other words, the robot cart 144 may have a mechanism thatA0011933W001 enables the robot cart 144 to remain fixed relative to the patient. The mechanism may better ensure that the robot 114 and / or any other components on the robot cart 144 do not move relative to the patient due to the mobility of the robot cart 144 once the robot cart 144 has been positioned in the desired location.

[0069] In some embodiments, reference markers (e.g., navigation markers) may be placed on the robot 114 (including, e.g., on the robotic arm 116, the RTU 136, the tool changer 140, and / or the end effector 146), the imaging device 112, or any other object in the surgical space. The reference markers may be tracked by the navigation system 118, and the results of the tracking may be used by the robot 114 and / or by an operator of the system 100 or any component thereof. In some embodiments, the navigation system 118 can be used to track other components of the system (e.g., imaging device 112) and the system can operate without the use of the robot 114 (e.g., with the surgeon manually manipulating the imaging device 112 and / or one or more surgical tools, based on information and / or instructions generated by the navigation system 118, for example).

[0070] As shown in Fig. 1C, the navigation markers 148A-148F may be or comprise one or more active markers, one or more passive markers, or a combination of active and passive markers. For example, the navigation markers 148A-148F may comprise a first navigation marker 148A, a second navigation marker 148B, a third navigation marker 148C, a fourth navigation marker 148D, a fifth navigation marker 148E, and a sixth navigation marker 148F. The navigation markers 148A- 148F may be, for example, LEDs, infrared LEDs, reflective markers, and / or the like. The navigation system 118 may be configured to obtain pose information describing a pose of the navigation markers 148A-148F, which may be used to determine a correlating pose of the RTU 136 and, based on the known connection of the RTU 136 to the robotic arm 116, the pose of the robotic arm 116.

[0071] Returning back to Fig. 1A, the navigation system 118 may provide navigation for a surgeon and / or a surgical robot during an operation. The navigation system 118 may be any now- known or future-developed navigation system, including, for example, the Medtronic StealthStation™ S8 surgical navigation system or any successor thereof. The navigation system 118 may include one or more cameras or other sensor(s) for tracking one or more reference markers (e.g., navigation markers 148A-148F), navigated trackers, or other objects within the operating room or other room in which some or all of the system 100 is located. The one or more cameras may be optical cameras, infrared cameras, or other cameras. In some embodiments, the navigationA0011933W001 system 118 may comprise one or more electromagnetic sensors. In various embodiments, the navigation system 118 may be used to track a position and orientation (e.g., a pose) of the imaging device 112, the robot 114 and / or robotic arm 116, and / or one or more surgical tools (or, more particularly, to track a pose of the RTU 136 or other navigated tracker attached, directly or indirectly, in fixed relation to the one or more of the foregoing). The navigation system 118 may include a display for displaying one or more images from an external source (e.g., the computing device 102, imaging device 112, or other source) or for displaying an image and / or video stream from the one or more cameras or other sensors of the navigation system 118. In some embodiments, the system 100 can operate without the use of the navigation system 118. The navigation system 118 may be configured to provide guidance to a surgeon or other user of the system 100 or a component thereof, to the robot 114, or to any other element of the system 100 regarding, for example, a pose of one or more anatomical elements, whether or not a tool is in the proper trajectory, and / or how to move a tool into the proper trajectory to carry out a surgical task according to a preoperative or other surgical plan.

[0072] With reference to Figs. 1A and IB, the RTU 136 of the robot may be or comprise a device that enables the navigation system 118 to track the robotic arm 116. The RTU 136 may have a proximal end that can be coupled to the distal end of the robotic arm 116, and a distal end that can be coupled to a proximal end of a tool changer 140. The RTU 136 comprises navigation markers 148A-148F. The RTU 136 may also include one or more sensors 142.

[0073] The sensor 142 may be any kind of sensor 142 for measuring a value such as, for example, force and / or torque exerted on the RTU 136. The sensor 142 may include one or more or any combination of components that are electrical, mechanical, electro-mechanical, magnetic, electromagnetic, or the like. The sensor 142 may include, but is not limited to, one or more of a torque sensor, a force sensor, a linear encoder, a strain gauge, a rotary encoder, a capacitor, and / or an accelerometer. In some embodiments, the sensor 142 may include a memory for storing sensor data. In still other examples, the sensor 142 may output signals (e.g., sensor data) to one or more sources (e.g., the computing device 102, the navigation system 118, and / or the robot 114).

[0074] In some embodiments, the sensor 142 may be used to sense a user input to enter and / or exit a planning mode (or any other type of on-screen activity that includes a cursor) in which the user interface 110 may display one or more computer programs corresponding to, for example, a surgical plan. In such planning mode, the RTU 136 may be used to maneuver the virtual cursorA0011933W001150 (e.g., the RTU 136 may be used as a mouse) to create an initial surgical plan or to modify an existing surgical plan. The user input may be received via, for example, a touchpad 156 or one or more buttons 158 shown in Figs. 1C and IE. In such instances, the sensor 142 may be in communication with the touchpad 156 or the one or more buttons 158 to monitor or measure a force received via the touchpad 156 and / or the one or more buttons 158. In some examples, the user input may be a voice command input via, for example, a microphone. The user input for entering and / or exiting the planning mode may be, for example, a force held for a duration of time (e.g., the user pressing or holding a button 158 or a touchpad 156 for the duration of time) or a series of forces received in the duration of time (e.g., the user pressing the button or the touchpad multiple times). It will be appreciated that though the illustrated embodiment of the RTU 136 shows both the touchpad 156 and the one or more buttons 158, the RTU 136 may only include the touchpad 156 or the one or more buttons 158 in other instances. In still other instances, the RTU 136 may also include more than one touchpad 156 and / or a joystick through which the user input may be received, and / or a microphone or other audio device through which the user input may be received.

[0075] When the RTU 136 is not in the planning mode, the sensor 142 may be used to measure a value such as the force and / or the torque exerted on the RTU 136 for other purposes such as, for example, monitoring for excessive force and / or torque exerted onto the RTU 136, the robotic arm 116, etc. In such instances, the sensor 142 may send the data to the computing device 102 when the sensor 142 detects a load such as, for example, force and / or torque the RTU 136 and / or the robotic arm 116.

[0076] The tool changer 140 may include a proximal end that is connectable to a distal end of the RTU 136, and a distal end that can be connected to the end effector 146 that can be used to carry out one or more surgical tasks. The end effector 146 may be or comprise a surgical tool. The end effector 146, for example, may support or be a surgical tool that may be configured to autonomously or semi-autonomously drill, burr, mill, cut, saw, ream, tap, etc. into anatomical tissues such as patient anatomy (e.g., soft tissues, bone, etc.). In other examples, the end effector 146 may comprise a trajectory guide through which a surgical tool can be manually operated by the user of the system 100.

[0077] In some embodiments, the system 100 may comprise multiple surgical tools, with each surgical tool performing a different surgical task (e.g., a surgical drill for drilling, a surgical millA0011933W001 for milling, a curette for removing anatomical tissue, an osteotome for cutting bone, etc.). In other embodiments, the surgical tool may provide an adapter interface to which different working ends can be attached to perform multiple different types of surgical maneuvers (e.g., the surgical tool may be able to receive one or more different tool bits, such that the surgical tool can drill, mill, cut, saw, ream, tap, etc. depending on the tool bit coupled with the surgical tool). The surgical tool may be operated autonomously or semi-autonomously. The navigation system 118 may track the pose (e.g., position and orientation) of and / or navigate the surgical tool. For example, the navigation system 118 may identify the navigation markers 148A-148F on the RTU 136 and, based on the identification and the coupling of the RTU 136 with the tool changer 140 and the coupling of the tool changer 140 with the end effector 146, determine the pose of the surgical tool.

[0078] The database 130 may store information that correlates one coordinate system to another (e.g., one or more robotic coordinate systems to a patient coordinate system and / or to a navigation coordinate system). The database 130 may additionally or alternatively store, for example, one or more surgical plans (including, for example, pose information about a target and / or image information about a patient’ s anatomy at and / or proximate the surgical site, for use by the robot 114, the navigation system 118, and / or a user of the computing device 102 or of the system 100); one or more images useful in connection with a surgery to be completed by or with the assistance of one or more other components of the system 100; information about the RTU 136 (e.g., the pose of the navigation markers 148A-148F on the RTU 136, types of connectors and other units that can attach to the RTU 136, etc.); and / or any other useful information. The database 130 may be configured to provide any such information to the computing device 102 or to any other device of the system 100 or external to the system 100, whether directly or via the cloud 134. In some embodiments, the database 130 may be or comprise part of a hospital image storage system, such as a picture archiving and communication system (PACS), a health information system (HIS), and / or another system for collecting, storing, managing, and / or transmitting electronic medical records including image data.

[0079] The cloud 134 may be or represent the Internet or any other wide area network. The computing device 102 may be connected to the cloud 134 via the communication interface 108, using a wired connection, a wireless connection, or both. In some embodiments, the computing device 102 may communicate with the database 130 and / or an external device (e.g., a computing device) via the cloud 134.A0011933W001

[0080] The system 100 or similar systems may be used, for example, to carry out one or more aspects of the method 200 described herein. The system 100 or similar systems may also be used for other purposes.

[0081] Turning to Figs. 1D-1E, the RTU 136 and the user interface 110 in communication with the RTU 136 are shown in block diagram and in schematic form, respectively. As previously mentioned, the user interface 110 may include the display 152 for displaying the GUI 154 and the virtual cursor 150. The GUI 154 may show a planning program 160 for viewing or editing the surgical plan. The virtual cursor 150 may be configured to enable navigation through, for example, the planning program 160 (or any program). In the illustrated embodiment, the virtual cursor 150 is a virtual mouse that is movable relative to the planning program and capable of executing a command such as selecting a virtual button, selecting one or more programs 162, or selecting any feature displayed in the GUI 154. In other embodiments, the virtual cursor 150 may be any shape or size.

[0082] The virtual cursor 150 is movable by a user such as, for example, a surgeon or other medical provider, via the RTU 136, the tool changer 140, and / or end effector 146 after the RTU 136 has been entered into the planning mode. In other words, the user may be able to move the virtual cursor 150 by exerting forces on the RTU 136, the tool changer 140, and / or the end effector 146. In some embodiments, the RTU 136 may be oriented at a specific orientation when in the planning mode. In other embodiments, the RTU 136 can be oriented in any position and / or orientation when in the planning mode. The virtual cursor 150 may be movable based on user input from the RTU 136. The user input may be received via, for example, the sensor 142, the touchpad 156, the one or more buttons 158, a joystick, a microphone, or any other device for receiving the user input. More specifically, the sensor 142 may be in communication with the touchpad 156, the one or more buttons 158, and / or the joystick and may yield sensor data based on the user pressing the touchpad 156 or the one or more buttons 158 or moving the joystick. In other words, the sensor 142 may itself operate as the input device to control the virtual cursor 150. The sensor data generated by the sensor 142 may correspond to the force applied to the RTU 136 received via the sensor 142 coupled with the touchpad 156 and / or the one or more buttons 158 and a duration of time which the force is applied to the RTU 136. A virtual cursor direction may be determined based on the sensor data generated by the sensor 142 and the virtual cursor 150 may be moved based on the virtual cursor direction for the duration of time.A0011933W001

[0083] The sensor data may be transmitted from the sensor 142 wirelessly to the computing device 102. In other embodiments, the RTU 136 may be connected to the computing device 102 via a wired connection. In any embodiment, the user interface 110, and more specifically, the display 152 may be positioned in a non-sterile environment while the RTU 136 is in the sterile environment. In other embodiments, both the display 152 and the RTU 136 may be positioned within the sterile environment.

[0084] By enabling the RTU 136 to control the virtual cursor 150 and the planning program 160, a user such as a surgeon or other medical provider can access the planning program 160 during a surgical procedure using the RTU 136, which is covered by a sterile drape and that is connected to the tool changer 140 and the end effector 146, both of which are sterilized. Conventionally, a computer workstation and / or mouse for accessing the planning program is available in a non-sterile environment, and thus, cannot be accessed by the user in the sterile operating environment. Thus, the RTU 136 enables the user to access the planning program 160 from a sterile environment.

[0085] Fig. 2 depicts a method 200 that may be used, for example, for entering and exiting a planning mode and for controlling a virtual cursor within a planning program.

[0086] The method 200 (and / or one or more steps thereof) may be carried out or otherwise performed, for example, by at least one processor. The at least one processor may be the same as or similar to the processor(s) 104 of the computing device 102 described above. The at least one processor may be part of a robot (such as a robot 114) or part of a navigation system (such as a navigation system 118). A processor other than any processor described herein may also be used to execute the method 200. The at least one processor may perform the method 200 by executing elements stored in a memory such as the memory 106. The elements stored in the memory and executed by the processor may cause the processor to execute one or more steps of a function as shown in method 200. One or more portions of a method 200 may be performed by the processor executing any of the contents of memory, such as sensor processing 120.

[0087] The method 200 comprises receiving sensor data (step 204). The sensor data may correspond to a load such as, for example, a force and / or torque exerted onto a RTU such as the RTU 136.

[0088] The sensor data may be received from, for example, one or more sensors such as the one or more sensors 142. The sensor data may be processed by a processor such as the processor 104 using a sensor processing such as the sensor processing 120. The sensor data may be processed toA0011933W001 determine, for example, whether to enter or exit a planning mode, a virtual cursor direction for moving a virtual cursor such as the virtual cursor 150 within a GUI such as the GUI 154 displayed on a display such as the display 152, and / or a command to execute in the GUI. In other embodiments such as when the RTU is not in the planning mode, the sensor data may be processed to obtain, for example, a pose of the object such as, for example, the robotic arm, the RTU, the end effector, or any other portion of the robot and / or a force exerted on the RTU.

[0089] The sensor may be any kind of sensor for measuring a value such as, for example, force and / or torque. The sensor may include one or more or any combination of components that are electrical, mechanical, electro-mechanical, magnetic, electromagnetic, or the like. The sensor may include, but is not limited to, one or more of a torque sensor, a force sensor, a linear encoder, a strain gauge, a rotary encoder, a capacitor, and / or an accelerometer. In some embodiments, the sensor may include a memory for storing sensor data. In still other examples, the sensor may output signals (e.g., sensor data) to one or more sources (e.g., a computing device such as the computing device 102, the navigation system, and / or the robot).

[0090] In some embodiments, the sensor may be used to sense a user input to enter and / or exit a planning mode in which a user interface such as the user interface 110 may display one or more computer programs on the GUI corresponding to, for example, a surgical plan. The RTU may also be used to maneuver the virtual cursor (e.g., the RTU may be used as a mouse). The user input may be received via, for example, a touchpad such as the touchpad 156 or one or more buttons such as the one or more buttons 158. In such instances, the sensor may be in communication with the touchpad or the one or more buttons to monitor or measure a force received via the touchpad and / or the one or more buttons.

[0091] The method 200 also comprises entering a planning mode (step 208). The planning mode may be entered when the sensor data comprises a pattern of forces that corresponds to a command for entering the planning mode. The pattern of forces may be, for example, a force held for a duration of time (e.g., the user pressing or holding a button or a touchpad for the duration of time) or a series of forces received in the duration of time (e.g., the user pressing the button or the touchpad multiple times). In some embodiments, the processor may cause the RTU and the computing device to enter the planning mode. When in the planning mode, the RTU may be used to control the computer program shown on the display via the virtual cursor. More specifically, theA0011933W001 sensor may measure sensor data, which may then be processed and used for controlling the virtual cursor, as will be described below.

[0092] The method 200 also comprises moving a robotic arm to a planning mode position (step 210). In some embodiments, one or more portions of the robotic arm (e.g., the robotic arm 116) may move to a predetermined and / or specific pose (e.g., position and orientation) when in the planning mode. For example, the RTU 136 may be moved into an orientation that enables the user to interact with the RTU 136 and / or components connected to the RTU 136 (e.g., the tool changer 140, the end effector 146, etc.). The planning mode position may be determined before the start of a surgery or surgical procedure (e.g., the planning mode position may be stored in the memory 106 and / or the database 130) and accessed by the processor 104 to cause the robotic arm to move to the planning mode position when the planning mode is entered. In other embodiments, the robotic arm and / or components attached thereto (e.g., the RTU 136) can be oriented in any position and / or orientation when in the planning mode. In some cases, the step 210 may be optional (e.g., the robotic arm does not move when in the planning mode).

[0093] The method 200 also comprises receiving sensor data while in the planning mode (step 212). The step 212 may be the same as or similar to the step 204. The sensor data may have a pattern of force(s) received that correspond to either moving the virtual cursor or executing a command.

[0094] The method 200 also comprises determining a virtual cursor direction and speed, or a command (step 216). In instances where the sensor data is used to move the virtual cursor, the sensor data may correspond to the force applied to the RTU and a duration of time that the force is applied to the RTU. In other instances where the sensor data is used to execute a command such as, for example, a select command, the sensor data may comprise a force received for a duration or time that is less than a predetermined duration of time. In other words, a quick click on the RTU may correspond to the virtual cursor selecting an object in the GUI whereas holding down one button of the one or more buttons may correspond to moving the mouse in a certain direction.

[0095] Depending on whether the sensor data corresponds to moving the virtual cursor or executing a command, the method 200 may proceed to step 220 for moving the virtual cursor or to step 224 for executing the command.

[0096] The method 200 also comprises causing the virtual cursor to move in the determined virtual cursor direction and speed (step 220). The virtual cursor may be moved in the determinedA0011933W001 virtual cursor direction for the duration of time. In embodiments where the user input is received via one or more buttons, each button may correspond to a different direction. In embodiments where the user input is received via a touchpad, the direction may be determined based on the user applying a force across the touchpad. The direction of the force applied across the touchpad may be used to determine the virtual cursor direction. The magnitude of the force exerted on the sensor (and specified in the sensor data) may define the speed at which the virtual cursor is moved. In one example, a greater magnitude of force results in a faster movement of the virtual cursor.

[0097] The method 200 also comprises causing the virtual cursor to execute the command (step 224). The virtual cursor may execute one or more commands such as, for example, selecting an object within the computer program. For example, the virtual cursor may be used to select a surgical step of a surgical plan, to change one or more properties or settings of the surgical step and / or the surgical plan, and / or the open different surgical plans.

[0098] The method 200 also comprises receiving sensor data (step 228). The step 228 may be the same as or similar to the step 204. The sensor data may have a pattern of force(s) received that corresponds to exiting the planning mode. Additionally or alternatively, the sensor data may comprise voice commands from the user to exit the planning mode (e.g., the user provides an input via a microphone or other audio device that is sent along with the sensor data to exit the planning mode).

[0099] In some cases, the step 228 may return to the step 212, where sensor data based on another pattern of force(s) is received. In other words, the steps 212, 216, 220, 224, and 228 may repeat one or more times until the sensor data received at the step 228 comprises a pattern of force(s) that correspond to exiting the planning mode.

[0100] The method 200 also comprises exiting the planning mode (step 232). The planning mode may be exited when the pattern of forces received corresponds to a command for exiting the planning mode. Additionally or alternatively, the planning mode may be exited when a voice command input by the user is received. In some embodiments, the processor may cause the RTU and the computing device to exit the planning mode. When not in the planning mode, control of the virtual cursor via the RTU may be disabled and the sensor of the RTU may be used for measuring, for example, forces and / or torque applied to the RTU and may be used to determine a pose of the RTU.A0011933W001

[0101] The pattern of forces received for entering and exiting the planning mode may be in some instances, the same pattern of forces. For example, a user may click a button or the touchpad twice to enter and exit the planning mode. In other embodiments, the pattern of forces received for entering and exiting the planning mode may be different. For example, the user may click the button or the touchpad 3 times to enter the planning mode and 4 times to exit the planning mode. Additionally or alternatively, the user may provide the same voice command to enter and exit the planning mode. For example, the user may say “plan” or other word or phrase to enter and exit planning mode. In other cases, the voice command to enter and exit the planning mode may be different. For instance, the user may say “enter” to enter the planning mode and “exit” to exit the planning mode.

[0102] The present disclosure encompasses embodiments of the method 200 that comprise more or fewer steps than those described above, and / or one or more steps that are different than the steps described above.

[0103] As noted above, the present disclosure encompasses methods with fewer than all of the steps identified Fig. 2 (and the corresponding description of the method 200 as well as methods that include additional steps beyond those identified in Fig. 2 (and the corresponding description of the method 200). The present disclosure also encompasses methods that comprise one or more steps from one method described herein, and one or more steps from another method described herein. Any correlation described herein may be or comprise a registration or any other correlation.

[0104] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, embodiments, and / or configurations for the purpose of streamlining the disclosure. The features of the aspects, embodiments, and / or configurations of the disclosure may be combined in alternate aspects, embodiments, and / or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, embodiment, and / or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the disclosure.A0011933W001

[0105] Moreover, though the foregoing has included description of one or more aspects, embodiments, and / or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, embodiments, and / or configurations to the extent permitted, including alternate, interchangeable and / or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and / or equivalent structures, functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

[0106] The following paragraphs provide non-limiting Examples for various embodiments that are disclosed herein.

[0107] Example 1 : A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140); at least one sensor (142) positioned on the robotic tracking unit (136), the at least one sensor (142) configured to sense a force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensor (142), the sensor data corresponding to the force applied to the robotic tracking unit (136) and a duration of time that the force is applied to the robotic tracking unit (136), determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit (136), and cause the virtual cursor (150) to move in the determined virtual cursor (150) direction for the duration of time.

[0108] Example 2: The system of example 1, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a first pattern of forces received by the sensor data, determine that at least one of the first pattern of forces and a voice command correspond to entering a planning mode, and cause the display (152) to enter the planning mode and enable control of the virtual cursor (150) via the robotic tracking unit (136).

[0109] Example 3: The system of any of example 1-2, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, furtherA0011933W001 cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a second pattern of forces received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display (152) to exit the planning mode and disable control of the virtual cursor (150) via the robotic tracking unit (136).

[0110] Example 4: The system of example 3, wherein the first pattern and the second pattern are the same.

[0111] Example 5: The system of example 3, wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

[0112] Example 6: The system of any of examples 1-5, wherein the robotic tracking unit (136) includes a joystick in communication with the at least one sensor (142), wherein the force is applied to the joystick.

[0113] Example 7: The system of any of examples 1-6, wherein the robotic tracking unit (136) includes at least one button in communication with the at least one sensor (142), wherein the force is applied to the at least one button.

[0114] Example 8: The system of any of example s 1-7, wherein the robotic tracking unit (136) includes a touchpad (156) in communication with the at least one sensor (142), wherein the force is applied to the touchpad (156).

[0115] Example 9: The system of any of examples 1-8, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).

[0116] Example 10: A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140); at least one sensor (142) positioned on the robotic tracking unit (136), the at least one sensor (142) configured to sense a force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor (150) movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensorA0011933W001(142) comprising a first pattern of forces received by the sensor data, determine that the first pattern of forces correspond to entering a planning mode, and cause the display (152) to enter the planning mode and enable control of the virtual cursor (150) via the robotic tracking unit (136).

[0117] Example 11: The system of example 10, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising at least one of a second pattern of forces and a voice command received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display (152) to exit the planning mode and disable control of the virtual cursor (150) via the robotic tracking unit (136).

[0118] Example 12: The system of example 11, wherein the first pattern and the second pattern are the same.

[0119] Example 13: The system of example 12, wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

[0120] Example 14: The system of any of examples 10-13, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).

[0121] Example 15: The system of any of examples 10-14, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142), the sensor data corresponding to the force applied to the robotic tracking unit (136) and a duration of time that the force is applied to the robotic tracking unit (136), determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit (136), and cause the virtual cursor (150) to move in the determined virtual cursor direction for the duration of time.

[0122] Example 16: A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140); at least one sensor (142) positioned on the robotic tracking unit (136), the at least one sensor (142) configured to sense aA0011933W001 force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor (150) movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).

[0123] Example 17: The system of example 16, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a first pattern of forces received by the sensor data, determine that the first pattern of forces correspond to entering a planning mode, and cause the display (152) to enter the planning mode and enable control of the virtual cursor (150) via the robotic tracking unit (136).

[0124] Example 18: The system of example 17, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a second pattern of forces received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display (152) to exit the planning mode and disable control of the virtual cursor (150) via the robotic tracking unit (136).

[0125] Example 19: The system of example 18, wherein the first pattern and the second pattern are the same.

[0126] Example 20: The system of any of examples 16-19, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142), the sensor data corresponding to the force applied to the robotic tracking unit (136) and a duration of time that the force is applied to the robotic tracking unit (136), determine at least one of a virtual cursor direction and a virtual cursor speed corresponding to the force applied to the robotic tracking unit (136), and cause the virtual cursor (150) to move in the determined virtual cursor direction for the duration of time.

[0127] Example 21: The system of example 2, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause theA0011933W001 at least one processor (104) to: cause, upon entering the planning mode, at least one of the robotic arm (116) and the robotic tracking unit (136) to move into a predetermined pose associated with the planning mode.

[0128] Example 22: The system of example 10, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: cause, upon entering the planning mode, at least one of the robotic arm (116) and the robotic tracking unit (136) to move into a predetermined pose associated with the planning mode.

[0129] Example 23: The system of example 17, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: cause, upon entering the planning mode, at least one of the robotic arm (116) and the robotic tracking unit (136) to move into a predetermined pose associated with the planning mode.

[0130] The foregoing is not intended to limit the disclosure to the form or forms disclosed herein. In the foregoing Detailed Description, for example, various features of the disclosure are grouped together in one or more aspects, embodiments, and / or configurations for the purpose of streamlining the disclosure. The features of the aspects, embodiments, and / or configurations of the disclosure may be combined in alternate aspects, embodiments, and / or configurations other than those discussed above. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed aspect, embodiment, and / or configuration. Thus, the following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate preferred embodiment of the disclosure.

[0131] Moreover, though the foregoing has included description of one or more aspects, embodiments, and / or configurations and certain variations and modifications, other variations, combinations, and modifications are within the scope of the disclosure, e.g., as may be within the skill and knowledge of those in the art, after understanding the present disclosure. It is intended to obtain rights which include alternative aspects, embodiments, and / or configurations to the extent permitted, including alternate, interchangeable and / or equivalent structures, functions, ranges or steps to those claimed, whether or not such alternate, interchangeable and / or equivalent structures,A0011933W001 functions, ranges or steps are disclosed herein, and without intending to publicly dedicate any patentable subject matter.

Claims

A0011933W001CLAIMSWhat is claimed is:

1. A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140); at least one sensor ( 142) positioned on the robotic tracking unit ( 136), the at least one sensor (142) configured to sense a force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensor (142), the sensor data corresponding to the force applied to the robotic tracking unit (136) and a duration of time that the force is applied to the robotic tracking unit (136), determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit (136), and cause the virtual cursor (150) to move in the determined virtual cursor (150) direction for the duration of time.

2. The system of claim 1, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a first pattern of forces received by the sensor data, determine that at least one of the first pattern of forces and a voice command correspond to entering a planning mode, and cause the display ( 152) to enter the planning mode and enable control of the virtual cursor (150) via the robotic tracking unit (136).A0011933W0013. The system of any of claims 1-2, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a second pattern of forces received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display ( 152) to exit the planning mode and disable control of the virtual cursor (150) via the robotic tracking unit (136).

4. The system of claim 3, wherein the first pattern and the second pattern are the same.

5. The system of claim 3, wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

6. The system of any of claims 1-5, wherein the robotic tracking unit (136) includes a joystick in communication with the at least one sensor (142), wherein the force is applied to the joystick.

7. The system of any of claims 1-6, wherein the robotic tracking unit (136) includes at least one button in communication with the at least one sensor (142), wherein the force is applied to the at least one button.

8. The system of any of claims 1-7, wherein the robotic tracking unit (136) includes a touchpad (156) in communication with the at least one sensor (142), wherein the force is applied to the touchpad (156).

9. The system of any of claims 1-8, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).

10. A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140);A0011933W001 at least one sensor ( 142) positioned on the robotic tracking unit ( 136), the at least one sensor (142) configured to sense a force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor (150) movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a first pattern of forces received by the sensor data, determine that the first pattern of forces correspond to entering a planning mode, and cause the display ( 152) to enter the planning mode and enable control of the virtual cursor (150) via the robotic tracking unit (136).

11. The system of claim 10, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising at least one of a second pattern of forces and a voice command received by the sensor data, determine that the second pattern of forces correspond to exiting the planning mode, and cause the display ( 152) to exit the planning mode and disable control of the virtual cursor (150) via the robotic tracking unit (136).

12. The system of claim 11, wherein the first pattern and the second pattern are the same, and wherein the first pattern and the second pattern comprise a force received for at least a duration of time.

13. The system of any of claims 10-12, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for a duration of time that is less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).A0011933W00114. The system of any of claims 10-13, wherein the memory (106) stores additional instructions for execution by the at least one processor (104) that, when executed, further cause the at least one processor (104) to: receive sensor data from the sensor (142), the sensor data corresponding to the force applied to the robotic tracking unit (136) and a duration of time that the force is applied to the robotic tracking unit (136), determine a virtual cursor direction corresponding to the force applied to the robotic tracking unit (136), and cause the virtual cursor (150) to move in the determined virtual cursor direction for the duration of time.

15. A system for controlling a virtual cursor (150), the system comprising: a robotic arm (116); a robotic tracking unit (136) attached to the robotic arm (116), the robotic tracking unit (136) configured to support and orient an end effector (140); at least one sensor ( 142) positioned on the robotic tracking unit ( 136), the at least one sensor (142) configured to sense a force applied to the robotic tracking unit (136); a display (152) configured to display a virtual cursor (150) movable within a Graphical User Interface (GUI) (154); at least one processor (104); and at least one memory (106) storing instructions for execution by the at least one processor (104) that, when executed, cause the at least one processor (104) to: receive sensor data from the sensor (142) comprising a force received for less than a predetermined duration of time, determine that the force corresponds to a selection command, and cause the virtual cursor (150) to select an object in the GUI (154).

Images