User input device for use in robotic surgery

Abstract

The present invention describes a user input device (UID) for controlling a surgical robotic system. The UID may include one or more tracking sensors to generate a corresponding spatial state signal based on the posture of the UID. At least one of the tracking sensors may be a camera. In the case of multiple tracking sensors, the spatial state signal is processed by a sensor fusion algorithm to generate a more robust single tracking signal and quality measurement. The digital control system then uses the tracking signal and the quality measurement to control the movement of the surgical robotic system actuator associated with the UID. Other embodiments are also described and claimed herein.

Description

[0001] Related Applications

[0002] This application claims the benefit of U.S. Patent Application No. 16 / 439,591, filed on June 12, 2019, which claims priority to U.S. Provisional Patent Application No. 62 / 685,821, filed on June 15, 2018, the entire contents of which are hereby incorporated by reference herein. Background Art Technical Field

[0004] Embodiments related to robotic systems are disclosed. More particularly, embodiments related to surgical robotic systems and corresponding user input devices are disclosed. Background of the Invention

[0006] Endoscopic surgery involves examining a patient's body and performing surgery inside the body using an endoscope and other surgical tools. For example, laparoscopic surgery may use a laparoscope to enter and view the abdominal cavity. Endoscopic surgery may be performed using manual tools and / or surgical robotic systems with robotic-assisted tools.

[0007] The surgical robotic system may be remotely operated by a surgeon to control a robotic-assisted tool located at the operating table. The surgeon may use a computer console located in the operating room to command the robot to manipulate the surgical tool mounted on the operating table, or the computer console may be located in a different city. The robotically controlled surgical tool may be a grasping forceps mounted on a robotic arm. Thus, the surgical robotic system may be controlled by a remote surgeon to grasp tissue during robotic surgery.

[0008] Control of the surgical robotic system may require control input from the surgeon. For example, the surgeon may hold in her hand a user input device (UID), such as a joystick or a computer mouse, which she manipulates to generate signals of control commands that control the movement of surgical robotic system components, such as actuators, robotic arms, and / or surgical tools of the robotic system. Summary of the invention

[0009] Existing UIDs that rely on a single tracking modality are spatially limited and prone to errors. Such errors can generate unexpected and potentially dangerous movements of the robotic manipulator. In the case of medical applications, movements within the sub-millimeter (for translation) and sub-degree (for orientation) ranges may be required to achieve clinically feasible operations. It should be noted that system noise that can cause control errors can be reduced by filtering the control signal from the UID. However, signal filtering can introduce time delays that have associated undesirable effects on the stable operation of the robotic manipulator. Therefore, noise-free, accurate, and real-time sensing methods are needed to detect the state, position, and orientation of the UID used to control the surgical robot system.

[0010] One aspect of the present disclosure is a UID for controlling a surgical robotic system based on a combination of several tracking modalities (in contrast to a UID that relies on a single tracking modality, which may be spatially limited and prone to errors). The tracking modalities may include visual modalities and inertial modalities to enable estimation of the pose of the UID. The visual / inertial odometry method combines i) an estimation of the pose based on the visual modality with ii) an estimation of the pose based on the inertial modality. More specifically, the estimate of the pose calculated using the imaging optical device is combined with measurements from tracking sensors such as an inertial measurement unit (IMU) and / or an electromagnet (EM) sensor to generate an accurate pose determination. The UID can achieve robust and fast (e.g., real-time) tracking over a wide (unrestricted) range of motion; tracking without EM noise or distortion; and tracking that can detect and compensate for objects or people in the surrounding environment.

[0011] The above summary of the invention does not include an exhaustive list of all aspects of the invention. It is contemplated that the present invention includes all systems and methods that can be implemented using all suitable combinations of the various aspects summarized above, as well as those disclosed in the following detailed description. Such combinations have specific advantages not explicitly described in the above summary of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Embodiments of the present invention are shown by way of example and not by way of limitation in the accompanying drawings, in which like reference numerals indicate similar elements. It should be noted that references to "one" embodiment of the present invention in this disclosure do not necessarily refer to the same embodiment, and that they represent at least one embodiment. In addition, for the purposes of simplicity and reducing the total number of drawings, a given drawing may be used to illustrate features of more than one embodiment of the present invention, and all elements in the drawings may not be required for a given embodiment.

[0013] Figure 1 is a diagram of an exemplary surgical robotic system in a surgical field, according to one embodiment.

[0014] Figure 2 is a perspective view of a user input device with an outward-looking camera according to one embodiment.

[0015] Figure 3A An exemplary arrangement of markers for vision or camera based tracking is shown.

[0016] Figure 3B Several other arrangements of indicia at a user console are shown.

[0017] Figure 4 is a perspective view of a printed circuit board for use in a user input device with a camera according to one embodiment.

[0018] Figure 5 is a side view of a user input device with a camera according to one embodiment.

[0019] Figure 6 is a perspective view of a user input device with several cameras according to one embodiment.

[0020] Figure 7 is a block diagram of a surgical robotic system according to one embodiment.

[0021] Figure 8 is a block diagram of the operation of a method of fusing spatial state signals representing a pose of a user input device to control motion of an actuator of a surgical robotic system, according to one embodiment. DETAILED DESCRIPTION

[0022] Embodiments of a user input device (UID) for controlling a robotic system, and more specifically a surgical robotic system, are described. However, the UID may be used to control other medical systems, such as an interventional cardiology system or a medical vision system, to name just a few possible applications.

[0023] In various embodiments, description is made with reference to the accompanying drawings. However, certain embodiments may be implemented without using one or more of these specific details or in combination with other known methods and structures. In the following description, many specific details, such as specific structures, sizes and processes, are set forth in order to provide a thorough understanding of the embodiments. In other cases, in order to avoid making this specification unnecessarily unclear, there is no particularly detailed description of well-known processes and manufacturing techniques. Reference to "one embodiment", "embodiment" etc. throughout the specification means that the specific features, structures, configurations or characteristics are included in at least one embodiment. Therefore, the phrases "one embodiment" or "embodiment" etc. that appear in various different places throughout the specification do not necessarily all refer to the same embodiment. In addition, specific features, structures, constructions or characteristics may be combined in one or more embodiments in any suitable manner.

[0024] The use of related terms throughout the specification may indicate relative positions or directions. For example, "distal" may indicate a first direction away from a reference point, such as a first direction away from an operator. Similarly, "proximal" may indicate a position in a second direction opposite to the first direction (e.g., toward the operator). Such terms are provided to establish a relative reference frame, but such terms are not intended to limit the use or orientation of the UID to the specific configurations described in the various embodiments below.

[0025] Figure 1 1 is a diagram of an exemplary surgical robotic system 100 located in a surgical field. The robotic system 100 includes a user console 120, a control tower 130, and one or more surgical robotic arms 112 located at a surgical robotic platform 111 (e.g., a table, a bed, etc.). The system 100 may incorporate any number of devices, tools, or accessories for performing a surgical procedure on a patient 102. For example, the system 100 may include one or more surgical tools 104 for performing a surgical procedure. The surgical tool 104 may be an end effector attached to a distal end of a surgical arm 112 for performing a surgical procedure.

[0026] Each surgical tool 104 may be manipulated manually, robotically, or both during a surgical procedure. For example, the surgical tool 104 may be a tool used to access, view, or manipulate the internal anatomy of the patient 102. In one embodiment, the surgical tool 104 is a grasping forceps that can grasp tissue of the patient 102. The surgical tool 104 may be manually controlled by an operator 106 at the bedside; or it may be robotically controlled via actuated movements of a surgical robotic arm 112 to which the surgical tool is attached. The robotic arm 112 is shown as a table mounted system, but in other configurations, the arm 112 may be mounted in a cart, ceiling or sidewall, or in another suitable structural support.

[0027] Generally, a remote operator 107, such as a surgeon or other operator, may use a user console 120 to remotely manipulate the arm 112 and / or surgical tool 104, for example, via teleoperation. Figure 1 , the user console 120 may be located in the same operating room as the rest of the system 100. However, in other environments, the user console 120 may be located in an adjacent or nearby room, or the user console may be in a remote location, such as in a different building, city, or country. The user console 120 may include a seat 122, foot-operated controls 124, one or more handheld user input devices (UIDs) 126, and at least one operator display 128, which is configured to display, for example, a view of a surgical site within the patient 102. In the exemplary user console 120, the remote operator 107 sits on the seat 122 and views the operator display 128 while manipulating the foot-operated controls 124 and the handheld UID 126 to remotely control the arm 112 and the surgical tool 104 (which are mounted on the distal end of the arm 112). The foot-operated controls 124 may be foot pedals, such as seven pedals, which generate motion control signals when actuated. The user console 120 may include one or more additional input devices, such as a keyboard or joystick, to receive manual input to control the operation of the user console 120 or the surgical robotic system 100 .

[0028] In some variations, the bedside operator 106 can also operate the system 100 in an "on-bed" mode, where the bedside operator 106 is now at the side of the patient 102 and simultaneously manipulates the robotically driven tools (the end effector attached to the arm 112) and the manual laparoscopic tools, for example, while holding the handheld UID 126 in one hand. For example, the bedside operator's left hand can manipulate the handheld UID 126 to control the robotic components while the bedside operator's right hand can manipulate the manual laparoscopic tools. Thus, in these variations, the bedside operator 106 can perform both robotic-assisted minimally invasive surgery and manual laparoscopic surgery on the patient 102.

[0029] During an exemplary operation (surgical procedure), the patient 102 is prepared for surgery and covered with a surgical sheet in a sterile manner, and anesthesia is administered. Initial access to the patient's anatomy can be achieved using known techniques such as by forming an incision in the skin. A trocar and / or other surgical tools can be inserted into the incision through an optical portal on the patient. The trocar can then be positioned at the surgical site. Initial access to the surgical site can be performed manually when the arm of the robotic system 100 is in a stowed or retracted configuration (to facilitate access to the surgical site) or in an operator-defined parking posture. Once the initial access is completed, the initial positioning or preparation of the robotic system (including its arm 112) can be performed. Next, the surgical procedure continues to the remote operator 107 at the user console 120 using the foot-operated controls 124 and UID126 to manipulate various end effectors and possible imaging systems to perform the surgical procedure. Bedside personnel wearing sterile surgical gowns may also provide manual assistance at the operating bed or table, for example, a bedside operator 106 who may perform tasks such as retracting tissue may perform manual repositioning of one or more of the robotic arms 112 and tool exchange. Non-sterile personnel may also be present to assist the remote operator 107 at the user console 120. When the procedure or surgical procedure is completed, the system 100 and / or the user console 120 may be configured or set to a state that facilitates post-operative procedures, such as cleaning or sterilization and health care record entry or printout via the user console 120.

[0030] In one embodiment, the teleoperator 107 holds the UID 126 and moves the UID to provide input commands to move the robotic arm actuator 114 in the robotic system 100. The UID 126 may be communicatively coupled to the rest of the robotic system 100, for example, via the console computer system 110. The UID 126 may generate a spatial state signal corresponding to the movement of the UID 126, such as the position and orientation of the handheld housing of the UID, and the spatial state signal may be an input signal to control the movement of the robotic arm actuator 114. The robotic system 100 may generate a control signal that varies with the spatial state signal to control the proportional movement of the actuator 114. In one embodiment, the console processor of the console computer system 110 receives the spatial state signal and generates a corresponding control signal. Based on these control signals that control how to stimulate the actuator 114 to move a section or link of the arm 112, the movement of the corresponding surgical tool, including the end effector attached to the arm, may mimic the movement of the UID 126. Similarly, interaction between teleoperator 107 and UID 126 may generate, for example, a grasp control signal that causes the jaws of a grasper of a surgical tool to close and grasp tissue of patient 102 .

[0031] The sensed motion of the UID 126 may alternatively be provided to control other aspects of the surgical robotic system 100. For example, a gesture detected by a finger clutch may generate a clutch signal to pause the motion of the actuator 114 and the corresponding surgical tool 104. For example, when an operator touches a finger to a finger clutch of the UID 126, the finger clutch may generate a clutch signal, and the clutch signal may be an input signal to pause the motion of the actuator 114. Similarly, one or more capacitive sensing pads may be located on the UID 126, and the operator may touch the capacitive sensing pads while performing a diagnostic, surgical, laparoscopic or minimally invasive surgical procedure or another robotic procedure to control an endoscope's camera view, a cursor on a display of the user console 120, and the like.

[0032] The surgical robotic system 100 may include several UIDs 126, wherein respective control signals are generated for each UID controlling an actuator and a surgical tool (end effector) of a respective arm 112. For example, the teleoperator 107 may move a first UID 126 to control the motion of an actuator 114 located in a left robotic arm, wherein the actuator responds by moving a link, gear, etc. in the arm 112. Similarly, the teleoperator 107 moves a second UID 126 to control the motion of another actuator 114, which in turn moves other links, gears, etc. of the robotic system 100. The robotic system 100 may include a right arm 112 secured to a bed or table on the right side of a patient, and a left arm 112 on the left side of the patient. The actuator 114 may include one or more motors that are controlled so that they drive the rotation of the joints of the arm 112 to, for example, change the orientation of an endoscope or a grasper of a surgical tool attached to the arm relative to the patient. The motion of several actuators 114 in the same arm 112 may be controlled by a spatial state signal generated from a particular UID 126. The UID 126 may also control the motion of a corresponding surgical tool grasper. For example, each UID 126 may generate a corresponding grasping signal to control the motion of an actuator, such as a linear actuator that opens or closes the jaws of a grasper at the distal end of a surgical tool to grasp tissue within the patient 102.

[0033] In some aspects, the communication between the platform 111 and the user console 120 can be through the control tower 130, which can translate the operator commands received from the user console 120 (and more specifically, from the console computer system 110) into robot control commands, which are transmitted to the arm 112 on the robotic platform 111. The control tower 130 can also transmit the status and feedback from the platform 111 back to the user console 120. The communication connection between the robotic platform 111, the user console 120 and the control tower 130 can be carried out via wired and / or wireless links using any suitable data communication protocol in a variety of data communication protocols. Any wired connection can be optionally built into the floor and / or wall or ceiling of the operating room. The robotic system 100 can provide video output to one or more displays, including displays in the operating room and remote displays accessible via the Internet or other networks. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0034] Will understand, Figure 1 The operating room scenes in are illustrative and may not accurately represent specific medical practices.

[0035] UID of the camera with visual / inertial odometry

[0036] For the purpose of teleoperation, a UID is used to control a robotic system to provide accurate sensing of the operator's intent and to initiate stable and robust motion of the robotic manipulator. Existing UIDs for controlling surgical robotic systems are based on one of several tracking modalities. These modalities include: a mechanical tracking system that senses the movement of the UID's linkage and outputs a control signal to an actuator based on the movement; an electromagnetic (EM) tracking system that senses the movement of the UID in EM space and outputs a control signal to an actuator based on the movement; and an optical tracking system that includes a camera located outside the UID to view the movement of a marker on the UID (and outputs a control signal to the actuator based on the movement). Each of these modalities is associated with defects, such as restrictions on the free movement of the operator (holding the UID) due to the mechanical linkage, errors in control signals caused by inherent system noise or drift, ferromagnetic interference or distortion of the EM field caused by nearby objects (e.g., a cell phone), or errors in control signals caused by nearby objects that block the external camera's view of the UID marker.

[0037] In one aspect, a UID for controlling a surgical robotic system is based on a combination of several tracking modalities (in contrast to a UID that relies on a single tracking modality, which may be spatially limited and prone to errors). The tracking modalities may include a visual modality ("seen" by means of digital image capture through imaging optics) and an inertial modality to achieve an estimate of the posture of the UID. The visual / inertial odometry method combines i) an estimate of the posture based on the visual modality with ii) an estimate of the posture based on the inertial modality. More specifically, the visual modality refers to an estimate of the posture that has been calculated by analyzing a digital image generated by a "looking outward" camera (i.e., the imaging optics of the camera are facing forward at the distal end of the UID 26 to view the markers). The imaging optics and image sensors can be configured so that the digital image captures the scene in front of the camera in the infrared spectrum (i.e., not necessarily in the human visible spectrum). These are effectively combined with measurements from tracking sensors such as an inertial measurement unit (IMU) and / or an EM sensor to generate a single tracking signal as an accurate posture determination. As described below, the UID provides: robust and fast (eg, real-time) tracking over a wide (unconstrained) range of motion; tracking without EM noise or distortion; and tracking that can detect and compensate for objects or people in the surrounding environment.

[0038] Reference Figure 2, shows a perspective view of a UID 126 having a camera 3 with a lens 6 as an imaging optic, according to one embodiment. As further described below, the UID 126 has components for visual / inertial odometry that allow its operator to interact with a surgical robotic system more reliably than conventional UIDs. The UID may control a robotic instrument, other robotic components, or a graphical user interface (GUI) on a display of the surgical robotic system. In one embodiment, the UID includes a device body 2 that extends in a longitudinal direction along a central axis 4. For example, the device body may extend longitudinally from a proximal end 5 to a distal end 7 having a forward-facing surface, the proximal end generally cupped in the hand of an operator (e.g., a remote operator 107) when the operator holds the UID.

[0039] The UID 126 may include a gripping structure to be grasped by an operator. For example, the UID may include a number of gripping links 8 extending outward from a centrally located device body 2. The operator may grasp portions of the gripping links between a number of fingers while being able to freely move the UID 126 as a whole within a workspace. The workspace may be the reach of the operator's arms. The UID may not be constrained by mechanical links that limit the size of the workspace (also referred to herein as a rootless UID).

[0040] In one embodiment, the gripping links are mounted on the device body about the central axis 4. For example, the gripping links may be symmetrically distributed about the central axis 4. Each gripping link may include a gripping crank that is pivotally coupled to the device body. For example, the gripping crank may be hinged to the device body at a proximal end and extend distally from the proximal end to the distal top end of the gripping link. Therefore, each gripping crank may be cantilevered from the device body. An operator may squeeze the cantilevered crank to pivot the crank toward the device body, which in response causes a sensor circuit within the device body 2 to generate a signal indicating the angle of the squeezed crank, thereby controlling an end effector on an associated robotic arm, for example, to generate a contraction motion of the end effector.

[0041] In one embodiment, the UID 126 also has a visual tracking modality based on data generated by the camera. The camera can be mounted on the device body 2 or integrated into the housing of the device body 2. For example, the camera can be mounted inside the housing of the device body 2, mounted at the distal end 7 of the device body along the central axis 4, as shown, or mounted at any position that enables the camera to view in a direction away from the gripping link of the central axis 4 (e.g., directing the camera's field of view forward from the forward-facing surface of the distal end 7, such as Figure 2. The camera may include several subcomponents, including a lens (as part of the imaging optics) and an associated image sensor, which may be part of a camera module (not shown in detail). The UID may also include a support structure, such as a lens support 9 mounted on the device body, which holds the lens, for example, as part of the camera module. The central axis 4 of the UID may intersect the lens, for example, coinciding with the central imaging axis of the camera. The lens may be offset from the central axis 4 and may be tilted to capture images off-axis relative to the central axis.

[0042] The camera provides visual data that is interpreted by a digital processor to determine the posture of the UID. By definition, the posture of the UID is the position and orientation of the UID in the workspace. For example, the position may include 3 degrees of freedom along an orthogonal axis, and the orientation may include 3 degrees of freedom around the axis. More specifically, the posture of the UID has a total of 6 degrees of freedom.

[0043] The camera can generate information about the posture of the UID by viewing optical markers at known or determinable positions within the workspace or surrounding environment (outside the UID). For example, the operator can hold the UID while sitting in the seat 122 of the user console and watching the real-time scene of the surgical operation on the operator display 128. The camera can point to the display when the operator manipulates the UID to simulate and control the movement of surgical tools located at the displayed surgical site. In one embodiment, a marker (such as a collection of bright spots) is located around the display so that the marker is viewed by the camera. The marker can be reliably detected based on a predetermined or known size and shape of the marker, such as based on a stored data structure, which describes the absolute position of the marker and the relative position of each object in a certain number of objects constituting the marker, for example, which relative position remains fixed during remote control operation. Assuming that the absolute position of the marker and the size and shape of the marker are known, such as at a pre-defined position on the user console, the posture of the camera can be accurately determined by detecting the object in the digital image generated by the camera (using pattern recognition or machine learning algorithms) and interpreting the object as a marker. For example, by detecting a marker in successive images captured by a camera's image sensor and then determining the marker's movement (e.g., changes in the shape of the marker captured from different camera positions or angles or changes in the marker's position relative to the camera), the movement and / or position of the camera (and therefore the posture of the UID) can be accurately determined.

[0044] In addition to tracking static markers (markers that are fixed relative to the UID), the UID may be able to track one or more moving markers around the workspace and surrounding environment. In one embodiment, the operator may hold two UIDs (one in each hand) to control several tools of the surgical robot system. The first camera on the first UID can view the markers located on the second UID. More specifically, the first UID can enable the camera to track the relative position between the first UID and the second UID, and the camera (and the tracking sensor) can also track the relative position of the first UID relative to the static markers on the user console. By determining the position of both the static marker of the first UID relative to the user console and the dynamic marker of the second UID (the UID marker is dynamic when the UID moves relative to each other), the position of the second UID relative to the static marker of the user console can be determined in association. In other words, the UIDs can be referenced to each other together with one of the UIDs being referenced to an external benchmark to determine the relative movement of the UIDs and the movement of the UIDs within the reference system.

[0045] In one embodiment, only one of the two UIDs (to be held in the hands of the remote operator 107) has a camera. Alternatively, both UIDs have cameras that can detect markers on the other UID and / or the user console. Position detection based on visual data provided by the camera can be supplemented by position determination based on sensor data provided by tracking sensors, such as an inertial measurement unit IMU, or an EM sensor of one of the UIDs. For example, by processing image data from the camera and sensor data from an IMU tracker or an EM tracker, a more reliable estimate of the position and orientation of the two UIDs within the reference frame can be determined.

[0046] Regarding visual modality estimation (camera-based position and / or orientation detection techniques), these may use one or more of the following features. In one instance, the marker 10 is located on the user console 120 (e.g., with reference to Figure 1 as well as Figure 3A and Figure 3B , among which Figure 3A, the marker 10 is a collection of eight light sources located on a generally inverted Y-shaped table of the operator display 128). The marker 10 may emit light in a specific band (marker wavelength), for example centered at 850nm, or centered at 940nm (infrared). The camera may be configured with a bandpass filter to observe only the marker wavelength. The camera may be configured to see (image) in the visible range and in, for example, the infrared range (e.g., marker wavelength), which allows imaging and tracking of natural features as well as marker tracking. Alternatively, an infrared pass filter that also blocks the visible spectrum may be added to the imaging path in front of the image sensor, resulting in the image sensor producing an image with a substantially black background and on which the constituent light-emitting structures of the marker 10 have been captured with high contrast. This solution is also independent of visible light conditions and any changes in visible lighting at the user console 120.

[0047] The marker may include multiple light emitting structures, such as lines, two-dimensional areas, or dots. The marker may have redundant features (e.g., more dots than required) to allow the tracking system to tolerate partial occlusion of the marker. The layout of the marker may be a predefined shape based on a pseudo-random distribution of features. Figure 3A An inverted Y-shaped structure (e.g., a work surface or support for an operator display 128) is shown between foot-operated controls 124 (e.g., foot pedals). In that example, eight points are shown arranged in a Y pattern, with each point extending from the front of the generally Y-shaped structure toward a seat 122 (also referred to as Figure 1 ) emits light. Figure 3B , which includes the following composition of light segments located at different locations: T7 above the operator display 128; T6 below the operator display 128, which has a left segment T6L and a spaced right segment T6R; center segments T1 and T2 on the vertical line on the inverted Y-shaped workbench, and left segment T3L and right segment T3R on the corresponding legs of the inverted Y; and left segment T5L and spaced right segment T5R on the bottom floor of the user console 120. It should be noted that the marker 10 does not need to have all of these segments in its composition; one or more of the segments shown can be omitted, but the composition as a whole can still serve as a reliable marker with sufficient redundancy to tolerate partial occlusion (of some segments). In addition, there can also be left and right marker segments (both not shown) on the operator's shoulder and on the ceiling segment above the operator 107. The light emitting structure (as part of the self-emissive marking) can be a light emitting diode (LED) positioned behind a material that is transparent to a specific marking light emission wavelength (eg, infrared). In one embodiment, the LED is positioned behind a continuous sheet of material that is transparent to infrared light.

[0048] Referring now to the camera 3, in one embodiment, the lens 6 is a wide angle lens, for example, having a field of view of at least 90 degrees. The field of view may affect the tracking efficacy of the UID. For example, a smaller field of view may reduce the tracking quality, and a larger field of view may limit the angular resolution of the camera and reduce the tracking accuracy at larger distances. Thus, the field of view may be in the range of 110 degrees to 180 degrees, for example 170 degrees.

[0049] The lens 6 can direct visible light and invisible light from a scene (eg, the scene includes visible light and / or invisible light emitted by the tag 10) to the image sensor 12 of the camera module (see Figure 4 ). In one embodiment, the image sensor is a CMOS image sensor having about 8.08M effective pixels. However, more generally, the image data captured by the image sensor can be full high definition (HD), but the image data can also be at a lower resolution or a higher resolution. As an example, the resolution of the camera can be 1080p (1920×1080 pixels) to achieve an RMS error of 0.5 mm and 0.05 degrees in determining and tracking the position of the UID at a distance of 60 cm from the nearest object.

[0050] In addition to the lens and camera module capabilities, additional settings that may affect tracking accuracy include the camera's latency, update rate, and shutter settings. In one embodiment, accurate tracking is based on the latency between the motion occurrences of events, such as the latency between an operator moving the UID in the workspace to applying or publishing the tracked posture when a tool attached to a robot arm moves in the same direction as the UID. For example, the camera module can capture images at a rate of 120 frames per second to meet and exceed the target latency. In one embodiment, the camera module includes a shutter that does not introduce motion artifacts. It has been determined that a rolling shutter can introduce motion artifacts. Therefore, the camera may have a fast (adjustable) or global shutter to avoid motion artifacts.

[0051] Reference Figure 5 , shows a side view of a UID 126 with a camera according to one embodiment, wherein there are one or more additional sensors in the device body 2, such as a touch sensor for detecting touch input, such as a clutch input that signals the system to tighten or pause remote control operation. The additional sensors may be located on an outer surface of the device body, or may be housed within the device body. For example, the device body may include a front cover 13 attached to a rear housing 15. The space between the front cover and the rear housing may accommodate components, such as Figure 4 The camera module and additional tracking components. The tracking components can also be accommodated in a tube 16 extending backward from the rear shell to the proximal end 5.

[0052] The UID 126 may include at least one capacitive sensing pad 18 located on the outer surface of each grip link 8 as shown. At least one grip link may have a plurality of capacitive sensing pads on its outer surface that are exposed to the touch of a user's finger. For example, the grip link may include a first capacitive sensing pad, a second capacitive sensing pad, and / or a third capacitive sensing pad sequentially mounted or arranged on the outer surface of the grip link. For example, the first capacitive sensing pad may be distal to the second capacitive sensing pad on the outer surface, and the second capacitive sensing pad may be distal to the third capacitive sensing pad, such as Figure 5 as shown in .

[0053] In one embodiment, the output signal of the linear array of grip link capacitive sensing pads may be monitored by the processor of the UID 126 to detect a swipe gesture by an operator of the UID. The operator may input a swipe gesture by swiping their finger over the outer surface of the grip link. The swipe may cause a sequence of changes in the corresponding capacitances of the first capacitive sensing pad, the second capacitive sensing pad, and / or the third capacitive sensing pad. The UID processor may detect the sequence of changes as a swipe gesture on the pad array. The swipe gesture may be used to command various outputs. For example, the swipe gesture may trigger a control signal that causes the associated robotic arm to perform a predetermined operation. Alternatively, the swipe gesture may command some elements of a graphical user interface (GUI) of a user console. For example, an operator may swipe the outer surface of the grip link as a control input for navigating a menu, scrolling a displayed view, zooming in and out of a displayed image, or controlling other aspects of the GUI.

[0054] In one embodiment, one or more tracking sensors are installed in the device body 2 of the UID 126, for example, in a volume defined inside the front cover 13 or inside the rear shell 15 or inside the tube 16. The tracking sensor is configured to generate a spatial state signal. The spatial state signal can be measured or sensed data related to any aspect of the spatial state of the UID, including sensed data that changes in real time according to the posture of the UID, such as in response to the movement of the device body 2 or in response to the device body 2 in the UID workspace at the user console 120. For example, the one or more tracking sensors may include an IMU sensor or an EM sensor. The IMU sensor and / or the EM sensor may each be capable of measuring the posture of the UID in one or more degrees of freedom.

[0055] In one embodiment, the surgical robot system has an inertial tracking mode for the UID, in which the processor interprets one or more spatial state signals as sensor data generated by the IMU to track the position and / or orientation of the UID. The IMU can measure the acceleration of the UID. The IMU can use at least 3 degrees of freedom to measure the acceleration. For example, the IMU of the UID may include one or more accelerometers and one or more gyroscopes. The accelerometer can measure the inertial acceleration of the UID along one or more axes, and the processor can interpret the inertial acceleration to determine the change in the position of the UID, such as translation along the X-axis, Y-axis, or Z-axis. The gyroscope can measure the angular acceleration or rotation of the UID around one or more axes, such as yaw, pitch, and roll. More generally, the IMU provides various spatial state signals (sensor data) that describe the relative posture of the UID in one or more degrees of freedom, such as accelerometer measurements and / or gyroscope measurements.

[0056] In one embodiment, the surgical robot system has an EM tracking mode, in which the processor interprets data generated by the EM sensor located in the device body 2 to track both the position and orientation of the UID. The EM sensor can be part of an EM tracking system, which also includes an EM field generator (not shown) located near the UID workspace (e.g., located on the user console 120). The EM field generator includes one or more EM transmitters that generate an electromagnetic field, and the UID 126 is held and manipulated by the remote operator 107 (while the remote operator is sitting on the seat 122) within the electromagnetic field. When the UID moves, an interaction occurs between the EM sensor and the electromagnetic field. This interaction is measured or sensed as, for example, an electrical signal from a sensor located in the UID. The measured signal is then interpreted to describe the posture of the UID using 6 degrees of freedom. The EM tracking system uses 6 degrees of freedom to determine the posture of the UID, such as 3 degrees of freedom describing the translation along the axis, and 3 degrees of freedom describing the rotation around the axis.

[0057] The UID 126 can transmit data from its camera, IMU, or EM sensor via one or more wired or wireless data connections. In one embodiment, the transmission is via a wired data connection, such as a cable 20 that is communicatively connected to one or more processors of the UID. Figure 6 ), the distal end or the side of the cable 20 of the UID. For example, the cable can be from Figure 5 The rear housing 15 shown in FIG. 1 exits laterally or from the volume between the front cover and the rear housing, or the cable may be routed along the central axis ( Figure 6 ) exit outward from the proximal end 5.

[0058] In one embodiment, the cable transmits data from one or more of the UID sensors. At 1080p resolution, the data throughput of the camera module can be up to 2 Gbits / sec (1920×1080 pixels×8-bit grayscale×120 frames / s=1.99 Gbits / sec). The cable can be a USB cable for transmitting visual data generated by the camera module. However, the maximum signaling rate of the USB 2.0 version is limited to 480 Mbits / sec. Therefore, the cable can be a USB cable of USB 3.0 version or higher (USB 3.0 is limited to 5 Gbits / sec and USB 3.1 is limited to 10 Gbits / sec).

[0059] Reference Figure 6 , shows a perspective view of another example of a UID 126, wherein the UID 126 in this case has several cameras, each camera having a corresponding imaging lens (three imaging lenses in this example, as shown). The lenses can be mounted on the front cover 13 of the UID and tilted relative to each other so that their cameras have three viewing directions (DOV) relative to the central axis 4. For example, the cameras can be symmetrically distributed around the central axis and pointed in directions that are at a non-zero angle to the central axis. Each lens can be a wide-angle lens, such as a lens having a field of view of at least 90°. The lenses can be combined to have a combined field of view of 360° or more.

[0060] The combined field of view of several cameras can provide a field of view that allows the UID to detect the mark in a wider range of UID orientations within the workspace (compared to a single camera). This means that even when the operator has moved the UID to a position or posture in front of the UID that is not facing the operator display 128 (for example, when the UID is used in a continuously misaligned manner), the tracking of the mark can be maintained. In addition, if the cable 20 extends from the front end of the UID (not shown) and hangs in front of one of the cameras in the camera, the portion of the image data (for posture estimation) containing the cable can be not processed to avoid image artifacts. That is because the image data from another camera module can be used to compensate for the omitted image. Similar compensation can be performed when the operator accidentally blocks one of the cameras in the camera (for example, moving their fingers in front of the lens of that camera). Since the operator's fingers are unlikely to be large enough to completely block the field of view of each camera at the same time, one or more of the marks in the workspace of the remote operator 107 can always remain visible to one or more of the cameras in the camera. Therefore, the redundancy of the images from several cameras ensures that one or more of the marks can be detected in order to accurately determine the UID posture.

[0061] As an alternative to having many cameras to extend the field of view, a single camera may be used in combination with an actuated and encoded tilt mechanism. The tilt mechanism may be controlled by a processor to automatically adjust the orientation of the camera (relative to a fixed "base" of the UID 126, e.g., a predetermined and fixed plane passing through the UID 126) so that the tracked marker is always maintained within the field of view of the camera. In this embodiment, in order to locate the UID in space (estimate the pose of the UID), the marker pose detected in the camera frame (the digital image produced by the camera) needs to be multiplied by the transformation between the camera frame and the UID base (which transformation may be defined by the position encoder output of the tilt mechanism).

[0062] Reference Figure 4 , shows a perspective view of a printed circuit board (PCB) for use in a UID 126 according to one embodiment. Sensors and an electronic signal processing unit are mounted to the PCB, which prepares the sensor signals for transmission to an external processor (e.g., part of the console computer 110). In one embodiment, the PCB includes a substrate 24 on which the electronic components are mounted. The substrate 24 can be shaped to be mounted between the front cover 13 and the rear housing 15 (e.g., see FIG. 1 ). Figure 5 ). One or more of a camera module, a lens, an IMU, or a microprocessor may be mounted on the substrate. Other components (such as EM sensors) may be mounted elsewhere within the device body 2 and wired to the components mounted on the substrate.

[0063] In one embodiment, the one or more processors mounted on the substrate 24 include a UID controller 26. The controller 26 can be communicatively coupled to the image sensor 12 and the IMU (not shown). For example, the controller can have a digital camera communication interface and general input-output pins to connect the controller to the image sensor and the sensors of the IMU. Thus, the controller 26 can simultaneously acquire visual data generated by the image sensor and inertial data generated by the IMU.

[0064] The controller 26 may include a data output connection to interface with other electronic components of the surgical robotic system. For example, the controller may output a data stream containing data from one or more of a camera, an IMU, or an EM sensor to the output connection. In one embodiment, the output connection is a universal serial bus (USB) output configured to connect to the cable 20. Thus, the controller 26 can push data to another processor of the surgical robotic system, such as a processor of the console computer 110, via a USB 3.0 connection.

[0065] Reference Figure 7, a block diagram of a surgical robot system according to one embodiment is shown. Cable 20 transmits image data and metadata such as position or orientation data from an IMIU and / or EM tracking sensor from UID 126 to console computer 110. In one embodiment, cable 20 includes several wired connections running in parallel. For example, a tracking data line carrying raw image data from a camera and raw sensor data from a tracking sensor of UID 126 may run in parallel with a grip data line carrying raw grip data indicating the grip state or clutch state of the UID (e.g., data from a capacitive sensor or a touch sensor or a mechanical switch sensor). The grip state may be defined by the sensed position of the grip link (e.g., when the operator squeezes or relaxes the grip link), a touch input detected by a capacitive sensor on the UID, or any other sensor data associated with a non-tracking input made by the operator. In some cases, the UID controller 26 may filter the raw image data, raw tracking sensor data, and / or raw grip data before sending it on cable 20.

[0066] The tracking data line may carry the tracking data to the sensor fusion module 28, which is executed by a processor of the console computer 110. The tracking data may include EM tracking data from the EM sensor, image data from the camera, and / or IMU data from the IMU sensor. In one embodiment, the EM tracking data is input to the EM tracking system 29 of the sensor fusion module, and the image data and IMU data are input to the visual-inertial odometry module 30 of the sensor fusion module 28. The tracking data may include image data with, for example, 120 frames per second at 1080p grayscale and IMU data sampled at 400Hz. Existing UIDs typically include a single cable to transmit tracking data to the robotic system, and the data is aggregated, collected, processed, integrated, sent, and deserialized sequentially. However, image data may consume a large amount of bandwidth, and therefore, continuous transmission of image data and IMU data may be impractical. Therefore, the UID controller 26 (see Figure 4 ) can simultaneously acquire, integrate, and time stamp image data and IMU data. The time stamped data including image data and IMU data can then be sent to the visual / inertial odometry module 30. The sensor fusion module 28 including the visual / inertial odometry module 30 can then output a tracking signal as a signal defining the UID attitude to the console controller 31 (through one or more processors of the console computer 110 executing software). The tracking module 32 of the console controller 31 can store and process the tracking signal.

[0067] Fusing sensor data from multiple sensors (which produce tracking signals) can enable a more accurate estimate of the UID pose (position and orientation). Additional details of sensor fusion are described below.

[0068] Still refer to Figure 8 , the grip data line can carry the grip state data to the console controller 31 of the console computer. In one embodiment, the grip module 34 of the console controller 31 stores and processes the grip state data. The console controller may include other modules for receiving inputs from user console components. For example, the seat control module 35 can store and process input data from the seat control. The seat control input can be used to output an actuator signal that causes the seat 122 to adjust for the operator 107. Similarly, the pedal control 37 can store and process input data from the foot pedal. The pedal input can be used to output signals that cause various changes in the system, such as selecting an element in a graphical user interface, starting or stopping the movement of a surgical tool, etc.

[0069] The console controller 31 may process inputs received from the UID and other components of the user console 120 to generate one or more output signals, which are transmitted to the surgical robotic system via a communication link. The output signal may be a control signal for controlling the movement of the arms and attached tools of the surgical robotic system. For example, at least one processor may be located in the control tower 130 and may be communicatively coupled to system components such as the surgical robotic platform 111 or one or more displays. The arms and / or tool actuators of the surgical robotic system may receive control commands from this processor to generate motions corresponding to the movement of the UID 126. More specifically, the control signal may cause the actuator to move the arms and / or tools of the surgical robotic system in coordination with changes in the posture of the UID.

[0070] Sensor Fusion

[0071] A UID having a camera and one or more tracking sensors as described above may generate several tracking data streams that may be used to complement each other. For example, IMU sensors and EM sensors may be associated with inherent noise or drift, which may result in tracking errors when only IMU sensor data or only EM sensor data is used to track the posture of the UID. In contrast, a camera implements optical tracking of position and orientation, which is free of the noise and drift that affect other tracking sensors. Therefore, optical tracking using a camera may act as a stabilizer to correct errors in the data from the tracking sensors. More specifically, data from the camera and data from one or more of the EM sensors or the IMU sensors may be combined to generate a tracking signal that is fed to the tracking module 32 ( Figure 7 ) for controlling surgical robotic systems.

[0072] On the one hand, two or more of the visual information from the camera, the EM tracking information from the EM sensor, and the inertial measurement information from the IMU sensor are fused in the sensor fusion module 28 to achieve a stable and reliable estimate of the UID attitude. The UID can obtain data from each UID sensor (e.g., camera, EM sensor, and IMU sensor) and simultaneously transmit the data to the sensor fusion module 28, which fuses information from the visual / inertial ranging module 30 and the EM tracking system 29. The visual / inertial ranging module 30 can fuse information from the camera and one or more IMU sensors. The sensor fusion module 28 can combine data used to correct errors and noise in the data of one tracking modality with data from another tracking modality. Therefore, the estimation of the UID attitude can be more robust, for example, less sensitive to the surrounding environment or noise, and the jitter / noise inherent in any tracking modality data stream in the tracking modality data stream can be reduced. Furthermore, unlike conventional filtering of EM-based tracking signals that reduces noise in exchange for introducing latency (a measure of time between one event measured by a sensor and the next possible processing and assignment of a timestamp to a sensor measurement), predictive models in sensor fusion modules 28, such as EM tracking system 29 and / or visual / inertial odometry module 30, may allow tracking signals to be estimated in real time.

[0073] Reference Figure 8 , a block diagram of the operation of a method for fusing spatial state signals from various sensors to control the movement of a surgical robot arm actuator 114 according to one embodiment is shown. The UID 126 may generate several spatial state signals, also referred to as input posture signals, based on the posture of the UID, for example, in response to the movement of the device body 2. The camera of the UID 126 may generate image data that captures markers in the field of view of the camera in the UID. The IMU may generate IMU measurements that are streamed as IMU tracking data corresponding to translational motion along several axes and rotational motion around the axes. Similarly, the EM sensor may generate EM measurements that are streamed as EM tracking data corresponding to movement of the EM sensor within several degrees of freedom. In Figure 8These image data, IMU data, and EM data are shown in , indicating that the UID in this example includes each type of tracking sensor. However, it will be appreciated that the UID need not have all of those sensors, but instead may include only two of those three sensor types. In any case, the data streams from the different sensors are processed by the sensor fusion module 28 to generate a tracking signal (which describes the UID pose) that is more robust and reliable than a tracking signal that could be achieved using only one of the sensor data streams alone.

[0074] In one embodiment, the data streams generated by each UID sensor are collected and aggregated by the processor of the UID controller 26. Aggregation of the data may include the use of a buffer that receives and outputs all data streams simultaneously. For example, image data may be sent to the buffer, and while the image data in the buffer is being transmitted, additional tracking data from the IMU and / or EM sensor may be added to the transmission buffer. This process of buffering all tracking data simultaneously may be contrasted with conventional techniques, because typically image data will be received, buffered, and sent sequentially with other data rather than in parallel. However, buffering all data streams in parallel allows for the high bandwidth and low latency transmission described above. More specifically, the continuous aggregation of image data interleaved with metadata describing the UID attitude may increase transmission bandwidth and reduce transmission latency compared to the process of sequentially sending image data as an entire packet followed by IMU data followed by EM data.

[0075] Each UID sensor in the UID sensor can detect a specific UID posture characteristic relatively better. For example, EM sensors and cameras can provide more accurate position data than IMU sensors. In comparison, IMU sensors can provide better speed information than EM sensors or cameras. Similarly, each sensor may have unique defects compared to other sensors. For example, a camera may be susceptible to occlusion by an operator or a nearby object that inadvertently blocks the image sensor's view of the mark. EM sensors may be more susceptible to jitter than other sensors. IMU sensors may have more inherent drift than other sensors. In one embodiment, the visual / inertial odometry module 30 executed by the processor of the console computer 110 takes into account each of these relative advantages and disadvantages to effectively combine separate sensor readings into a single output that accurately defines the UID posture, such as a tracking signal.

[0076] The sensor fusion module 28 systematically fuses multiple sensor data streams to achieve practical filtering of undesirable frequency components and robust and reliable estimation of the UID's attitude. In one embodiment, the sensor fusion module uses a linear quadratic estimator (LQE) for such filtering, which simultaneously corrects noisy input data and upsamples the data rate to match the robot control algorithm update rate. This can be achieved by combining sensor streams in a Kalman filter as part of the LQE. Other specializations of the LQE concept can achieve robust tracking using input sensors. Based on a combination of corresponding spatial state signals such as EM data and IMU data, the sensor fusion module 28 can generate not only a tracking signal corresponding to the UID's attitude as its output, but also a quality measure corresponding to the difference between the corresponding spatial state signals as its output. The output can be a low-latency signal at a higher update rate than each individual sensor source. In addition, the output signal can be smooth, which means that the high-frequency components of the individual signals are removed by the estimation model of the sensor fusion module.

[0077] The sensor fusion module 28 can combine the measurements of several sensors such as cameras, EM tracking systems, and IMUs to reliably determine the UID posture. The sensor fusion module 28 may include a visual-inertial odometry module 30 (VIOM 30), which receives image data from the camera and IMU data from the IMU. The visual / inertial odometry module can determine the posture of the UID based on the image data and the IMU data. The sensor fusion module may also include a sensor fusion algorithm, which can receive a processed tracking stream including combined camera and IMU data from the visual / inertial odometry module. The sensor fusion algorithm can also receive EM data from the EM sensor. In one embodiment, the sensor fusion algorithm fuses the tracking stream of the camera view, IMUD data, and the posture of the EM tracking. For example, the sensor fusion algorithm can combine the posture estimate output by the visual / inertial odometry module with the posture estimate of the EM tracking to generate an output tracking signal of the sensor fusion module 28.

[0078] This can help determine the spatial relationship between the sensors. In one embodiment, the data streams can be brought into the same coordinate system. This can be achieved based on predetermined knowledge of the UID construction. For example, the physical location of the IMU sensor within the device body 2 of the UID can be located at a known distance along the central axis from the location of the EM sensor. This known distance can be used to reference the position and motion data obtained from each sensor to a single principal coordinate, such as the center of gravity of the UID along the central axis between the IMU sensor and the EM sensor. The defined relationship between the sensors can be further refined by sampling sufficient corresponding output sequences from the sensors and solving one or more equations for the unknown spatial relationship (e.g., AX=XB, AY=YB, etc.).

[0079] In addition to correlating the spatial positions of the sensors to each other, a time calibration of the sensors may be performed. Time calibration may detect when the readings from the sensors arrive at the sensor fusion module 28, and may be a one-time calibration process performed at the beginning of each surgical procedure. Time calibration may estimate the relative delays in transmitting sensor data between different sensors, thereby establishing correlation between the measurements of the sensors (e.g., between the measurements of the EM tracking system and the measurements of the IMU tracking system).

[0080] Such consideration of the spatial and temporal relationships between the measured sensor data enables the sensor fusion module 28 to use the received sensor data to estimate the UID pose in real time (where the estimated UID pose is updated in a timely manner to ensure responsive and smooth control of the end effector on the robot arm.). In one embodiment, one of the several UID sensors may be updated at a higher rate than another sensor. Based on the spatial and temporal correlation of the sensors, the difference in sensor readings may be continuously determined while monitoring the sensor that is updated at the highest rate. This comparison may be used to continuously update the position data of the UID and detect divergences in the sensor readings.

[0081] The sensor fusion module 28 may use data from each tracking system to determine the bias of the tracking system. The bias includes characteristics of the tracking system, such as expected noise profile, expected accuracy, expected drift, and other characteristics that define the system bias. These characteristics may be incorporated into a predictive model that may be part of a sensor fusion algorithm in the sensor fusion module 28 to estimate the UID pose based on fusing at least two tracking streams into a single tracking stream that is an estimate of the UID pose.

[0082] Real-time sensor fusion may include generating a state vector for the UID that is a prediction made by a model that combines the current position value of the UID with the expected position value of the UID. That is, in the prediction phase, the current UID posture may be estimated at high frequency using previous sensor measurements and the prediction model. The prediction model incorporates forward-looking data (e.g., velocity data) obtained from previous sensor measurements to predict the current UID posture.

[0083] A state vector may be generated by a fusion algorithm and may define the current pose of the UID. The state vector may include information about the 6 degrees of freedom pose measured by the EM tracking sensors (the orientation and position of the EM sensors relative to a global reference frame), a linear velocity vector derived from IMU measurements, an angular velocity vector derived from IMU measurements, an acceleration derived from IMU measurements, a gravity direction derived from IMU measurements, a deviation of the acceleration vector and the angular velocity vector based on the determined deviation of the IMU tracking system, and a predetermined 6 degrees of freedom transformation from the IMU tracking system to the EM tracking system determined by the spatial relationship of the sensors.

[0084] The state vector may be updated over time by minimizing the accumulated errors of the constraints. More specifically, one or more of a tightness or loop constraint, an IMU constraint on acceleration, an IMU constraint on angular velocity, or a defined motion model constraint may have an associated error that is minimized to update the state vector. An IMU constraint on acceleration may constrain acceleration to: a – R_world_to_IMU * G + b_a, where acc is an IMU measurement. An IMU constraint on angular velocity may constrain angular velocity to: w = omega + b_w, where w is an IMU measurement. A motion model constraint may define a motion model to: omega_t = omega_t-1; v_t = v_t-1 + (a –omega_t-1 cross product v_t-1) * dt; or a = a_t-1.

[0085] In one embodiment, a weight is assigned to each of the constraints. For example, the weight may correspond to a previously identified bias of the tracking system. Alternatively, the weight may represent an inherent error score, which may include information specific to the tracking system, such as the quality of the EM field. Such factors allow the system to determine how reliable the tracking system is at any point in time and weight the system readings accordingly.

[0086] The sensor fusion module 28 may rely on other mathematical operations to perform the estimation. The sensor fusion module may try to find the minimum cumulative error of the sum of weighted constraints. The sensor fusion module may use Shur complement to remove old states from the packet. The sensor fusion module may adjust the constraint weights to include orientation in the equation described above. Alternatively, the orientation may be calculated separately by using the gyroscope measurements of the IMU in combination with the EMT orientation measurements. In addition, the orientation average may be calculated using quaternions or rotation matrices. The sensor fusion module may model noise in order to improve fusion quality. Optionally, the sensor fusion module may operate at a higher frequency to achieve an increase in the sampling rate.

[0087] Based on the model described above, the sensor fusion module 28 can output tracking signals derived from the fused tracking stream as well as quality metrics. The output tracking signals and quality metrics can be transmitted to the console controller 31 for further analysis and use. More specifically, the console controller 31 can use the tracking signals and quality metrics to generate output signals for use by digital control system software (e.g., being executed by one or more processors in the control tower 130) to control the associated surgical robotic arm 114 of the system.

[0088] The quality metric may correspond to the difference between the EM measurement and the IMU measurement. For example, the quality metric may represent the deviation of one of the sensor data streams. The quality metric may be based on the covariance of the data streams. The quality metric may also represent an inherent error score of each tracking system in the separate tracking systems. The quality metric may be based on one or more of these factors, for example, may be based on a combination of all of these factors.

[0089] In one embodiment, the quality metric may be used as an integrity check. The quality metric, which characterizes or compares the quality of sensor readings from the UID, may be compared to a predetermined threshold to determine whether the estimate of the UID pose (given in the tracking signal from the sensor fusion module 28) is accurate. For example, the quality metric is suggestive of the noise on each sensor, and when the noise exceeds a certain level, the pose estimate may be determined to be unreliable (and therefore the movement of the associated end effector or tool on the associated robotic arm will be paused). Thus, when the quality metric of the associated UID tracking of the associated end effector or tool is below a predetermined threshold, thereby indicating an excessively high difference between the readings from at least two of the tracking sensors, the console controller 31 may be configured to at least pause the associated end effector or tool (abort its movement).

[0090] The following additional statements of the present disclosure are made with respect to a UID having a camera for visual / inertial odometry. Aspect 1. A user interface device (UID) for a surgical robotic system, the UID comprising: a device body extending along a central axis; a plurality of gripping links mounted on the device body around the central axis; and a camera mounted on the device body. 2. The UID according to aspect 1, further comprising one or more tracking sensors mounted within the device body, wherein the one or more tracking sensors are configured to generate a spatial state signal in response to movement of the device body. 3. The UID according to aspect 2, wherein the one or more tracking sensors include an inertial measurement unit (IMU) and an electromagnetic (EM) sensor, and wherein the EM sensor is capable of measuring six degrees of freedom. 4. The UID according to aspect 1, wherein the camera is mounted on an end of the device body to view in a direction of the central axis away from the plurality of gripping links. 5. The UID of aspect 1, further comprising: a printed circuit board mounted within the device body, wherein the printed circuit board comprises one or more processors coupled to an image sensor of the camera and to an inertial measurement unit (IMU), wherein the image sensor generates visual data and the IMU generates inertial data; and a USB cable connected to the one or more processors to transmit the visual data and the inertial data. 6. The UID of aspect 5, wherein the USB cable is USB 3.0 version or higher.

[0091] The following additional statements of the present disclosure are made with respect to sensor fusion. Aspect 1. A surgical robotic system comprising: a user interface device (UID) having a device housing and a plurality of tracking sensors configured to generate corresponding spatial state signals in response to movement of the device housing; and a processor configured to generate a tracking signal corresponding to a posture of the UID and a mass metric corresponding to a difference between the corresponding spatial state signals based on a combination of the corresponding spatial state signals. 2. A surgical robotic system according to aspect 1, wherein the plurality of tracking sensors include an electromagnetic (EM) sensor configured to generate EM measurements and an inertial measurement unit (IMU) configured to generate IMU measurements, and wherein the mass metric corresponds to the difference between the EM measurements and the IMU measurements.

[0092] 3. A surgical robotic system according to aspect 2, wherein the processor is configured to pause the movement of the corresponding tool of the surgical robotic system when the quality measure is below a predetermined threshold.

[0093] In the foregoing description, the invention has been described with reference to specific exemplary embodiments of the invention. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. For example, as mentioned above, where the UID is rootless (not limited by mechanical linkages constraining the size of the operator's workspace), the operator may also perform teleoperation while being away from the user console 120 (e.g., at the edge of a table (next to the surgical robot platform 111)). In such embodiments, where the UID is tracked using multiple tracking modalities operating simultaneously as described above, there may be a marker 10 for visual modality estimation located, for example, on the shoulder of the operator or on the ceiling above the position of the operator 107 (at the edge of the table). Therefore, the specification and drawings should be considered in an illustrative rather than a restrictive sense.

Claims

1. A user input device for a surgical robot system, the user input device comprising: a device body extending from a proximal end to a distal end, the device body being configured to be held by an operator's hand for remote control of a surgical robotic arm, wherein the distal end of the device body is located forward of the proximal end and the proximal end is cupped in the operator's hand; a plurality of grip links mounted on the device body about a central axis of the device body and configured to be manipulated by the operator's hand to remotely operate an end effector coupled to the surgical robot arm; a camera having an image sensor and an imaging lens, the camera being mounted on the device body so that the imaging lens faces forward at the distal end of the device body to view indicia; and a user input device controller coupled to the image sensor, wherein the user input device controller sends a digital image of the marker produced by the image sensor to a processor, the processor computing a visual modality estimate of the pose of the user input device based on detecting the marker in the digital image.

2. A user input device according to claim 1, wherein the user input device also includes a first tracking sensor mounted on the device body, wherein the first tracking sensor is configured to generate a first spatial state signal in response to movement of the device body, and the user input device controller is coupled to the first tracking sensor and is configured to send the first spatial state signal from the first tracking sensor to the processor, and wherein the processor combines information from the visual modality estimation and the first spatial state signal to generate a tracking signal indicating the posture of the user input device.

3. The user input device of claim 2, wherein the first tracking sensor is an electromagnetic tracking sensor. 4 . The user input device of claim 2 , wherein the first tracking sensor is an inertial measurement unit and the first spatial state signal comprises accelerometer measurements and gyroscope measurements.

5. The user input device according to claim 4 further comprises a second tracking sensor, which is mounted on the device body and is an electromagnetic tracking sensor that generates a second spatial state signal in six degrees of freedom based on the posture of the user input device, wherein the processor generates the tracking signal by combining information from the visual modality estimation, the first spatial state signal and the second spatial state signal.

6. The user input device of claim 1, wherein the plurality of grip links are located behind the camera.

7. The user input device according to claim 2, further comprising: A serial data communication cable is connected to the user input device controller to transmit to the processor i) the digital image of the marker generated by the image sensor and ii) the first spatial state signal from the first tracking sensor.

8. A user input device according to claim 7, wherein the user input device also includes a second tracking sensor, which is mounted on the device body and is an electromagnetic tracking sensor that generates a second spatial state signal based on the posture of the user input device and in six degrees of freedom, wherein the processor generates the tracking signal by combining information from the visual modality estimation, the first spatial state signal and the second spatial state signal.

9. A user input device for a surgical robot system, the user input device comprising: a device body extending from a proximal end to a distal end, the device body being configured to be held by an operator's hand for remote control of the surgical robotic arm, wherein the distal end is located forward of the proximal end and the proximal end is cupped in the operator's hand; a camera having an image sensor and an imaging lens, the camera being mounted on the device body such that the imaging lens faces forward at the distal end of the device body to view indicia including a plurality of infrared light sources located on a user console of the surgical robotic system; an electromagnetic tracking sensor mounted on the device body and generating a first spatial state signal according to the posture of the user input device; and and a user input device controller coupled to the image sensor and the electromagnetic tracking sensor, wherein the user input device controller sends a digital image of the marker produced by the image sensor and the first spatial state signal generated by the electromagnetic tracking sensor to a processor, wherein the processor calculates a visual modality estimate of the pose of the user input device based on detecting the marker in the digital image, and combines the visual modality estimate with the first spatial state signal to produce a tracking signal indicative of the pose of the user input device.

10. A user input device according to claim 9, wherein the user input device also includes an inertial measurement unit mounted on the device body, wherein the inertial measurement unit will generate a second spatial state signal based on the posture of the user input device, and the user input device controller is coupled to the inertial measurement unit and is configured to send the first spatial state signal, the second spatial state signal and the digital image of the marker to the processor, and wherein the processor will generate the tracking signal by combining information from the visual modality estimation, the first spatial state signal and the second spatial state signal.

11. The user input device of claim 10, the user input device in combination with the processor calculating a quality metric indicative of a difference between i) electromagnetic tracking sensor measurements in the first spatial state signal and ii) inertial measurement unit measurements in the second spatial state signal.

12. A method for remotely controlling a surgical robot arm in a surgical robot system, the method comprising: generating, by a camera looking outward from a user input device of a surgical robotic system, a plurality of digital images capturing indicia including a plurality of infrared light sources located on a user console of the surgical robotic system; digitally processing the digital image to detect an object in the digital image corresponding to the marking; interpreting the detected objects to compute a visual modality estimate of a pose of the user input device, wherein the visual modality estimate tracks movement of the user input device; as well as Actuator control commands are generated based on the visual modality estimation to control the surgical robotic arm to cause movement of an end effector coupled to the surgical robotic arm that mimics the movement of the user input device.

13. The method according to claim 12, further comprising: generating a first spatial state signal according to the posture of the user input device by an electromagnetic tracking sensor; as well as Information from the visual modality estimate and from the first spatial state signal is combined to produce a tracking signal indicative of the pose of the user input device, wherein generating the actuator control command is based on the tracking signal.

14. The method according to claim 12, further comprising: generating a first spatial state signal according to the posture of the user input device by an inertial measurement unit in the user input device; as well as Information from the visual modality estimate and from the first spatial state signal is fused, by a processor, to produce a tracking signal indicative of the pose of the user input device, wherein generating the actuator control command is based on the tracking signal.

15. The method of claim 14, wherein the first space state signal comprises accelerometer measurements and gyroscope measurements.

16. The method according to claim 14, further comprising: generating a second spatial state signal according to the posture of the user input device and in six degrees of freedom by an electromagnetic tracking sensor in the user input device; Information from i) the visual modality estimate, ii) the first spatial state signal, and iii) the second spatial state signal is fused, by the processor, to produce the tracking signal.

17. The method according to claim 16, further comprising: A mass metric is calculated, the mass metric indicating a difference between i) an electromagnetic tracking sensor measurement of the electromagnetic tracking sensor and ii) an inertial measurement unit measurement of the inertial measurement unit, wherein generating the actuator control command is based on the mass metric.

18. The method of claim 17, wherein the processor is configured to pause movement of the end effector coupled to the surgical robotic arm in response to the quality metric being below a predetermined threshold, even if the user input device continues to move.

19. The method of claim 12, wherein the digital image is an infrared image, and wherein digitally processing the digital image to detect the object corresponding to the marker comprises A stored data structure is accessed that describes a relative position of each of a plurality of objects that comprise the tag.

20. The method of claim 19, wherein the plurality of infrared light sources are arranged in an inverted Y-shape on a table of an operator display at the user console, wherein the operator display will display the video from the endoscope during teleoperation.

Images