Aligning a positioning system using mechanical linkages

Abstract

The positioning system includes a group of positioning devices including a first device with a first positioning source associated with a first positioning modality configured to view a first field, a second device with a second positioning source associated with a second positioning modality configured to view a second field and of a different type than the first positioning modality, a third device with one or more first markers detectable in the first field using the first positioning modality, and a fourth device with one or more second markers detectable in the second field using the second positioning modality. A coupling structure physically couples two of the group of positioning devices to each other in a fixed and precise relative position and orientation.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Patent Application No. 63 / 295,651, filed December 31, 2021, and entitled "POSITIONING SYSTEM REGISTRATION USING MECHANICAL LINKAGES," the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0002] The present disclosure relates to positioning systems, including positioning systems implemented for tracking medical instrumentation. In particular, aspects of the present disclosure relate to the alignment of different positioning systems / modalities to one another. [Brief description of the drawings]

[0003] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numbers may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an embodiment of a robotic medical system configured for diagnostic and / or therapeutic ureteroscopy procedures in accordance with one or more embodiments. [Diagram 2] 1 illustrates a robotic system arranged for diagnostic and / or therapeutic bronchoscopy, according to one or more embodiments. [Diagram 3] 1 illustrates a pedestal-based robotic system in accordance with one or more embodiments. [Figure 4] 4 illustrates medical system components that may be implemented in any of the medical systems of FIGS. 1-3, according to one or more embodiments. [Diagram 5] 1 illustrates an arrangement of multiple positioning systems in accordance with one or more embodiments. [Figure 6] 1 shows an arrangement of multiple positioning systems associated with an integrated marker. [Figure 7] 1 illustrates various positioning system markers in accordance with one or more embodiments. [Figure 8] 1 illustrates an electromagnetic and fluoroscopic positioning system associated with integrated marker(s) in accordance with one or more embodiments. [Figure 9] 1 illustrates an electromagnetic and robotic positioning system associated with integrated marker(s) in accordance with one or more embodiments. [Figure 10] 1 illustrates a multiple positioning system with mechanically fixed sources in accordance with one or more embodiments. [Figure 11] 1 illustrates an electromagnetic and robotic positioning system with a relatively fixed source in accordance with one or more embodiments. [Figure 12] 1 illustrates a three-dimensional position overlay of an instrument representation on a two-dimensional image in accordance with one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] The headings provided herein are merely for convenience and do not necessarily affect the scope or meaning of the claimed invention. Although certain preferred embodiments and examples are disclosed below, the inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or applications, as well as to modifications and equivalents thereof. Thus, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described in sequence as multiple separate operations in a manner that may be helpful in understanding a particular embodiment, however, the order of description should not be construed to imply that these operations are order dependent. Furthermore, the structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments are described. Not necessarily all such aspects or advantages are realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] While certain spatially relative terms, such as "outer," "inner," "superior," "lower," "below," "upper," "vertical," "horizontal," "top," "bottom," "lateral," and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it is understood that these terms are used herein for ease of description to describe positional relationships between elements / structures, such as with respect to the illustrated orientations of the drawings. It is understood that the spatially relative terms are intended to encompass different orientations of the elements / structures in use or operation in addition to the orientation shown in the drawings. For example, an element / structure described as being "above" another element / structure may represent a position below or to the side of such other element / structure, relative to the intended patient or alternative orientations of the element / structure, and vice versa. It is understood that the spatially relative terms, including those listed above, may be understood with respect to the illustrated orientation of each of the referenced figures.

[0006] Certain reference numbers are reused across different figures of the set of figures of this disclosure for convenience of devices, components, systems, features, and / or modules having features that may be similar in one or more respects. However, with respect to any of the embodiments disclosed herein, the reuse of common reference numbers in the figures does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, one skilled in the art may be informed by the context as to the extent to which the use of common reference numbers may imply similarity between the referenced subject matter. The use of a particular reference number in the context of the description of a particular figure may be understood to relate to the device, component, aspect, feature, module, or system identified in that particular figure, and not necessarily to any device, component, aspect, feature, module, or system identified by the same reference number in another figure. Furthermore, aspects of separate figures identified by a common reference number may be interpreted as sharing characteristics or as being entirely independent of each other. In some contexts, features associated with separate figures identified by a common reference number are not related and / or similar with respect to at least certain aspects.

[0007] The present disclosure provides systems, devices, and methods for co-registering different image / positioning spaces associated with different positioning systems and / or modalities. In particular, the systems, devices, and methods according to one or more aspects of the present disclosure can facilitate simplified multi-modal registration by implementing mechanical linkages / fixations between positioning sources and / or detectable markers / sensors associated with each modality.

[0008] With respect to the medical instruments described in this disclosure, the term "instrument" is used according to its broad and ordinary meaning and may refer to any type of tool, device, assembly, system, subsystem, apparatus, component, etc. In some contexts herein, the term "device" may be used substantially interchangeably with the term "instrument."

[0009] Robotic surgical systems can be utilized to facilitate instrument navigation to regions within a patient's body. In some embodiments, the robotic system can be configured to provide an interface that allows an operator to navigate the robotically controlled instrumentation by directing the movement of the instrumentation in multiple degrees of freedom. For example, an operator may direct the axial translation (i.e., insertion and / or retraction), articulation angle, and / or roll (i.e., articulation angle direction) of an endoscope, access sheath, guidewire, working instrument (e.g., needle, basket, lithotripsy device, etc.). Navigation within organs, branch vessel ostia, or other relatively open three-dimensional spaces can be challenging due to the need to understand the three-dimensional relationship of the navigated / tracked instrument to the anatomical target and / or determine in which plane the instrument will bend. This task can be challenging, in part, because navigating an instrument through a patient's lumen from a remote patient access point to the desired site of the procedure requires manipulating the instrument without a direct line of sight to the instrument. A positioning / tracking system may be used to help locate the desired site of the procedure and visualize the navigation of the instrument to the desired site of the procedure. The positioning / tracking system allows a user to visualize the patient's internal anatomy and the position and / or orientation of the instrument's detectable markers within the patient's anatomy.

[0010] The positioning system may include imaging systems / modalities such as positron emission tomography (PET), x-ray computed tomography (CT), x-ray fluoroscopy, magnetic resonance imaging (MRI), camera-based optical systems, and ultrasound or other sound wave imaging systems. The positioning system may further include electromagnetic (EM) tracking systems (e.g., using electromagnetic field generators as described in detail herein), fiber optic tracking systems, and robot tracking / positioning based on robot data (e.g., robot actuator, torque, pose data). Some imaging systems / modalities are not suitable for continuous real-time tracking of instruments, such as PET, CT, and MRI, and generally generate and combine many cross-sectional images of an object to generate a computer-processed image. Such image capture processes may be relatively slow, and movement within the image field during the image capture process may generate image artifacts that make such systems unsuitable for real-time tracking of moving instruments within the body. In addition, some imaging systems / modalities, such as x-ray CT and fluoroscopy, emit potentially harmful ionizing radiation, and it may be desirable to limit their duration of use.

[0011] Electromagnetic (EM) and fiber optic tracking systems can provide real-time instrument tracking. EM tracking generally works by detecting / determining the position / orientation of an EM sensing coil (i.e., an EM marker / sensor) in a changing magnetic field. The changing magnetic field induces a current in the coil based on the coil's position and orientation in the magnetic field. Thus, the position and orientation of the coil can be determined by measuring the current in the coil. In some cases, a single EM sensor / marker can sense its position and orientation in three-dimensional space with five degrees of freedom. That is, the EM sensor can provide data indicative of orientation in every direction except around the coil's axis of axial symmetry (i.e., roll). Two EM sensors / markers held at a fixed relative position and orientation on the instrument or other marker device may be used to sense all six degrees of freedom of the instrument. In a navigation system employing EM tracking, images of an anatomical space can be acquired, and the system control circuitry is configured to determine an alignment between the frame of reference of an EM sensor / marker associated with a tracked instrument and the frame of reference of an imaging system / modality used to image the anatomical space to describe movement of the tracked instrument within the imaged anatomical space.

[0012] Although certain aspects of the present disclosure are described in detail herein in the context of bronchoscopy and ureteroscopy procedures, it will be understood that such context is provided for convenience and clarity, and that the instrument positioning concepts disclosed herein are applicable to any suitable medical procedure.

[0013] With respect to ureteroscopy procedures, a surgeon may insert an endoscope (e.g., a ureteroscope) through the urethra and into the ureter to remove urinary stones from the bladder and ureter, such as for the purpose of removing kidney stones. In some procedures, a physician may use a percutaneous nephrolithotomy ("PCNL") technique, which involves inserting a nephrolithoscope through the skin (i.e., percutaneously) and intervening tissue to gain access to the treatment site for crushing and / or removing the stone. Relatively large kidney stones can be broken into relatively small pieces to facilitate their removal using certain instrumentation, such as laser lithotripsy devices. According to some procedures, a basket device / system may be used to capture and extract the relatively smaller stone fragments from the treatment site and out of the patient. Any of the instrumentation associated with such ureteroscopy procedures may be robotically controlled and / or positionally tracked by tracking / detecting marker(s) / sensor(s) associated with the instrumentation using a positioning modality as described in detail herein.

[0014] The present disclosure provides systems, devices, and methods for aligning the coordinate frames of two or more positioning systems / modalities with each other. Such alignment can be achieved in a simplified manner made possible by the implementation of specific mechanical / physical couplings / connections between emitters and / or markers associated with different positioning modalities. In some implementations, such mechanical couplings are implemented by integrating and / or establishing a fixed relationship between markers (e.g., sensors) that are visible or identifiable to the multiple different positioning modalities. For example, such integration / fixing can include physically coupling, embedding, or overlaying markers associated with two or more different positioning modalities with / in / on each other such that the markers have a common center location and / or correspond to a common coordinate frame with a common origin. Such co-location of marker frames can eliminate the need to calculate and / or perform a transformation to convert between the physical location of the markers of the first modality and the physical location of the markers of the second modality when aligning a first modality with a second modality. The term "marker" is used herein according to its broad ordinary meaning and can refer to any device or structure, or group thereof, having one or more surfaces, structures, or components that are detectable, visible, or otherwise identifiable within a positioning space (e.g., field of view) associated with a positioning modality.

[0015] In some implementations, a mechanical / physical coupling between positioning modalities for the purpose of simplifying registration may involve establishing a fixed relationship between emitters / sources for multiple positioning modalities, such as fluoroscopy, optical, robotic posture-based, and / or electromagnetic field positioning modalities. For example, both positioning system sources / emitters may be rigidly coupled and / or integrated into a common rigid physical structure, such as the C-arm of a fluoroscopy system. Such mechanical fixation may establish a constant positional relationship and constant registration between a first modality and a second modality, thereby eliminating the need to calculate or perform a transformation to convert between the physical location of the emitter / source of the first positioning modality and the physical location of the emitter / source of the second positioning modality when registering the first modality to the second modality.

[0016] Healthcare System FIG. 1 illustrates an example medical system 100 for performing various medical procedures, according to aspects of the present disclosure. The medical system 100 may be used, for example, in ureteroscopic procedures. Robotic medical solutions may provide relatively greater precision, greater control, and / or greater hand-eye coordination for certain instruments compared to strictly manual procedures. Although the system 100 of FIG. 1 is presented in the context of a ureteroscopic procedure, it should be understood that the principles disclosed herein may be implemented in any type of endoscopic procedure.

[0017] The medical system 100 includes a robotic system 10 (e.g., a mobile robotic cart) configured to engage and / or control a medical instrument 19 (e.g., a ureteroscope) including a proximal handle 31 and a shaft 40 coupled to the handle 31 at its proximal portion to perform a procedure on a patient 7. It should be understood that the instrument 40 may be any type of shaft-based medical instrument, including an endoscope (such as a ureteroscope or bronchoscope), a catheter (such as a steerable or non-steerable catheter), a needle, a nephroscope, a laparoscope, or other type of medical instrument. The instrument 40 may access the internal patient anatomy through direct access (e.g., through a natural orifice) and / or through percutaneous access via a skin / tissue puncture.

[0018] The medical system 100 includes a control system 50 configured to interface with the robotic system 10 to provide information regarding a procedure and / or perform various other operations. For example, the control system 50 may include one or more displays 56 configured to present certain information to assist the physician 5 and / or other technicians or individuals. The medical system 100 may include a table 15 configured to hold a patient 7. The system 100 may further include an electromagnetic (EM) field generator, such as a robot-mounted EM field generator 80 or / and an EM field generator 85, mounted on the table 15 or other structure.

[0019] While various robotic arms 12 are shown in various positions and coupled to various tools / devices, it should be understood that such configurations are shown for convenience and illustrative purposes and that such robotic arms may have different configurations over time and / or at different points during a medical procedure. Additionally, the robotic arms 12 may be coupled to different devices / instruments than those shown in FIG. 1, and in some cases or periods, one or more of the arms may not be utilized or coupled to a medical instrument. Coupling of instruments to the robotic system 10 may be via a robotic end effector 6 associated with a distal end of each arm 12. The term "end effector" is used herein according to its broad and ordinary meaning and may refer to any type of robotic manipulator device, component, and / or assembly. The terms "robotic manipulator" and "robotic manipulator assembly" are used according to their broad and ordinary meaning and may collectively or individually refer to a robotic end effector and / or a sterile adaptor or other adaptor component coupled to an end effector. For example, a "robot manipulator" or a "robot manipulator assembly" may refer to an instrument device manipulator (IDM) that includes one or more drive outputs, whether embodied in a robotic end effector, adapter, and / or other component.

[0020] In some embodiments, the physician 5 can interact with the control system 50 and / or the robotic system 10 to cause / control the robotic system 10 to advance and navigate the medical instrument shaft 40 (e.g., scope) through the patient's anatomy to a target site and / or to perform certain operations using associated instrumentation. The control system 50 may provide information associated with the medical instrument 40 and / or other instruments of the system 100, such as real-time endoscopic images captured thereon, via the display 56, to assist the physician 5 in navigating / controlling such instruments. The control system 50 may also provide imaging / location information to the physician 5 based on a particular positioning modality, such as fluoroscopy, ultrasound, optical / camera imaging, EM field positioning, or other modalities, as described in detail herein.

[0021] Various scope / shaft-type instruments disclosed herein, such as shaft 40 of system 100, can be configured to navigate within the human anatomy, such as within a natural orifice or lumen of the human anatomy. The terms "scope" and "endoscope" are used herein according to their broad and ordinary meaning and may refer to any type of elongated (e.g., shaft-type) medical instrument having imaging, viewing, and / or capturing capabilities and configured to be introduced into any type of organ, cavity, lumen, chamber, or space of the body. Scopes can include, for example, ureteroscopes (e.g., for accessing the urinary tract), laparoscopes, nephroscopes (e.g., for accessing the kidney), bronchoscopes (e.g., for accessing the airways such as the bronchi), colonoscopes (e.g., for accessing the colon), arthroscopes (e.g., for accessing a joint), cystoscopes (e.g., for accessing the bladder), colonoscopes (e.g., for accessing the colon and / or rectum), borescopes, and the like. A scope / endoscope, in some cases, may comprise at least a portion of a rigid and / or flexible tube and may be sized to be passed through an outer sheath, catheter, introducer, or other luminal device, or may be used without such a device. Endoscopes and other instruments described herein may have associated with them at their distal ends or other portions certain markers / sensors configured to be visible / detectable within a field of view / space associated with one or more positioning (e.g., imaging) systems / modalities.

[0022] The system 100 is illustrated as including a fluoroscopy system 70, which includes an x-ray generator 75 and an image detector 74 (referred to in some contexts as an “image intensifier”). Either component 74, 75 may be referred to herein as a “source” and both may be mounted on a movable C-arm 71. A control system 50 or other system / device may be used to store and / or manipulate images generated using the fluoroscopy system 70. In some embodiments, the bed 15 is radiolucent so that radiation from the generator 75 can pass through the bed 15 and a target area of ​​the patient's anatomy, and the patient 7 is positioned between the ends of the C-arm 71. The structure / arm 71 of the fluoroscopy system 70 may be rotatable or fixed. The fluoroscopy system 70 may be implemented to allow live images to be viewed to facilitate image-guided surgery. The structure / arm 71 may be selectively movable to allow various images of the patient 7 and / or the surgical field to be taken by the fluoroscopy panel source 74.

[0023] In the exemplary urology configuration shown in FIG. 1, the robotic arm 12c is shown holding a field generator 80. Because the electric field generated by the electric field generator 80 may be distorted by the presence of metal or other conductive components therein, it may be desirable to position the arm 12c such that other components of the system do not substantially interfere with the electric field. For example, it may be desirable to position the field generator 80 at least 8 inches or more away from a support arm 71 associated with a fluoroscopy system. In some embodiments, the system 100 includes an EM field generator 85 mounted to a table 15 or other structure (e.g., a freestanding structure).

[0024] The system 100 (as well as other systems disclosed herein) can include an optical imaging source 99, such as a camera device (e.g., a stereoscopic camera assembly). The optical imaging source 99 may be configured / used to view a field of view within a surgical environment and identify specific markers disposed within the field of view. For example, in some embodiments, the imaging source 99 can emit infrared (IR) or other frequency electromagnetic radiation and / or detect reflections of such radiation to identify markers including surfaces that reflect such radiation. Such optical deflections can indicate the location and / or orientation of the marker associated with a particular optical modality. The system 100 can have specific markers / fiducials 65, 67 that can be detectable / locatable in one or more reference / coordinate frames / spaces associated with respective localization modalities.

[0025] FIG. 2 illustrates a cart-based robotic system 101 arranged for diagnostic and / or therapeutic bronchoscopy, according to one or more embodiments. During bronchoscopy, the arm 12 of the robotic system 10 can be configured to drive a medical instrument shaft 40, such as a steerable endoscope, which can be a procedure-specific bronchoscope for bronchoscopy, through a natural orifice access point (e.g., the mouth of a patient 7, which in this example is positioned on a table 15) to deliver diagnostic and / or therapeutic tools. As shown, the robotic system 10 (e.g., a cart) can be positioned adjacent to the patient's upper torso to provide access to the access point. The arrangement of FIG. 2 can also be utilized when performing gastrointestinal (GI) procedures using a gastroscope.

[0026] Similar to the system 100 of FIG. 1, the instrument / scope 40 may access the target anatomical structure through an access sheath. For surgical bronchoscopy, the endoscope 40 may be directed downstream of the patient's trachea and lungs after insertion using precise commands from the robotic system 10 until the target surgical site is reached. For example, the endoscope 52 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule within the patient's lung. The system 101 may include an optical / camera-based positioning / imaging source 90.

[0027] 3 illustrates a table-based robotic system 102 in accordance with one or more embodiments of the present disclosure. System 102 incorporates robotic components 105 into a table / platform 115, thereby allowing for a reduced amount of capital equipment in an operating room compared to some cart-based robotic systems. Table-integrated robotic systems such as system 102 can allow for greater access to the patient in some instances. Similar to cart-based systems 101, 102, the instrument device manipulator assembly associated with the robotic arm 112 of system 102 can generally comprise an instrument and / or instrument feeder designed to manipulate an elongated medical instrument / shaft, such as a catheter 40.

[0028] As shown, the robotic platform system 104 can include a column 144 coupled to one or more carriages 141 (e.g., a ring-shaped movable structure) from which one or more robotic arms 112 can emanate. The carriages 141 can translate along a vertical column interface through at least a portion of the length of the column 144 to provide different vantage points from which the robotic arms 112 can be positioned. In some embodiments, the carriages 141 can rotate about the column 144 to allow the robotic arms 112 to access multiple sides of the table 104. The rotation and / or translation of the carriages 141 can enable the system 102 to align medical instruments, such as endoscopes and catheters, to different access points on a patient.

[0029] In any of the systems of Figures 1-3, tracking of the position of the robotically advanced instrumentation within the patient can be facilitated by the use of multiple localization modalities, including, for example, fluoroscopy, EM field sensing, optical imaging, robot pose estimation, ultrasound, etc. As described in detail herein, registering multiple modalities with each other allows a position determined in one modality to be tracked and / or superimposed on an image field associated with another modality. Markers may be placed on or otherwise integrated with the tracked instrument to allow such markers / instruments to be detected and located within the respective localization / image space of the associated localization modality. With regard to tracking of a target anatomical structure within the patient, an endoscope or other instrument may be used to deliver / indicate fiducials to "mark" the location of the target anatomical structure.

[0030] 1-3 and 4, which illustrates components of an exemplary embodiment of any of the subsystems of FIG. 1-3, the control system 50 can be configured to provide various functions to assist in the performance of a medical procedure. For example, the control system 50 can communicate with the robotic system 10 via a wireless or wired connection (e.g., to control the robotic system 10). In some embodiments, the control system 50 can communicate with the robotic system 10 to receive position data from the robotic system regarding the location of the distal end of the scope 40 or other instrumentation. Such positioning data may be derived using one or more markers (e.g., electromagnetic sensors, radiopaque markers, etc.) associated with the respective instrumentation and / or based at least in part on robotic system data (e.g., arm position / pose data, known parameters or dimensions of various system components, etc.). In some embodiments, the control system 50 can communicate with the EM field generators 80 / 85 to control the generation of EM fields in the area around the patient 7 and / or around the tracked instrumentation.

[0031] 4 further illustrates an exemplary embodiment of the robotic system 10 of any of FIGS. 1-3. The robotic system 10 can include one or more robotic arms 12, each of which can include multiple arm segments 23 coupled to joints 24 that can provide multiple motion / degrees of freedom. Once the robotic system 10 is properly positioned, the scope 40 can be inserted into the patient 7 robotically using the robotic arms 12, manually by the physician 5, or a combination thereof. One of the arms 112 can be associated with an instrument coupling / manipulator 31 configured to facilitate advancement and movement of the scope 40.

[0032] The robotic system 10 can be physically and / or communicatively coupled to any component of a medical system, such as the control system 50, the table 15, the EM field generators 80 / 85, the scope 40, the fluoroscopy system 70, and / or any type of percutaneous access instrument (e.g., needle, catheter, nephroscope, etc.). The robotic system 10 can be configured to receive control signals from the control system 50 to perform certain operations, such as positioning one or more robotic arms 12 in a particular manner, manipulating the scope 40, etc. In response, the robotic system 10 can be controlled to perform operations using certain control circuits 211, actuators 217, and / or other components of the robotic system 10. For example, the control circuit 211 may control various motors / actuators associated with various joints of the robotic arms 12 and / or arm support 17. In some embodiments, the robotic system 10 and / or the control system 50 are configured to receive images and / or image data from the scope 40 that are representative of the internal anatomy of the patient 7 and / or portions of the access sheath or other device components.

[0033] The robotic system 10 generally includes an elongated support structure 14 (also referred to as a "column"), a robotic system base 25, and a console 13 at the top of the column 14. The column 14 may include one or more arm supports 17 (also referred to as a "carriage") for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 1). The arm supports 17 may be configured to translate vertically along the column 14. In some embodiments, the arm supports 17 may be connected to the column 14 through slots 20 positioned on either side of the column 14 to guide the vertical translation of the arm supports 17. The slots 20 accommodate a vertical translation interface for positioning and holding the arm supports 17 at various vertical heights relative to the robotic system base 25. The base 25 balances the weight of the column 14, the arm supports 17, and the arms 12 on the floor.

[0034] The robotic arm 12 may generally comprise a robotic arm base 21 and end effectors 6 separated by a series of articulated arm segments 23 connected by a series of joints 24, each joint comprising one or more independent actuators 217. Each actuator may comprise an independently controllable motor. Each independently controllable joint 24 may provide or represent an independent degree of freedom available to the robotic arm. In some embodiments, each of the arms 12 has seven joints, thus providing seven degrees of freedom, including a "redundant" degree of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its respective end effector 6 at a specific position, orientation, and trajectory in space using different linkage positions and joint angles. This allows the system to position and orient the medical instrument from a desired point in space, while allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions. When positioned at the top of the column 14, the console 13 can provide both a user interface for receiving user input and a display screen 56 (or a dual-purpose device, e.g., a touch screen) for providing both pre-operative and intra-operative data to the physician / user. The robotic cart 10 can further include a handle 27, as well as one or more wheels 28.

[0035] Each end effector 6 of the robotic arm 12 may include an instrument device manipulator (IDM; e.g., scope handle 31) or may be configured to couple with an instrument device manipulator (IDM), which may be attached using a sterile adapter component in some examples. The combination of the end effector 6 and associated IDM, and any intervening artifices or couplers (e.g., sterile adapters), may be referred to as a manipulator assembly 111. The IDM may provide a power and control interface. For example, the interface may include connectors for transmitting pneumatic, electrical power, electrical signals, and / or optical signals from the robotic arm 12 to the IDM. The IDM 29 may be configured to manipulate a medical instrument (e.g., a surgical tool / instrument), such as a scope 40, using techniques including, for example, direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, and the like.

[0036] As mentioned above, system 100 may include certain control circuits configured to perform certain functions described herein, including control circuitry 211 of robotic system 10 and control circuitry 251 of control system 50. That is, the control circuitry of systems 100, 101, 102 may be part of robotic system 10, control system 50, or some combination thereof. Thus, all references herein to control circuits may refer to circuitry embodied in a robotic system, a control system, or any other component of a medical system, such as medical systems 100, 101, and 102 shown in FIGS. 1-3, respectively. The term "control circuitry" is used herein according to its broad and ordinary meaning and may refer to any collection of processors, processing circuits, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active and / or passive devices and / or connectivity circuits), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuits, analog circuits, digital circuits, and / or any devices that manipulate signals (analog and / or digital) based on hard-coded and / or operational instructions in the circuits. The control circuitry referred to herein may further include one or more circuit boards (e.g., printed circuit boards), conductive traces and vias, and / or mounting pads, connectors, and / or components. The control circuitry referred to herein may further comprise one or more storage devices, which may be embodied in a single memory device, multiple memory devices, and / or embedded circuitry of the device. Such data storage devices may comprise read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device that stores digital information.It should be noted that in embodiments in which the control circuitry comprises hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage devices / registers that store any associated operational instructions may be embedded within or external to the circuitry that comprises the state machines, analog circuits, digital circuits, and / or logic circuits.

[0037] The control circuitry 211, 251 may comprise a computer-readable medium that stores and / or is configured to store hard-coded and / or operational instructions corresponding to at least some of the steps and / or functions illustrated in one or more of the figures and / or described herein. Such computer-readable medium may, in some cases, be included in an article of manufacture. The control circuitry 211 / 251 may be maintained / disposed entirely locally or may be at least partially remotely located (e.g., indirectly communicatively coupled via a local area network and / or wide area network). Either of the control circuits 211, 251 may be configured to perform any aspect(s) of the various processes disclosed herein.

[0038] 4, the control system 50 can include various I / O components 258 configured to assist the physician 5 or others in performing a medical procedure. For example, the input / output (I / O) components 258 can be configured to allow user input to control / navigate the scope 40 and / or basket system within the patient 7. In some embodiments, for example, the physician 5 can provide input to the control system 50 and / or the robotic system 10, and in response to such input, can send control signals to the robotic system 10 to operate the scope 40 and / or other robotically controlled instrumentation.

[0039] The control system 50 and / or the robotic system 10 may include specific user controls (e.g., controls 55), which may comprise one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, track pads, track balls, control pads, and / or any type of user input (and / or output) device or device interface, such as sensors (e.g., motion sensors or cameras) that capture hand and finger gestures, touch screens, and / or interfaces / connectors therefor. Such user controls are communicatively and / or physically coupled to respective control circuits. The control system may include a structural tower 51, and one or more wheels 58 that support the tower 51. The control system 50 may further include specific communication interfaces 254 and / or power interfaces 259.

[0040] In some embodiments, the endoscope assembly 30 includes a handle or base 31 that is coupled to an endoscope shaft 40 (which may, in certain contexts herein, be referred to simply as an "endoscope" or "scope"). For example, the endoscope 40 may include an elongated shaft that includes one or more lights 49 and one or more cameras or other imaging devices 48. The scope 40 may further include one or more working channels 44, which may span the length of the scope 40.

[0041] The scope assembly 30 may further comprise one or more positioning markers and / or sensors 63, which may be configured to generate a signal indicative of the position of the marker(s) / sensor(s) 63 within the electromagnetic field. Such markers 63 may comprise, for example, one or more conductive coils (or other embodiments of antennas), which may be arranged at known, fixed orientations relative to one another to enable the determination of multiple degrees of freedom for position determination. The markers 63 may be configured to generate and / or transmit sensor position data to another device and / or generate a detectable distortion or signature within the electromagnetic field. The sensor / marker position data may indicate the position and / or orientation of the medical instrument 40 (e.g., its distal end 42) and / or may be used to determine / estimate the position / or orientation of the medical instrument.

[0042] The scope 40 may be articulatable, such as with respect to at least a distal portion 42 of the scope 40, such that the scope 40 may be maneuvered within the human anatomy. In some embodiments, the scope 40 is configured to be articulated with six degrees of freedom, including, for example, XYZ coordinate translation, as well as pitch, yaw, and roll. Certain position sensors (e.g., electromagnetic sensors) of the scope 40, if implemented, may have similar degrees of freedom with respect to the position information they generate / provide.

[0043] In the case of a robotic implementation, a robotic arm of a robotic system may be configured / configurable to manipulate the scope 40. For example, an instrument device manipulator (e.g., a scope handle) may be coupled to an end effector of the robotic arm and may manipulate the scope 40 using an elongated translation member. The elongated translation member may include one or more pull wires (e.g., pull or push wires), cables, fibers, and / or a flexible shaft. For example, the robotic arm may be configured to actuate multiple pull wires (not shown) coupled to the scope 40 to deflect the tip 42 of the scope 40.

[0044] In various embodiments, the anatomical space in which the scope 40 or other instrument may be located (i.e., the position of the scope / instrument is determined / estimated) is a three-dimensional portion of the patient's vasculature, tracheobronchial airway, urinary tract, gastrointestinal tract, or any organ or space accessed via such lumen. Various positioning / imaging modalities may be implemented to provide an image / representation of the anatomical space. Suitable imaging subsystems include, for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography, cone-beam CT, 3DRA, single photon emission computed tomography (SPECT), MRI, optical coherence tomography (OCT), and ultrasound. One or both of pre-treatment and intra-treatment images may be acquired. In some embodiments, the pre-treatment and / or intra-treatment images are acquired using a C-arm fluoroscope. In connection with some embodiments, specific positioning and imaging systems / modalities are described. It should be understood that such description may relate to any type of positioning system / modality.

[0045] FIG. 5 illustrates an arrangement of multiple positioning systems / modalities according to one or more embodiments. In particular, the system of FIG. 5 includes a first positioning system / modality 350 and a second positioning system / modality 380. Each of the positioning systems 350, 380 includes a generator / source device, such as a camera, an electromagnetic field generator, a radio frequency radiation generator, an ionizing radiation generator (e.g., X-ray / fluoroscopy), etc. Each of the modalities / systems is configured to detect / identify a respective marker or set of markers 365, 367 to determine its location. It should be understood that the modalities / systems illustrated in FIG. 5 can be any type of imaging or positioning system. Thus, while some aspects of the system 500 are described below in the context of a particular modality / system, it should be understood that such exemplary modalities / systems are described for convenience and that the description also pertains to other types of modalities / systems.

[0046] In an exemplary system, the first modality 350 may be an imaging modality, such as an optical camera-based system, and the source 350 comprises one or more cameras associated with a field of view 353. Placement of a particular marker within the field of view may enable positioning / localization of the marker within the field of view by processing of the image(s) captured by the camera(s). In some implementations, the imaging source 350 comprises multiple cameras set at relative angular offsets with respect to one another to provide images containing information from which three-dimensional (3D) positioning can be derived.

[0047] In some embodiments, the positioning system 350 is a fluoroscopy system or other imaging system. For example, as shown in Figures 1-3, a fluoroscopy system may include a C-arm or other structural support component. In some implementations, it may be necessary or desirable to physically move and / or reorient the imaging source 350 to acquire multiple images at different angles that allow deriving 3D positioning from the generated images.

[0048] As another example, the second modality / system 380 may be an electromagnetic (EM) field generator system, as described in detail herein. The field generator 380 may comprise one or more antennas configured to emit an electromagnetic field 383 within a defined region, and a number of sensors 363, such as conductive coils, positioned within the electromagnetic field 383 are detectable by currents induced therein due to the electromagnetic field.

[0049] The EM positioning system 380 can be used to track a medical instrument such as an endoscope. For example, the sensor / marker 363 may be integrated with the distal end of the endoscope to enable localization of the scope using the EM system 380. As used herein, the positioning system may also be referred to as a tracking system, a shape tracking system, or a localization subsystem. The term "localization" is used herein according to its broad and ordinary meaning and may refer to systems and methods for determining and / or monitoring the position (i.e., location and / or orientation) of an object, such as a medical instrument or tool, in a reference coordinate system. The term "localized" is similarly used herein according to its broad and ordinary meaning and may refer to a detected / sensed marker, the position and / or orientation of which has been identified relative to a particular coordinate system / frame. The reference coordinate system may be, for example, an image of the patient or part of the patient's anatomy.

[0050] An associated system control circuit may be utilized to detect and / or receive data from one or more markers / sensors 367. In some embodiments, positioning system 380 is a fiber optic tracking system or other positioning / location system. Some positioning systems / components are referred to herein as EM sensor systems / components to avoid listing a large number of sensors for each embodiment, although it should be emphasized that any positioning system may be used, including fiber optic sensor systems.

[0051] Each of the positioning systems / sources 350, 380 may define a separate frame of reference in which detectable markers may be located within the field of view of the respective system. This disclosure relates to relating (i.e., "registering") the coordinate system / frame of one positioning system (e.g., an EM positioning system) to another coordinate system / frame of another positioning system (e.g., a fluoroscopy system). Control circuitry associated with the system(s) may use various registration techniques to register different modalities with each other.

[0052] With respect to EM to fluoroscopy (or other positioning system) registration, EM sensor markers 367 are integrated into the tracked instrument (e.g., scope), located, and registered to the image space (e.g., fluoroscopy image) of the anatomical structure such that the position of the instrument is determined relative to the anatomical image space, and a positionally accurate representation of the instrument can be provided in the coordinate frame of the anatomical image. As the instrument moves through the patient, the marker tracking information can be used to update the position of the instrument relative to the anatomical structure / image such that a representation of the instrument can be displayed moving in real time in the anatomical image. Additionally, with the instrument and anatomical image provided in the same frame of reference, target anatomical structures can be identified in multiple images (e.g., fluoroscopy views) to identify the location of the target in three-dimensional (3D) space relative to the instrument.

[0053] In some embodiments, the tracked instrument or calibration structure is equipped with two five-degree-of-freedom (DOF) EM sensors / markers that, when positioned at a specific, known relative position / orientation, can collectively provide six degrees of freedom including position (x, y, z), heading (pitch and yaw), and roll orientation information. For example, two 5-DOF coils can be combined into a rigid assembly within the medical instrument or calibration structure with known sensor positions, and the two coils are configured to have different orientations of their axes of symmetry.

[0054] As referenced above, registration is the process that involves relating the frame of reference of a first positioning system / modality (e.g., an EM positioning system) to the frame of reference of a second positioning system / modality (e.g., fluoroscopy, optical / camera, or other imaging system). When the positions of two or more objects are known in the same frame of reference (i.e., mapped to the same coordinate system), the actual position of each object relative to one another can be ascertained. Thus, with this information, a user can drive or manipulate one of the objects relative to the other object.

[0055] In various embodiments, the EM field reference frame (e.g., frame 388) is aligned to the fluoroscopy reference frame (e.g., frame 358). For example, in some embodiments, the EM reference frame is measured relative to the fluoroscopy system reference frame. For example, in some embodiments, a sensing probe / structure 369 is used, which has an EM sensor and a radiopaque marker visible under fluoroscopy located at the same physical location on the probe and / or at a fixed location relative to the EM sensor. The sensing probe / structure can be placed within the field of view of the EM field generator and the fluoroscopy source. The two-dimensional position of the probe / structure can be specified by the user within the fluoroscopy field of view in images acquired at two different C-arm roll angles. In some implementations, the position of the probe is specified by the user at two or more different locations. These measurements can be used to synchronize the sensor position measurements with the selected fluoroscopy position. In this way, the EM coordinate system can be aligned to the fluoroscopy coordinate system.

[0056] 5 illustrates a first marker 365 detectable within the field of view 353 of a first positioning system 350 (e.g., fluoroscopy) and a second marker 367 detectable within the field of view 383 of a second positioning system 380. In some implementations, the marker(s) associated with either or both of the positioning systems / modalities may be mechanically fixed to a fixture 369, such as a fixture / probe coupled to a robotic arm or other structure within the surgical environment. In situations where the markers 365, 367 are not mechanically fixed to a common structure, the physical translation "X" between the markers will generally affect the alignment of the reference frames 358, 388.

[0057] In order to register different positioning modalities / systems with each other, it may be necessary to perform one or more transformations that account for differences in the physical locations and / or orientations of the sources 350, 380 and / or the respective markers 364, 363. For example, if one of the modalities 350, 380 provides a particular image of the surgical field that provides visual assistance to the operator, it may be necessary or desirable to determine where within such image / image space the detected markers associated with the other of the modalities 350, 380 reside as a means of merging the information provided by both modalities in a common positioning frame of reference. Registering multiple positioning modalities, such as electromagnetic sensing, fluoroscopy, ultrasound, etc., may allow the positioning spaces of each modality to be brought together, thereby allowing for the extraction of information from different modalities / systems and the fusion of such systems for use during navigation and / or other processes.

[0058] In general, the alignment of the modalities 350, 380 with respect to one another may be based on a physical position / orientation transformation between the source "Y" and / or marker "X" as shown in the illustrated diagram. Specifically, the alignment of the modalities 380, 350 may be based at least in part on the following equations / relationships: AX=YB (1) Here, "A" represents the physical transformation / relationship between the source 380 and the sensor / marker 367 of the positioning system 350, and "B" represents the physical transformation / relationship between the source 350 and the sensor / marker 365 of the positioning system 380. Thus, to determine alignment between the modalities 350, 380, it may be necessary to determine at least one of the physical transformations "X", "Y" to place the determined position of the marker within the coordinate space of the other positioning system / modality. Computing such transformations may be undesirably complex, time consuming, and / or resource intensive. The embodiments of the present disclosure provide a simplified alignment solution, where the need to calculate either or both of the physical transformations "X", "Y" is eliminated through the use of mechanical linkages between the sources 350, 380 and / or the markers 365, 367.

[0059] The term "registration" is used herein according to its broad and ordinary meaning and may refer to means, processes, and / or mechanisms for finding a coordinate transformation between two separate coordinate / reference frames. Similarly, the term "calibration" is used herein according to its broad and ordinary meaning and may refer to means, processes, and / or mechanisms for correcting and / or compensating for errors in the definition of a coordinate / reference frame.

[0060] External calibration according to embodiments of the present disclosure can include calibrating a frame of an imaging sensor relative to another reference / coordinate frame. Such calibration can be performed by aligning the frames with each other. Thus, alignment and external calibration can be considered part of the same process for imaging sources (e.g., camera-based, optical-based, fluoroscopy-based systems). "Alignment" can be used herein as a more general term than external calibration to describe the determination of a transformation between any two coordinate frames. With respect to the various embodiments and figures described herein, "alignment" can refer to the transformation "Y" described in detail herein. In some contexts, performing an alignment process can be referred to as external calibration.

[0061] In addition to the external calibration referenced above, it should be understood that if one of the positioning systems (e.g., system 350) is an imaging modality such as an optical imaging system, an internal calibration may further be required, for example, to compensate for distortions and any discrepancies regarding focal length and / or other parameters between manufacturing specifications and real-world conditions (e.g., actual focal length). The intrinsic calibration of the imaging modality may involve calibration of pixel coordinates and camera coordinates in an image frame. The embodiments of the present disclosure provide a solution to simplify or obviate aspects of extrinsic calibration, for example, regarding the determination of relative positions / or orientations between markers of the source and / or sensing modality, using mechanical linkages. Any of the markers 365, 367 and / or the positioning system sources 350, 380 may be considered a positioning device. That is, the positioning devices of the embodiments of the present disclosure may be positioning markers or sources, either alone or in combination, and / or structures associated therewith.

[0062] Mechanically-Based Alignment Solutions According to aspects of the present disclosure, alignment between multiple positioning modalities may be facilitated by implementing mechanical linkages between different positioning sources and / or markers. For example, by implementing a fixed / known physical relationship between sources of two different modalities / systems and / or markers of two different modalities / systems, either or both of the transformations "X" and "Y" as shown in FIG. 5 may be determined without further calculations, thereby simplifying the alignment equation AX=YB. That is, by implementing a known / fixed physical relationship between sources 350, 380 and / or markers 363, 364 for the equation AX=YB, either or both of the parameters "X", "Y" may be set to a known constant (e.g., a value, vector, matrix, etc.). For example, the value of "X" may be equal to a value of 1 (i.e., identity transformation) and thus ignored for purposes of calculation in implementations where sources and / or markers are co-located at a common physical location and / or at a location having a common center and / or coordinate frame. Thus, embodiments of the present disclosure can enable replacing algorithmic registration solutions with mechanical known / direct calculations.

[0063] Location marker integration As mentioned above, certain markers or sensors, referred to in some contexts herein as "fiducials", may be used for localization within a coordinate frame associated with a particular positioning modality. For example, such markers may be visible / identifiable within a field of view (e.g., electromagnetic field, camera image field, fluoroscopic x-ray field) due to the presence of certain properties / characteristics that are visible with respect to the respective modality. Markers that are detectable / identifiable with sufficient accuracy within a particular positioning space may be useful for completing a particular procedure and / or for assessing the accuracy of one or more modalities. Such markers may be used to create reference points (e.g., in terms of location and / or orientation) within a single image space, such as computed tomography (CT), optical camera space, ultrasound, or the like. An embodiment of the present disclosure provides multi-modal markers / fiducials that may be used to establish reference points within one or more image spaces, such as CT / fluoroscopic space and EM space.

[0064] When multi-modal markers are used in image spaces associated with two separate positioning modalities / systems, the alignment equation AX=YB described above as providing a general formula for aligning one image space to another, the transformation "X" (see FIG. 5), which represents the physical transformation between the markers detectable by the two separate modalities, can be set to a known constant or removed from the equation as equal to the sameness, leaving the equation AX=BY as containing only a single unknown variable / transformation "Y".

[0065] The embodiments of the present disclosure provide a marker that presents or indicates a reference point in one or more image spaces (i.e., the image spaces of two separate localization modalities) by providing a marker with a co-localized center point or other reference point for multiple modalities (e.g., CT / fluoroscopy and EM). Physically locating the center of the marker for multiple modalities can simplify the transformation between detectable markers for multiple modalities. If such a correspondence (transformation "X" in FIG. 5) is known, the "X" transformation can be treated as an identity transformation.

[0066] Combining markers of multiple modalities in a single marker structure / device to provide a unique, known correspondence between such markers can be achieved by implementing a unique size and / or shape for each marker / fiducial. For example, an ellipsoid or other shape with an asymmetric viewpoint that provides different viewing shapes from different viewing angles can be used to increase the amount of position information that can be derived from such markers.

[0067] 6 illustrates a system 600 including an arrangement of multiple positioning systems 650, 680 associated with integrated markers 665, 667. The system 600 includes an integrated marker 660 that comprises a mechanical part / component that provides a reference point in one or more modalities. For example, the integrated marker 660 may be visible in an image space associated with both a first field / space 650 and a second field / space 680. For example, the first modality 650 may comprise a camera imaging system, a fluoroscopic x-ray imaging system, or a computed tomography (CT) system, while the second modality 680 may comprise an electromagnetic field generator positioning system, as described in detail herein. The positioning system 650 may be associated with a coordinate frame 658, while the positioning system 680 may be associated with a different coordinate frame 688b.

[0068] In the system 600, the centers 601 of the marker frames 665, 667 are co-located to provide a common marker / frame 660. The mechanical design of the integrated marker 660 provides visibility in multiple modalities, such visibility indicating a common center point in the multiple modalities. Although the integrated marker 660 is shown as having a common center and / or frame of reference for both modalities, it should be understood that in some implementations, the integrated markers are not identical, but have center points and orientations offset from each other by a known distance and / or orientation such that the transformation "X" is a known constant between the two. The markers 665, 667 are combined by some physical linking structure or form, which may cause the coordinate centers of the multiple markers to overlap in three-dimensional space. For example, a physical inking structure may fix the markers 665, 667 to each other / together in a fixed relationship.

[0069] As used herein, an integrated marker combines marker elements visible in any combination of modalities. For example, FIG. 7 illustrates various combinations of markers 715 in integrated physical form. For reference, FIG. 7 also illustrates an exemplary non-integrated marker 705, each visible in a single image space. For example, marker 761 represents one embodiment of an EM visible marker including multiple conductive coils 762, 763 positioned at angled orientations relative to one another, such coils being detectable within an electromagnetic (EM) field generated by an EM field generator system and capable of providing orientation information such that position information derivable therefrom provides more than three or five degrees of freedom. For example, the orthogonal orientation of coils 762, 763 may provide six degrees of freedom in some cases.

[0070] Exemplary markers 764 include geometric shapes / forms (e.g., spherical beads) that may be visible in a particular imaging modality space. For example, the shapes / beads 764 may be radiopaque so as to be visible under radiation-based imaging modalities such as CT or fluoroscopy (e.g., X-ray), or may be reflective to certain sound wave signals and therefore visible under ultrasound examination. Alternatively, the shapes / beads 764 may be of a shape that is recognizable using a camera-based imaging modality. For example, the beads 764 may reflect infrared (IR) radiation / light to allow for its positioning. In some embodiments, the center of the geometric shapes / forms 764 is resolvable in a particular image space.

[0071] FIG. 7 further illustrates another exemplary marker 765 that may be visible in any positioning modality, such as EM positioning, comprising a patch or other fixture that may be placed in the image field for calibration, such as on a patient, an operating table, or other structure. The electromagnetic patch sensor 765 may be positioned in a coordinate frame that provides six degrees of freedom. Although the patch sensor 765 is described as an electromagnetic patch sensor, it may also be an optical marker that is detectable in the optical system image space. The optically visible marker may be configured such that infrared light / radiation emitted by an associated emitter of the system reflects off the surface of the marker such that it is detectable with respect to the position of the reflection. In some embodiments, the optical imaging source includes a stereoscopic camera.

[0072] FIG. 7 further illustrates various exemplary integrated markers 715, including a combined EM visible and CT, fluoroscopy, and / or camera visible geometric form 701 with an electromagnetic visible conductive coil 703 located at or near the center. An additional electromagnetic sensor coil 703 integrated within the geometric form 704 can be used to co-locate the electromagnetic sensor with the center of the geometric form. The geometric form 704 may be, for example, spherical or elliptical. Using different shapes for different markers can serve to enable such markers to be uniquely identifiable in images (e.g., camera, fluoroscopy, CT, etc.). Furthermore, the shape selection of the markers can indicate a unique match between the EM sensor and the form / bead with which they are combined in a particular embodiment. For example, in a system utilizing multiple EM sensor coils for positioning, each coil can be embedded or otherwise incorporated into an image visible form / bead having a different shape, thereby providing additional position correspondence information. Other examples for uniquely identifying image visible markers may include using different etches / patterns on the marker surface and / or using markers with different material properties that allow for unique identification.

[0073] If the resolvable centers of the EM sensor 703 and the visible form 704 are co-located and known, a reference point can be derived in multiple modalities. That is, the center of the marker 701 can be derived / detectable in both EM space and optical or other imaging space. In some cases, the integrated marker 715 may be visible in more than one image space. For example, the shape of the form 704 may be identifiable in optical space and the sensor coil 703 may be identifiable in EM space. Additionally or alternatively, the marker 701 may comprise one or more radiopaque markings / surfaces that indicate the center of the marker, such markings being identifiable in radiation-based imaging systems (e.g., fluoroscopy, CT) and / or sonic imaging systems (e.g., ultrasound). Co-location of markers for multiple modalities can simplify such inter-modality registration, as described in detail above. Thus, an identified marker in one space can simply be translated to a position in another space associated with a separate modality. Thus, the correspondence between the two imaging spaces can be determined mechanically and / or on the fly, while other solutions may require additional calibration / alignment steps.

[0074] The combined optical and EM space markers can be used to identify areas within the EM field volume that are subject to distortion by comparing EM measurements with optical or other imaging measurements, depending on the type of integrated marker implemented. That is, optical imaging may be more accurate than EM localization in some cases due to the potential presence of distortions in the generated electromagnetic field. Such distortions may be caused, for example, by the presence of a fluoroscopic C-arm or other metallic structures in the vicinity of the generated field. The integrated EM and optical markers may be placed within the EM and optical overlap field volume, and the movements of the markers may be tracked in both EM and optical space to identify areas of deviation between derived positions in optical and EM space as indicative of electromagnetic distortion. Such distortions, once known, may be accounted for and / or offset in the associated localization calculations. In some implementations, the integrated marker 701 may be constructed by drilling a hole or channel in the geometric form 704 and placing the EM sensor 703 therein in a position such that the center of the geometric form is co-located with the detectable fiducial of the EM sensor. Co-localizing markers for multiple modalities in accordance with aspects of the present disclosure can eliminate the need to construct coordinate frame transformations between markers for multiple modalities, thereby simplifying the registration of the modalities with each other.

[0075] FIG. 7 further shows an integrated marker 750 that includes an optically visible geometry 751 having a radiopaque surface 757 such that the marker 750 is visible in optical image space as well as radiation-based image space (e.g., fluoroscopy, CT).

[0076] As another example, the integrated marker 702 provides an EM visible structure that defines a coordinate frame, with the ends or other portions of each branch / axis of the coordinate frame including optically visible (or x-ray visible) beads / shapes 709 such that the marker 702 can be used for calibration in both EM and optical (or, for example, x-ray) imaging spaces.

[0077] With respect to registering two positioning modalities / systems with each other, embodiments of the present disclosure are generally described herein in the context of a positioning modality / system that provides three-dimensional position information regarding detected / visible markers. For example, in the case of an optical positioning modality, a camera-based imaging source may include multiple cameras that are angularly offset with respect to each other such that multiple images from alternating angles are captured from which three-dimensional position information can be derived. An electromagnetic field generator modality may similarly determine three-dimensional position information according to several solutions when a sensor / marker is detected within an electromagnetic field volume. That is, for some modalities, three-dimensional position information is derivable from a single source position / orientation. However, for some imaging modalities, three-dimensional position information may not be derivable from a single angle. For example, a single camera optical system and / or an X-ray / fluoroscopy system may be configured to generate only two-dimensional position information from a given position / orientation of the imaging source (e.g., camera, X-ray emitter).

[0078] 8 illustrates an electromagnetic (EM) 89 and fluoroscopy 74 positioning system associated with an integrated marker 860, according to one or more embodiments, where repositioning of the fluoroscopy source 74 may be implemented as a means to generate three-dimensional position information of the integrated / co-localized marker 860 within the fluoroscopy imaging space. For example, it may be necessary to acquire two or more fluoroscopy / x-ray images to provide sufficient data to triangulate a visible marker (e.g., a radiopaque fiducial) in three-dimensional space using fluoroscopy.

[0079] FIG. 8 illustrates a first position 70-1 of a fluoroscopy source 74, from which imaging of the marker 860 may be implemented to obtain a first image and imaging field of the detected marker 860. The C-arm 71 may then be rotated about an axis or otherwise moved in an orientation configured to generate an imaging field including the marker 860, with the adjusted position 70-2 of the C-arm 71 directed toward and angularly offset θ from the previous position 70-1. Additional images may be generated from a second position 70-2 to provide multiple fluoroscopic images of the marker 860 from multiple angles / orientations. The angular separation between the first position 70-1 and the second position 70-2 of the fluoroscopy source 74 may be any angular separation. In some implementations, a separation of at least 15° (θ) is implemented to provide information from which three-dimensional position information may be derived. In some implementations, a 90° angle rotation / translation is implemented to provide an orthogonal image, which may advantageously provide maximum data for 3D positioning.

[0080] Using an identifiable integrated marker 860 with a common or known transformation "X" for the two modalities, a transformation "Y" between the positioning systems / sources 74, 89 can be determined based on a three-dimensional position determination "F" in fluoroscopic image space based on multiple fluoroscopic images / positions, as well as a three-dimensional position determination "E" in EM space based on detection of the marker 860 in an EM field generated from a single position of the EM field generator 89.

[0081] Figure 9 illustrates an electromagnetic (EM) 989 and robot 910 positioning system associated with an integrated marker(s), in accordance with one or more embodiments. The robot system 910 also includes a robot arm 912. Figure 9 illustrates two exemplary positions 912a, 912b of the robot arm 912 and associated components. The following description can be understood with respect to either position.

[0082] The distal end 922 of the robot arm 912 may include an end effector component 922 that may be configured to manipulate certain robotic instrumentation, as described in detail herein. Positioning of the distal end 922 of the robot arm 912 may be implemented by actuating certain motors or other actuators to move and / or articulate various joints and linkages of the robot arm 912 and / or its associated carriage to a desired position / pose. That is, robot data indicating the current configuration / position of the various motor / actuator components of the robot system 910 may indicate the position of the end effector 922 and / or other part(s) of the robot arm / system. Thus, the coordinate frame / space of the robot system 910 may define a coordinate frame 966, in which the end effector 922 and / or other part(s) of the robot system are positioned relative to, for example, a base of the robot system.

[0083] To transform position information based on robot data to a coordinate frame space associated with a secondary modality, such as an EM field generator space or other positioning / imaging modality disclosed herein, and / or vice versa, alignment between the robot space and the secondary modality may be necessary, and such alignment may be conceptually similar to alignment between imaging / positioning modalities as described above. For example, if the position of the end effector and / or points thereon (and / or other part(s) of the robot system) is based on robot data (represented by position transformation "A" in FIG. 9), the "marker" on which position "A" is based may be considered to be the end effector 922 and / or other position indicated by the robot data. With respect to a marker / fiducial associated with a secondary modality (e.g., an EM position sensor), position vector "B" may be the position of the associated EM sensor / marker 960 relative to the field generator 989, as shown. That is, the electromagnetic sensor 960 represents the marker as seen in the image space of the field generator 989.

[0084] To simplify the alignment process by fixing the physical relationship between the markers / fiducials 922, 960 of multiple positioning systems / frames, similar to other embodiments disclosed herein, a physical coupling or co-location of the EM sensor 960 with the tracked portion of the robot arm (e.g., end effector 922) may be implemented to set the transformation "X" between the markers of the two systems to a known constant transformation and / or identity transformation (e.g., the co-location of the markers). Thus, embodiments of the present disclosure may provide for positioning of an EM sensor or other marker associated with an imaging modality on or coupled to the robot end effector 922 such that the marker 960 is co-located (or in a fixed relationship) with the end effector 922, thereby simplifying alignment between the robot space and the EM image space. For example, the EM sensor 960 may be physically placed at a specific location on or in the end effector or other portion of the robot arm 912 such that its center is co-located with a location / point trackable using the robot data. Alternatively, a fixture 940 or other rigid structure may couple the sensor 960 to the end effector 922 or other portion of the robotic system such that a constant, fixed position transformation couples the sensor 960 to the robot marker 922. The fixture 940 may be utilized for marker placement as part of a calibration process.

[0085] As mentioned above, with reference to FIG. 5, the alignment of the two modalities can be based on the relationship AX=YB. The above description demonstrates that by creating a fixed physical linkage / relationship between markers associated with multiple positioning modalities, the alignment between the modalities can be simplified by eliminating the need to calculate the transformation "X". Although the above disclosed embodiments are presented in the context of aligning separate positioning / imaging modalities that include detection of markers in a field of view, the disclosed solutions and concepts related thereto can be implemented in the context of aligning a field of view positioning / imaging modality to a robot positioning space based on robot pose data. By mechanically defining the transformation between the robot space and the electromagnetic (EM) field space (or other positioning / imaging space), the markers detected through field of view positioning / imaging (e.g., EM positioning) can be correlated with the robot space without requiring additional alignment steps and / or additional workflow steps to determine the transformation "X" between the fiducials / markers of the associated system.

[0086] Fixing the positioning system source The intrinsic and imaging systems may be characterized for alignment purposes. For example, to integrate fluoroscopic imaging into a robotic system such as the robotic systems of Figs. 1-3, an initial intrinsic calibration of the fluoroscopic system may be implemented, such as by using a reference fixture / probe that is placed in the image space of the fluoroscopic system to calibrate the fluoroscopic system. For example, a checkerboard or other pattern may be printed on the fixture, and imaging of the fixture using the fluoroscopic system may be used to compensate for distortions and / or focal distance. Distortions in the pattern of the reference fixture / structure may indicate distortions in the radiation field. Fluoroscopic image calibration to correct for artifact distortions may be implemented to account for pincushion and / or other distorting effects that may be present in some fluoroscopic environments. Making experimental measurements of the associated electric field deviations can account for distortions caused by the proximity of the metal C-arm of the fluoroscopic system to the electromagnetic field generator.

[0087] Referring back to FIG. 5, alignment between multiple positioning systems can be based on the relationship AX=YB, where "X" represents the physical relationship between the tracked markers / fiducials of each modality, while the transformation "Y" represents the relationship between the sources of each image space / modality. The embodiments disclosed above demonstrate how mechanical linkages between the markers / fiducials of multiple modalities can simplify alignment between the respective coordinate frames. Similarly, as described herein, mechanical linkages between sources of different modalities can be implemented to simplify multi-modality alignment. That is, with respect to the paradigm illustrated in FIG. 5 and described in detail herein, the transformation "Y" can be mechanically fixed, thereby setting the transformation "Y" to a known constant transformation based on a particular mechanical fixation of the imaging source. When an imaging modality (e.g., a camera-based, optical-based, fluoroscopy-based system) is utilized, fixing the "Y" transformation may be insufficient alone to transform information or data from another modality's frame of reference to the imaging modality's frame of reference. For example, it may be further necessary or desirable to implement an intrinsic calibration of the imaging modality in order to obtain the correct pixel coordinates for implementing the transformation between reference frames.

[0088] Figure 10 illustrates a multiple positioning system 170, 189 with a mechanically fixed source in accordance with one or more embodiments. Although Figure 10 illustrates the fixation of an electromagnetic (EM) field generator 189 to a fluoroscopy mounting arm 171 or other structure that holds / supports the source 174 (e.g., a metal can structure), it should be understood that the concepts disclosed in connection with Figure 10 are applicable to fixation of sources of any type of positioning / imaging modality, and the particular embodiment illustrated in Figure 10 is presented for convenience in order to demonstrate the inventive concepts related thereto.

[0089] Referring back to Figure 5, a common problem associated with aligning the reference frames of two modalities with one another is the determination of the relative physical position / transform (identified as transformation "Y" in Figure 5) between the position / orientation of the source of the first modality / system and the position / orientation of the source of the second modality / system. In the system of Figure 10, the EM field generator 189 is mechanically fixed to the fluoroscopy source / receiver 174 and / or C-arm structure 171 via a rigid (e.g., non-adjustable / bendable) mounting arm / structure 150. Another exemplary fixed mounting to the C-arm structure 171 is shown as the field generator 187 mounted to the mid-section 606 of the C-arm 171 via a rigid mounting arm 151.

[0090] In the illustrated embodiment, the field generator 189 is rigidly attached to the structure of the fluoroscopy source / system 170, thereby mechanically defining the physical relationship between the two imaging system sources 189, 174. The C-arms and the fluoroscopy images they are used to generate provide a visualization modality suitable for intraoperative procedures. EM-based devices / tools can also be used in such settings. It can be difficult to align these two modalities in a practical manner, and the embodiments disclosed herein can simplify such alignment. Once the fluoroscopy image space and the EM image space are aligned, the EM detectable tools / markers 178 can be mapped to the fluoroscopy image. Furthermore, once aligned, navigation algorithms implemented to direct surgical instrumentation can utilize the alignment, such as through operator guidance and / or algorithmic processes.

[0091] In some embodiments, an EM field generator having a relatively small form factor may be implemented as a means for physically attaching the EM field generator to the fluoroscopy structure 171. That is, field generator 189 may comprise a miniature field generator that is smaller in size than the exemplary embodiment of field generator 67 shown in Figure 1. A rigid connection between field generator 189 and the fluoroscopy system structure 171 may simplify alignment of the EM system with the fluoroscopy system.

[0092] Establishing a physical relationship between the EM field generator 189 and the fluoroscopy emitter 175 and / or receiver 174 can be accomplished by mounting both units / devices to the same support structure (e.g., C-arm 171). With the relationship "Y" defined between the EM field 602 and the fluoroscopy field 601, at least two fluoroscopic images can be generated to provide three-dimensional position data in the fluoroscopy image space, allowing registration to be completed between the two modalities. The electromagnetic field position data may be relied upon from a single physical position of the field generator 189 or from two positions associated with the two fluoroscopic images. That is, since the transformation between the physical positions / orientations of the field generator 189 and the fluoroscopy system 174 is known, when the fluoroscopy system is moved to acquire multiple images for three-dimensional position determination, additional position information may be utilized for the EM field generator system as well, which may provide additional data points for position determination in the electromagnetic field space.

[0093] Physically mounting the field generator 189 to the C-arm 171 may result in distortion of the fluoroscopy field 601 and / or the electromagnetic field 602 due to the physical proximity between such sources and / or to the metal structure 171. In some implementations, a calibration may be performed to compensate for determined static distortion resulting from the physical location of the arm 171 on the electromagnetic field 602 relative to the electrical conductivity and / or other properties of the structure 171. In some embodiments, the mounting structure / arm 150 may be long enough to avoid the risk of problematic distortion due to the components being in close proximity to one another. For example, undesired radiation may be caused by the structural arm 171 interfering with the electromagnetic field 602 of the field generator 189 when the structure 171 is positioned within 8 inches or 10 inches of the EM field generator 189. It may therefore be desirable to construct the rigid mounting / coupling arm 150 to position the field generator 189 at least such a distance away from the fluoroscopy source 174 and / or the structural support arm 171. In such a case, distortion resulting from interference by arm 171 may be relatively minimal such that such distortion may be compensated for by the system control circuitry. Further, in some implementations, the system may be configured to compensate for the distortion profile of fluoroscopy field 601 relative to EM field 602.

[0094] If multimodal fiducials / markers as described above are not utilized, the estimation of the transformation "X" between the fiducials / markers of the two positioning systems / modalities can be performed by adjusting the position of the source of the positioning modality. For example, positioning data can be generated from a first position of the C-arm 171 and fixed EM field generators 189, 187, after which the C-arm 171 can be positioned in a second position that may be angled differently from the first position (e.g., lateral repositioning 20°). Another fluoroscopic image can then be generated while collecting EM data. The difference in reading / sensing at the two positions relative to the fixed mechanical relationship between the positioning sources can be used to derive the relative position of the markers / fiducials. A single repositioning such that two separate fluoroscopic images and associated EM data are collected may be sufficient for alignment. However, it may be desirable to generate fluoroscopic images and associated electromagnetic sensor data at more unique C-arm positions to obtain higher fidelity alignment, although such additional processing may result in increased workflow time.

[0095] The fluoroscopic image may undergo specific processing by the system control circuitry to identify each radiopaque marker and their respective position / orientation in the fluoroscopic image space. The relative transformation "X" between the markers / reference points of the two systems may be solved from the process data using hand-eye calibration, point cloud singular value decomposition (SVD), or other processes. With the field generator 189 mounted on the C-arm 170, the transformation between the field generator 189 and the fluoroscopic source 174 is mechanically defined, and thus the detected 3D pose of the instrument can be directly superimposed on the fluoroscopic image without performing point cloud registration or other registration processes.

[0096] Although fluoroscopy and EM field positioning modalities are shown in FIG. 10, similar alignment concepts may be implemented to align fluoroscopy to an optical vision system, or an optical vision system to an electromagnetic sensor system. For example, a rigid physical coupling between the EM field generator 189 and a structure associated with the optical camera imaging source may be used to fix the transformation "Y" between such modalities. In some camera systems, it may not be necessary to acquire multiple images from multiple positions / angles to derive three-dimensional position information. For example, a stereoscopic camera imaging system may be configured to determine three-dimensional position information from a single physical position / orientation of the source device.

[0097] FIG. 11 illustrates an electromagnetic (EM) 789 and robot 710 positioning system associated with a fixed positioning system source, according to one or more embodiments. As mentioned above, with reference to FIG. 5, the alignment of the two modalities can be based on the equation AX=YB, where "B" represents the determined relative position between a first positioning source (e.g., an EM field generator) and a marker detected / located in the frame of reference of the first positioning source. "A" represents the determined relative position between a second positioning system source (e.g., a fluoroscopy detector, a robotic system base, etc.) and a marker detected / located in the frame of reference of the second positioning source. "X" represents the physical transformation between the markers of the first system and the markers of the second system, and "Y" represents the physical transformation between the sources of the first system and the sources of the second system.

[0098] The above description demonstrates that by creating a fixed physical linkage / relationship between the positioning system sources associated with multiple positioning modalities, the alignment between the modalities can be simplified by eliminating the need to calculate the transformation "Y". Although the above disclosed embodiments are presented in the context of aligning separate positioning / imaging modalities that include detection of markers in a field of view, the disclosed solutions and concepts related thereto can be implemented in the context of aligning a field of view positioning / imaging modality to a robot positioning space based on robot pose data. By mechanically defining the transformation between the robot space and the electromagnetic (EM) field space (or other positioning / imaging space), the markers detected through field of view positioning / imaging (e.g., EM positioning) can be correlated with the robot space without requiring additional alignment steps and / or additional workflow steps to determine the transformation "Y" between the sources of the associated systems.

[0099] FIG. 11 illustrates a robotic system 710 including a robotic arm 712. The distal end 722 of the robotic arm 712 may include an end effector component 722 that may be configured to manipulate certain robotic instrumentation, as described in detail herein. Positioning of the distal end 722 of the robotic arm 712 may be implemented by actuating certain motors or other actuators to move and / or articulate various joints and linkages of the robotic arm 712 and / or its associated carriage to a desired position / pose. That is, robot data indicating the current configuration / position of the various motor / actuator components of the robotic system 710 may indicate the position of the end effector 722 and / or other part(s) of the robotic arm / system. Thus, the imaging space of the robotic system 710 may define a coordinate frame 766, in which the end effector 722 and / or other part(s) of the robotic system are positioned relative to, for example, a base of the robotic system. In general, the identified position of the end effector 722, which is determined / controlled based on robot data rather than an absolute position, may be within a position space relative to the physical positioning of the robot system 710, which may be ambulatory in some cases.

[0100] To transform position information based on robot data to an image space associated with a secondary modality, such as an EM field generator space or other positioning / imaging modality disclosed herein, and / or vice versa, alignment between the robot space and the secondary modality may be necessary, and such alignment may be conceptually similar to alignment between imaging / positioning modalities as described above. For example, if the position of the end effector 722 (and / or other portion(s) of the robot system 710) is based on the robot data (represented by position transformation "A" in FIG. 11), the "marker" on which position "B" is based may be considered the end effector 722 and / or other portion of the robot arm 712 indicated by the robot data. With respect to a marker / fiducial associated with a secondary modality (e.g., an EM position sensor), position vector "B" may be the position of the associated EM sensor / marker 760 relative to the field generator 789, as shown. That is, the electromagnetic sensor 760 represents the marker that is visible in the image space of the field generator 789.

[0101] As referenced, the robot end effector 722 may be considered as a relevant reference / marker for the robot system 710, and its position "A" may be measured using robot data (e.g., forward kinematic data) relative to the robot base 710. To simplify the alignment process by partially fixing the physical relationship between the sources 789, 710 of the multiple positioning systems / frames, as in other embodiments disclosed herein, a physical link between the EM field generator 789 and the robot system 710 may be implemented to determine the transformation "Y" between the two systems to be equal to a known constant transformation with respect to the position "A", which may be determined according to the robot data indicating the position of the instrumentation 722 in the robot space 766. For example, the EM source 789 may be fixed to the robot arm 712 such that the source 789 is fixed at a relative position transformation "V" with respect to the reference point 722 of the robot arm 712. Thus, the transformation "Y" between the two positioning system sources 789, 710 may be considered to be equal to the combined vectors "B" and "V". That is, since the illustrated transformation "V" is mechanically defined, "Y" (i.e., the alignment between the EM coordinate frame and the robot coordinate frame) can be known by fusing the robot (e.g., kinematic) data with the known relationship "V" according to the relationship / equation Y=AV. Thus, a separate alignment step is not required and the EM sensor space can be directly transformed / translated into the robot space. Note that in the implementation of FIG. 11, the field generator 789 is fixed relative to the end effector 722, not directly to the robot base 710. "Y" is determined / known based on the mechanical linkage "V", rather than being strictly mechanically determined.

[0102] The fixed relationship / transformation "V" may be achieved by mounting the EM field generator 789 to a fixture / tool ​​740 that is coupled to the robot arm 712. By implementing a fixed coupling "V", embodiments of the present disclosure may provide simplification of alignment between the robot space and the EM image space. The fixture 740 may be utilized as part of a calibration process to align reference frames with one another prior to or during a surgical procedure.

[0103] 11 is implemented, the "X" transformation relationship between the positions of the markers / fiducials 760, 722 of the two systems can also be known / determined without additional process steps. For example, with respect to the vector transformations identified in FIG. 11, the transformation "X" can be considered to be equal to a position "A" in the EM field space combined with a mechanically fixed vector "V".

[0104] As mentioned above, with reference to FIG. 5, the alignment of the two modalities can be based on the equation AX=YB. The above discussion demonstrates that by creating a fixed physical linkage / relationship between sources associated with multiple positioning modalities, determining the relationship between the markers / fiducials of the two modalities is simplified by avoiding the need to calculate a transformation "Y". As referenced above, the transformation "Y" can be based on a fixed physical transformation "V". However, creating a precise fixed transformation "V" between the end effector 722 and the EM field generator 789 can be non-trivial and / or difficult to implement in some cases. For example, in the example illustrated in FIG. 11, the transformation "V" is mechanically fixed using a fixture 740 coupled to the end effector 722. However, the process of achieving a precise fixed mechanical relationship "V" using a fixture or other linkage structure can be relatively tedious and / or prone to error in some cases. For example, if the known physical specifications of the associated components / assemblies are not precisely accurate when installed, the expected mechanical definition of the transform "V" may be off by some amount (e.g., due to manufacturing variations, mechanical slop / clearances, etc.). It may therefore be desirable to implement alternatives to simplify the process of determining the transform "X" with respect to the EM-robot spatial alignment.

[0105] 11, the unknown transformations "X" and "Y" can be estimated in a simplified manner based on the movement of the robot arm (i.e., changing "A") in two or more poses. The first position 712a may be understood to represent an initial position, while the position 712b may be understood to represent a subsequent position after the movement of the arm 712.

[0106] For the system configuration in Fig. 11, posture estimation is performed based on the change in the position of the robot arm reference point B Δ and the resulting change in position of the EM sensor 760, A ΔFor example, the following algorithm may be implemented to perform pose estimation, and the calibration / estimation of the pose / position of the EM sensor 760 relative to the robotic system reference 722 is as follows:

[0107]

number

[0108] 12 illustrates a three-dimensional position overlay of an instrument representation 291 on a two-dimensional image 295, in accordance with one or more embodiments. FIG. 12 provides an exemplary demonstration of how electromagnetic (EM) sensors can be used to track medical instruments and localize them relative to a reference image using a registration process that leverages fixed mechanical linkages as described in detail herein. That is, embodiments of the present disclosure advantageously provide a mechanism for projecting an object pose known or identified in one image space, such as EM field positioning space, optical space, onto a two-dimensional fluoroscopy / x-ray space.

[0109] 12 shows a generated fluoroscopic image 295 that includes certain instrumentation 292 (e.g., scope, needle, etc.) in the image. An overlay process can be performed to overlay the instrument(s) 291 onto the captured fluoroscopic image. The system control circuitry can be utilized to identify and display the location of the instrument 291 relative to the patient's anatomy depicted in the image 295.

[0110] The tracked instrument 291 can be simulated by rendering it in three-dimensional computer graphics and displaying, overlaying, or superimposing it on the fluoroscopic image 295. The current location and orientation of the tracking sensors / markers is known relative to the fluoroscopic image space based on alignment with EM space facilitated by a mechanical linkage as described herein. The instrument 291 includes one or more EM sensors / markers that allow alignment from EM space to fluoroscopic space. From these known data points, a virtual instrument 921 can be drawn / generated for representation. Robotic data for the robotic end effector / system used to control the instrument 291 can also be tracked, and this motion can be used to extrapolate the instrument shape between sensor locations. The rotational orientation of the instrument 291 may also be determined from the sensors as described above to provide an overall three-dimensional reconstruction of the instrument.

[0111] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the algorithms or processes described herein may be performed in a different order, added, merged, or omitted entirely, and thus, in a particular embodiment, not all of the described acts or events are necessary to the execution of a process.

[0112] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not intended to imply that features, elements, and / or steps are generally required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like, are synonymous and used in their ordinary sense and are used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, each be present.

[0113] In the above description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. However, the method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Thus, it is intended that the scope of the invention disclosed herein and claimed below should not be limited by the specific embodiments described above, but should be determined solely by a fair reading of the following claims.

[0114] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from another element having a similar or identical name (apart from the use of the ordinal term). In addition, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0115] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0116] Spatially relative terms such as "outer", "inner", "upper", "lower", "below", "upper", "vertical", "horizontal", and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the drawings. It should be understood that the spatially relative terms are intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the drawings. For example, if the devices shown in the drawings are inverted, a device positioned "below" or "under" another device may be positioned "above" the other device. Thus, the illustrative term "lower" may include both lower and upper positions. Devices may also be oriented in other directions, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0117] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," and the like are intended to encompass the notion of equality. For example, "less" can mean not only "less than" in the strict mathematical sense, but also "less than or equal to."

[0118] [Embodiment] (1) A positioning system comprising: A group of positioning devices, a first device comprising a first positioning source associated with a first positioning modality, the first positioning source configured to view a first field; a second device comprising a second positioning source associated with a second positioning modality that is of a different type than the first positioning modality, the second positioning source configured to view a second field; a third device comprising one or more first markers detectable within the first field using the first positioning modality; a fourth device comprising one or more second markers detectable in the second field using the second positioning modality; and a coupling structure that physically couples two of the group of positioning devices to each other in a fixed, precise relative position and orientation. (2) A positioning system as described in embodiment 1, wherein the connecting structure comprises a rigid mounting arm that attaches the second device to the first device. (3) the first device comprises a movable C-arm; A positioning system as described in embodiment 2, wherein the first positioning source includes at least one of an X-ray generator or an X-ray detector. (4) The positioning system of embodiment 3, wherein the mounting arm mounts the second device to the C-arm of the first device. (5) A positioning system as described in embodiment 3, wherein the mounting arm mounts the second device to a structure of the first positioning source.

[0119] (6) A positioning system described in any one of embodiments 3 to 5, wherein the second device is equipped with an electromagnetic field generator. (7) The positioning system of claim 6, wherein the coupling structure attaches the second device to the first device so as to maintain the electromagnetic field generator at least 20.32 cm (8 inches) away from the movable C-arm and the first positioning source. (8) The positioning system of embodiment 1, wherein each of the first positioning source and the second positioning source is one of the group consisting of a camera device, an X-ray emitter, an X-ray detector, an ultrasound emitter, an ultrasound detector, and an electromagnetic field generator. (9) A positioning system according to any one of embodiments 1 to 5, 7, or 8, wherein the connecting structure fixes the third device to the fourth device. (10) A positioning system as described in embodiment 9, wherein the one or more first markers define a first central position that is held in a co-localized position with a second central position defined by the one or more second markers by the connecting structure.

[0120] (11) A positioning system as described in embodiment 1, wherein the first field at least partially overlaps with the second field. (12) A positioning marker assembly, comprising: one or more first markers configured to be detected using a first localization modality; one or more second markers configured to be detected using a second modality; and and a physical coupling structure that fixes the one or more first markers and the one or more second markers in a fixed relative position and orientation. (13) A positioning marker assembly as described in embodiment 12, wherein the physical connection structure fixes the one or more first markers and the one or more second markers in a colocalized configuration. (14) A positioning marker assembly as described in embodiment 13, wherein in the colocalization configuration, the one or more first markers define a first coordinate center that substantially overlaps with a second coordinate center defined by the one or more second markers. (15) The one or more first markers include one or more conductive coils; A positioning marker assembly as described in embodiment 12, wherein the one or more second markers include one or more radiopaque surfaces.

[0121] (16) The first positioning modality is an optical, ultrasound, or X-ray imaging modality; the one or more first markers include a bead form that is visible using the first localization modality; A positioning marker assembly described in any of embodiments 12 to 15, wherein the one or more second markers include one or more conductive coils at least partially embedded within the bead form of the one or more first markers. (17) A positioning system comprising: a first emitter associated with a first localization modality; a second emitter associated with a second localization modality of a different type than the first localization modality; a rigid linkage arm that physically holds the first emitter in a fixed, precise relative position and orientation with respect to the second emitter. (18) The first emitter is an electromagnetic field emitter; The positioning system of embodiment 17, wherein the second emitter is an X-ray emitter. (19) The positioning system of embodiment 18, further comprising a C-arm configured to support the second emitter. (20) The positioning system of embodiment 19, wherein the connecting arm physically connects the first emitter to the C-arm.

[0122] (21) The positioning system of claim 19, wherein the articulating arm is coupled to an intermediate portion of the C-arm. (22) A positioning system according to any one of embodiments 17 to 21, wherein the connecting arm physically couples the first emitter to a structure associated with the second emitter.

Claims

1. 1. A positioning system comprising: A group of positioning devices, a first device comprising a first positioning source associated with a first positioning modality, the first positioning source configured to view a first field; a second device comprising a second positioning source associated with a second positioning modality that is of a different type than the first positioning modality, the second positioning source configured to view a second field; a third device comprising one or more first markers detectable within the first field using the first localization modality; and a fourth device comprising one or more second markers detectable in the second field using the second positioning modality; and a coupling structure that physically couples two of the group of positioning devices to each other in a fixed, precise relative position and orientation.

2. The positioning system of claim 1 , wherein the coupling structure comprises a rigid mounting arm that attaches the second device to the first device.

3. the first device comprises a movable C-arm; The positioning system of claim 2 , wherein the first positioning source includes at least one of an X-ray generator or an X-ray detector.

4. The positioning system of claim 3 , wherein the mounting arm mounts the second device to the C-arm of the first device.

5. The positioning system of claim 3 , wherein the mounting arm mounts the second device to a structure of the first positioning source.

6. The positioning system according to any one of claims 3 to 5, wherein the second device comprises an electromagnetic field generator.

7. 7. The positioning system of claim 6, wherein the coupling structure attaches the second device to the first device to maintain the electromagnetic field generator at least 8 inches away from the movable C-arm and the first positioning source.

8. 2. The positioning system of claim 1, wherein each of the first positioning source and the second positioning source is one of the group consisting of a camera device, an X-ray emitter, an X-ray detector, an ultrasound emitter, an ultrasound detector, and an electromagnetic field generator.

9. The positioning system of claim 1 , wherein the coupling structure secures the third device to the fourth device.

10. The positioning system of claim 9 , wherein the one or more first markers define a first central location that is held in a co-localized position with a second central location defined by the one or more second markers by the connecting structure.

11. The location system of claim 1 , wherein the first field at least partially overlaps with the second field.

12. 1. A positioning marker assembly comprising: one or more first markers configured to be detected using a first localization modality; one or more second markers configured to be detected using a second modality; and and a physical coupling structure that fixes the one or more first markers and the one or more second markers in a fixed relative position and orientation.

13. The localization marker assembly of claim 12 , wherein the physical linking structure secures the one or more first markers and the one or more second markers in a co-localized configuration.

14. 14. The positioning marker assembly of claim 13, wherein in the co-localized configuration, the one or more first markers define a first coordinate center that substantially overlaps with a second coordinate center defined by the one or more second markers.

15. the one or more first markers comprise one or more conductive coils; The positioning marker assembly of claim 12 , wherein the one or more second markers include one or more radiopaque surfaces.

16. the first localization modality is an optical, ultrasound, or x-ray imaging modality; the one or more first markers include a bead form that is visible using the first localization modality; A positioning marker assembly according to any one of claims 12 to 15, wherein the one or more second markers comprise one or more conductive coils at least partially embedded within the bead form of the one or more first markers.

17. 1. A positioning system comprising: a first emitter associated with a first localization modality; a second emitter associated with a second localization modality of a different type than the first localization modality; a rigid linkage arm that physically holds the first emitter in a fixed, precise relative position and orientation with respect to the second emitter.

18. the first emitter is an electromagnetic field emitter; The positioning system of claim 17 , wherein the second emitter is an X-ray emitter.

19. 20. The positioning system of claim 18, further comprising a C-arm configured to support the second emitter.

20. 20. The positioning system of claim 19, wherein the linkage arm physically couples the first emitter to the C-arm.

21. 20. The positioning system of claim 19, wherein the articulated arm is coupled to an intermediate portion of the C-arm.

22. The positioning system of any one of claims 17 to 21, wherein the linking arm physically couples the first emitter to a structure associated with the second emitter.

Images