Navigation updates for medical systems

Abstract

This disclosure provides methods, devices, and systems for planning and performing medical procedures. The present implementations more specifically relate to techniques for navigating an instrument to a target within an anatomy. In some aspects, a controller for a medical system may generate a graphical interface depicting a spatial relationship between the instrument and the target and update the graphical interface to depict an updated spatial relationship between the instrument and the target based on sensor data received via a sensor disposed on the instrument and image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy. More specifically, the controller may determine a mapping between a sensor space and an image space based on the sensor data and the image data and may determine the updated spatial relationship based on the mapping between the sensor space and the image space.

Description

AUR6338WOPCT1 PCT NAVIGATION UPDATES FOR MEDICAL SYSTEMS CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This patent application claims priority to US Non-Provisional Patent Application No.19 / 006,086, filed December 30, 2024, and entitled “REGISTRATION OF IMAGING SYSTEM WITH SENSOR SYSTEM FOR INSTRUMENT NAVIGATION,” which claims priority and benefit under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No.63 / 571,768, filed March 29, 2024, and entitled “REGISTRATION OF IMAGING SYSTEM WITH SENSOR SYSTEM FOR INSTRUMENT NAVIGATION.” This patent application further claims priority to US Non-Provisional Patent Application No. 19 / 034,431, filed January 22, 2025, and entitled “NAVIGATION UPDATES FOR MEDICAL SYSTEMS,” which claims priority and benefit under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No.63 / 571,909, filed March 29, 2024, and entitled “INTRAOPERATIVE NAVIGATION UPDATES FOR MEDICAL SYSTEMS. The disclosures of the prior Applications are considered part of and are incorporated by reference in this Patent Application. TECHNICAL FIELD

[0002] This disclosure relates generally to medical systems, and specifically to intraoperative navigation updates for medical systems. DESCRIPTION OF RELATED ART

[0003] Many medical procedures include steps that can be performed pre-operation (also referred to as a “preoperative phase”), intra-operation (also referred to as an “intraoperative phase”), or post-operation (also referred to as a “postoperative phase”). For example, during a preoperative phase, an imaging system may be used to scan or otherwise capture images or video of a patient’s anatomy. Example suitable imaging technologies include computed tomography (CT), X-ray, fluoroscopy, positron emission tomography (PET), PET-CT, CT angiography, cone beam CT (CBCT), three-dimensional rotational angiography (3DRA), single-photon emission CT (SPECT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, among other examples. The images may be used, during an intraoperative phase, to help guide or navigate a medical instrument to a target (also referred to as a “treatment site”) within the patient’s anatomy. However, images acquired during a preoperative phase may not accurately reflect a spatialAUR6338WOPCT1 PCT relationship between the medical instrument and the target during an intraoperative phase. For example, among various other factors, preoperative images are often acquired several days (or even weeks) before the intraoperative phase, such that changes in the patient’s anatomy may cause deviations in the spatial positioning of the target. Thus, there is a need to provide more accurate information about the spatial relationship between the medical instrument and the target during the intraoperative phase. SUMMARY

[0004] This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to generate a graphical interface depicting a spatial relationship between the instrument and a target within the anatomy; receive first image data captured by a first imaging system external to the anatomy while the instrument is disposed within the anatomy; receive first sensor data via one or more sensors associated with a sensor system, where the one or more sensors include at least a first sensor disposed on the instrument; determine a mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system based at least in part on the first image data and the first sensor data; and update the graphical interface to depict an updated spatial relationship between the instrument and the target based at least in part on the mapping between the first coordinate space and the second coordinate space.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to receive sensor data from a sensor disposed on an instrument within an anatomy; determine a position of the instrument in a first coordinate space based on the received sensor data; receive image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy; determine a position of the instrument in a second coordinate space based on the received image data; and determine aAUR6338WOPCT1 PCT mapping between the first coordinate space and the second coordinate space based at least in part on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The present implementations are illustrated by way of example and are not intended to be limited by the figures of the accompanying drawings.

[0008] FIG.1 shows an example medical system, according to some implementations.

[0009] FIG.2 shows example components of the control system and the robotic system of FIG.1, according to some implementations.

[0010] FIG.3 shows a block diagram of an example localization system, according to some implementations.

[0011] FIG.4 shows a block diagram of an example navigation system, according to some implementations.

[0012] FIG.5 shows an example spatial relationship between a skeleton image space, a sensor space, and an intraoperative image space.

[0013] FIG.6 shows an example update to target position relative to instrument pose.

[0014] FIG.7 shows an example update to instrument pose relative to target position.

[0015] FIG.8 shows example corrections to instrument pose and target position in their respective coordinate spaces and updates to a spatial relationship between the instrument and the target in a common coordinate space.

[0016] FIG.9 shows a block diagram of an example registration system, according to some implementations.

[0017] FIG.10 shows another example medical system, according to some implementations.

[0018] FIG.11 shows an example mapping between an imaging space and a sensor space, according to some implementations.

[0019] FIG.12 shows another example medical system, according to some implementations.

[0020] FIG.13 shows another example medical system, according to some implementations.AUR6338WOPCT1 PCT

[0021] FIG.14 shows another example medical system, according to some implementations.

[0022] FIG.15 shows an example medical instrument, according to some implementations.

[0023] FIG.16 shows a block diagram of an example controller for a medical system, according to some implementations.

[0024] FIG.17 shows another block diagram of an example controller for a medical system, according to some implementations.

[0025] FIG.18 shows an illustrative flowchart depicting an example operation for navigating an instrument within an anatomy, according to some implementations.

[0026] FIG.19 shows an illustrative flowchart depicting an example operation for registering an image space with a sensor space, according to some implementations. DETAILED DESCRIPTION

[0027] In the following description, numerous specific details are set forth such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term “coupled” as used herein means connected directly to or connected through one or more intervening components or circuits. The terms “electronic system” and “electronic device” may be used interchangeably to refer to any system capable of electronically processing information. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the aspects of the disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the example implementations. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data bits within a computer memory.

[0028] These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present disclosure, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signalsAUR6338WOPCT1 PCT capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.

[0029] Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing the terms such as “accessing,” “receiving,” “sending,” “using,” “selecting,” “determining,” “normalizing,” “multiplying,” “averaging,” “monitoring,” “comparing,” “applying,” “updating,” “measuring,” “deriving” or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system’s registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

[0030] Certain standard anatomical terms of location may be used herein to refer to the anatomy of animals, and namely humans, with respect to the example implementations. Although certain spatially relative terms, such as “outer,” “inner,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” and similar terms, are used herein to describe a spatial relationship of one element, device, or anatomical structure to another device, element, or anatomical structure, it is understood that these terms are used herein for ease of description to describe the positional relationship between elements and structures, as illustrated in the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of the elements or structures, in use or operation, in addition to the orientations depicted in the drawings. For example, an element or structure described as “above” another element or structure may represent a position that is below or beside such other element or structure with respect to alternate orientations of the subject patient, element, or structure, and vice-versa. As used herein, the term “patient” may generally refer to humans, anatomical models, simulators, cadavers, and other living or non- living objects.

[0031] In the figures, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by that block may be performed in a single component or across multiple components, or may be performed using hardware, using software, or using a combination of hardware and software. To clearlyAUR6338WOPCT1 PCT illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described below generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure. Also, the example systems or devices may include components other than those shown, including well-known components such as a processor, memory and the like.

[0032] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium including instructions that, when executed, performs one or more of the methods described herein. The non-transitory processor- readable data storage medium may form part of a computer program product, which may include packaging materials.

[0033] The non-transitory processor-readable storage medium may comprise random access memory (RAM) such as synchronous dynamic random-access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, other known storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, or executed by a computer or other processor.

[0034] The various illustrative logical blocks, modules, circuits and instructions described in connection with the implementations disclosed herein may be executed by one or more processors (or a processing system). The term “processor,” as used herein may refer to any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, or state machine capable of executing scripts or instructions of one or more software programs stored in memory.AUR6338WOPCT1 PCT

[0035] As described above, many medical procedures include a preoperative phase that precedes an intraoperative phase. During the preoperative phase, for some medical procedures, an imaging system may be used to scan or otherwise capture images or video of at least a portion of a patient’s anatomy. For example, a computed tomography (CT) scanner may be used to acquire tomographic images (also referred to as “tomograms” or “CT scans”) of a patient’s lungs during the preoperative phase for a bronchoscopy. A tomogram is a cross-section or slice of a three-dimensional (3D) volume. For example, multiple tomograms can be stacked or combined to recreate the 3D volume (such as a 3D model of the patient’s lungs). Thus, tomograms can be used to detect a precise location or position (in 3D space) of a nodule or target in the patient’s lungs. During the intraoperative phase, for some medical procedures, a medical system may use the preoperative images to generate a graphical interface for navigating a medical instrument within the patient’s anatomy. For example, during a bronchoscopy, the medical system may detect a pose of an endoscope (such as a position and orientation of the scope in 3D space) based on sensor data received via an electromagnetic (EM) sensor disposed on the tip of the scope and map the pose of the endoscope to a 3D model of the patient’s lungs depicted by the tomograms.

[0036] Accordingly, the graphical interface may depict a spatial relationship between the medical instrument and the target within the anatomy based on the sensor data and the image data. However, images acquired during a preoperative phase may not accurately reflect the spatial relationship between the medical instrument and the target during an intraoperative phase. For example, changes in the patient’s anatomy or the medical environment can cause the spatial relationship between the endoscope and target to deviate from what is depicted by the graphical interface at any given time, which can lead to inaccurate navigation. Example factors include EM distortion, poor registration (or mapping) between the sensor space and the image space associated with the preoperative scans (also referred to as the “preoperative image space”), outdated preoperative scans, and anatomical deformations, among other examples. Aspects of the present disclosure recognize that some modern imaging technologies (such as cone beam CT) can be used to scan a patient’s anatomy during an intraoperative phase. In some aspects, a medical system may capture updated images of the patient’s anatomy during the intraoperative phase and use the updated image data to improve the representation of the spatial relationship between the medical instrument and the target.AUR6338WOPCT1 PCT

[0037] The updated image data and sensor data are often associated with different coordinate spaces. Thus, in some implementations, the medical system may “register” the updated image space with the sensor space to facilitate real-time navigation. As used herein, the term “registration” refers to a mapping or transformation between different coordinate spaces. For example, a medical system may register an imaging system used for capturing images of a patient’s anatomy (such as a cone beam CT scanner) with a sensor system used for tracking a pose of a medical instrument within the anatomy (such as an EM field generator) by determining a mapping or spatial transformation that maps any point or vector in the image space to a respective point or vector in the sensor space (such as a transformation matrix). The terms “mapping,” “transformation,” “spatial transformation,” and “registration matrix,” may be used interchangeably herein. The terms “respective” and “corresponding” also may be used interchangeably herein.

[0038] Although certain aspects of the present disclosure are described in detail herein in the context of bronchoscopy, it should be understood that the systems and techniques of the present disclosure may be applicable to any medical procedure. Example medical procedures may include minimally invasive procedures (such as laparoscopy), non- invasive procedures (such as endoscopy), therapeutic procedures, diagnostic procedures, percutaneous procedures, and non-percutaneous procedures, among other examples. Example endoscopic procedures include bronchoscopy, ureteroscopy, gastroscopy, nephroscopy, and nephrolithotomy, among other examples. The terms “scope,” “endoscope,” “catheter,” and “instrument” may be used interchangeably herein.

[0039] Aspects of the present disclosure may be used to perform robotic-assisted medical procedures, such as endoscopic access, percutaneous access, or treatment for a target anatomical site. For example, robotic tools may engage or control one or more medical instruments (such as an endoscope) to access a target site within a patient’s anatomy or perform a treatment at the target site. In some implementations, the robotic tools may be guided or controlled by a physician. In some other implementations, the robotic tools may operate in an autonomous or semi-autonomous manner. Although systems and techniques are described herein in the context of robotic-assisted medical procedures, the systems and techniques may be applicable to other types of medical procedures (such as procedures that do not rely on robotic tools or only utilize robotic tools in a very limited capacity). For example, the systems and techniques described herein may be applicable to medical procedures that rely on manually operated medical instruments (such as an endoscope that isAUR6338WOPCT1 PCT exclusively controlled and operated by a physician). The systems and techniques described herein also may be applicable beyond the context of medical procedures (such as in simulated environments or laboratory settings, such as with models or simulators, among other examples)

[0040] FIG.1 shows an example medical system 100 (also referred to as a “surgical medical system” or a “robotic medical system”), according to some implementations. As shown in FIG.1, the medical system 100 may be arranged for diagnostic or therapeutic bronchoscopy. The medical system 100 can include and utilize a robotic system 102 which can be implemented, for example, as a robotic cart. Although the medical system 100 is shown as including various cart-based systems or devices, the concepts disclosed herein can be implemented in any type of robotic system or arrangement, such as robotic systems employing rail-based components, table-based robotic end-effectors, or manipulators, among other examples. The robotic system 102 may include one or more robotic arms 104 (also referred to as “robotic positioners”) configured to position or otherwise manipulate a medical instrument 106 (such as a steerable endoscope or another elongate instrument). For example, the medical instrument 106 can be advanced through a natural orifice access point (such as the mouth 108 of a patient 110 positioned on a table 112) to deliver diagnostic or therapeutic treatment. Although described in the context of a bronchoscopy procedure, the medical system 100 also may be used to perform other types of medical procedures. Example suitable procedures include gastro-intestinal (GI) procedures, renal procedures, urological procedures, and nephrological procedures, among other examples.

[0041] With the robotic system 102 properly positioned, the medical instrument 106 can be inserted into the patient 110 robotically, manually, or a combination thereof. For example, the one or more robotic arms 104, or instrument drivers 114 coupled thereto, can control the medical instrument 106. In some implementations, the medical instrument 106 may be advanced within a sheath 116. For example, the sheath 116 may be coupled to, or controlled by, a robotic arm 104. In some implementations, the medical instrument 106 and the sheath 116 may each be coupled to a respective instrument driver from a set of instrument drivers 114. The instrument drivers 114 can be repositionable in space by manipulating the one or more robotic arms 104 into different angles or positions.

[0042] In the example of FIG.1, the medical instrument 106 can be directed down the patient’s trachea and lungs after insertion or advanced to a target destination or operative site. In some implementations, to enhance navigation through the patient’s lung network or reachAUR6338WOPCT1 PCT the desired target, the medical instrument 106 may be manipulated to telescopically extend from the outer sheath 116 to obtain enhanced articulation or greater bend radius. The use of separate instrument drivers 114 can allow the medical instrument 106 and sheath 116 to be driven independently of each other.

[0043] In some implementations, the medical instrument 106 may include an elongate member or shaft configured to be inserted or retracted, articulated, or otherwise moved within the anatomy. Further, in some implementations, the medical instrument 106 may include one or more imaging devices (such as cameras) positioned on a distal end of the elongate shaft or deployed through a working channel of the elongate shaft. The imaging devices can be configured to generate or capture image (or video) data or send the image data to another device or component. In some implementations, the medical instrument 106 may include an instrument base or one or more handles positioned at a proximal end of the medical instrument 106. The instrument base can be coupled to a manipulator (such as an end of a robotic arm 104). The instrument base can include one or more drive inputs coupled to one or more drive outputs of the manipulator, wherein the drive inputs or drive outputs act as an interface.

[0044] In some implementations, the medical instrument 106 may include a working channel configured to receive one or more other instruments or elements therein or provide other functionality. The working channel can extend axially, such as along the length of the medical instrument 106. Furthermore, the medical instrument 106 can include or be associated with one or more elongate movement members (such as pulls wires) that can extend from a proximal end through the elongate shaft to the distal end of the elongate shaft. The elongate movement members can be manipulated, such as by manipulators on the one or more robotic arms 104, to control actuation of the elongate movement members.

[0045] In some implementations, the medical instrument 106 may include one or more sensors, such as electromagnetic (EM) sensors, shape sensors (such as shape sensing fiber), accelerometers, gyroscopes, satellite-based positioning sensors (such as global positioning system (GPS) sensors), or radio-frequency (RF) transceivers, among other examples. The sensors can be configured to generate or produce sensor data or provide the sensor data to another device or component. The sensors can be disposed at a distal end of the elongate shaft or along a length of the elongate shaft. In some implementations, the medical instrument 106 may be configured to receive an elongate member or device through a working channel, wherein the elongate member includes one or more sensors along a lengthAUR6338WOPCT1 PCT of the elongate member. One or more sensors on the medical instrument 106 may provide sensor data to control circuitry of the medical system 100, which is then used to determine a position, orientation, or shape of the medical instrument 106.

[0046] The medical system 100 can also include a control system 118 (also referred to as a “control tower” or “mobile tower”). The control system 118 can be communicatively coupled (such as via wired or wireless connections) to the robotic system 102 to control various aspects of the robotic system 102 (such as electronics, optics, sensors, or power) or one or more subsystems associated with the robotic system 102, such as a fluid management system (not shown). Placing such functionality in the control system 118 can allow for a smaller form factor of the robotic system 102 that may be more easily adjusted or re- positioned by an operator or user. Additionally, the division of functionality between the robotic system 102 and the control system 118 can reduce operating room clutter and facilitate efficient clinical workflow.

[0047] The medical system 100 can include an electromagnetic (EM) field generator 120, which is configured to broadcast or emit an EM field that can be detected by various EM sensors, such as a sensor disposed on the medical instrument 106. The EM field can induce small electric currents in coils of the EM sensors, which can be analyzed to determine a position, angle, or orientation of the EM sensors relative to the EM field generator 120. Although EM fields and EM sensors are described in many examples herein, position sensing systems or sensors can include various other types of position sensing systems or sensors, such as optical position sensing systems or sensors, image-based position sensing systems or sensors, among other examples.

[0048] The medical system 100 can further include an imaging system 122 (also referred to as an “imaging device”) configured to generate, provide, or send image data (also referred to as “images”) to another device or system. For example, the imaging system 122 can generate image data depicting an anatomy of the patient 110 and provide the image data to the control system 118, the robotic system 102, or another device. The imaging system 122 may include an emitter or energy source (such as an X-ray source) or a detector (such as an X-ray detector) mounted on a C-shaped arm support 124, which allows for flexibility in positioning around the patient 110 to capture images from various angles without moving the patient 110. Use of the imaging system 122 can provide visualization of internal structures or anatomy, which can be used for a variety of purposes, including navigation of the medical instrument 106 (such as by providing images of internal anatomy to a user) and localizationAUR6338WOPCT1 PCT of the medical instrument 106 (based on an analysis of image data), among other examples. In some aspects, the imaging system 122 may enhance the efficacy or safety of a medical procedure, such as a bronchoscopy, by providing clear, continuous visual feedback to the operating surgeon or team.

[0049] In some implementations, the imaging system 122 may be a mobile device configured to move around an environment. For example, the imaging system 122 can be positioned next to the patient 110 (as shown in FIG.1) during a particular phase of a procedure and removed when the imaging system 122 is no longer needed. In some other implementations, the imaging system 122 may be part of the table 112 or other equipment in an operating environment. The imaging system 122 can be implemented as a Computed Tomography (CT) machine or system, X-ray machine or system, fluoroscopy machine or system, Positron Emission Tomography (PET) machine or system, PET-CT machine or system, CT angiography machine or system, Cone-Beam CT (CBCT) machine or system, three-dimensional rotational angiography (3DRA) machine or system, single-photon emission computed tomography (SPECT) machine or system, Magnetic Resonance Imaging (MRI) machine or system, Optical Coherence Tomography (OCT) machine or system, or ultrasound machine or system, among other examples. In some implementations, the medical system 100 may include different types of imaging systems that can be used or positioned over the patient 110 during different phases or portions of a procedure depending on the needs at that time.

[0050] In some implementations, the imaging system 122 may be configured to process multiple images (also referred to as “image data”) to generate a three-dimensional (3D) view or model. For example, the imaging device 122 can be implemented as a CT machine configured to capture or generate a series of images (also referred to as “tomograms) or image data representing two-dimensional (2D) cross-sections or slices of a 3D volume from different angles around the patient 110, and then use one or more algorithms to reconstruct these images or image data into a 3D model. The 3D model can be provided to the control system 118, robotic system 102, or another device, such as for processing or display.

[0051] In some implementations, image data from the imaging system 122 may be used to localize various elements, such as the medical instrument 106, a target within the anatomy, or specific anatomical features, among other examples. As used herein, the terms “localize,” “localization,” or “localizing” refer to any processes for determining or estimatingAUR6338WOPCT1 PCT a position (or location) and / or orientation (or heading), collectively referred to as the “pose,” of the instrument or the target (or any other element) within a given space or environment. For example, the control system 118 can be configured to provide navigation information during a procedure to assist a user navigating the medical instrument 106 within the anatomy to reach a target (such as a desired treatment site or location). In some implementations, a target can include a nodule, such as in the context of certain bronchoscopy procedures. To illustrate, the control system 118 can display a navigation view or graphical data 126 that includes an instrument indicator 128 representing the medical instrument 106, a target indicator 130 representing the target, and an anatomical map. The navigation data 126(A) (such as initial navigation data) can be determined based on sensor data from a sensor of the medical instrument 106 (such as EM sensor data associated with the EM field generator 120), a map of the anatomy, or a location of the target. In some implementations, the map or location of the target may be determined based on preoperative data, such as data obtained during a preoperative procedure to find a target location or map the anatomy.

[0052] In some implementations, the navigation data 126(A) may be dynamically updated based on image data 132 from the imaging system 122. For example, the control system 118 can receive the image data 132 and analyze the image data 132 to determine a current or actual spatial relationship between the medical instrument 106 and the target. In some implementations, the control system 118 may display the image data 132 to a user, receive user input indicating a position of the medical instrument 106 or a position of the target in the image data 132, and analyze the image data 132 based on the user input to determine the current spatial relationship. If the control system 118 determines that the navigation data 126(A) incorrectly depicts the location of the medical instrument 106 (such as where the spatial relationship associated with the image data 132 is different than the spatial relationship associated with the navigation data 126(A)), the control system 118 may update the navigation data 126(A) at 134 and provide updated navigation data 126(B) that reflects the current or near real-time position of the medical instrument 106 relative to the target or the map.

[0053] The various components of the medical system 100 can be communicatively coupled to each other over a network, which can include a wireless or wired network. Example networks include one or more personal area networks (PANs), local area networks (LANs), wide area networks (WANs), Internet area networks (IANs), cellular networks, the Internet, personal area networks (PANs), body area network (BANs), etc. In some examples,AUR6338WOPCT1 PCT various communication interfaces can include wireless technology, such as Bluetooth, Wi-Fi, near-field communication (NFC), or the like. Furthermore, in some examples, the various components of the medical system 100 can be connected for data communication, fluid exchange, power exchange, and so on, via one or more support cables, tubes, connections, or the like.

[0054] FIG.2 shows example components of the control system 118 and the robotic system 102 of FIG.1, according to some implementations. In the examples of FIG.2, the control system 118 and the robotic system 102 are implemented as a tower and a robotic cart, respectively. However, the control system 118 and robotic system 102 can be implemented in other manners. The control system 118 can be coupled to the robotic system 102 and operate in cooperation therewith to perform a medical procedure. For example, the control system 118 can include communication interface(s) 202 for communicating with communication interface(s) 204 of the robotic system 102 via a wireless or wired connection (such as to control the robotic system 102). In some implementations, the control system 118 may communicate with the robotic system 102 to receive position or sensor data therefrom relating to the position of sensors associated with an instrument or member controlled by the robotic system 102. For example, the control system 118 may communicate with the EM field generator 120 to control generation of an EM field in an area around a patient. The control system 118 can further include one or more power supply interface(s) 206.

[0055] The control system 118 can include control circuitry 208 configured to cause one or more components of the medical system 100 to actuate or otherwise control any of the various system components, such as carriages, mounts, arms or positioners, medical instruments, imaging devices, position sensing devices, or sensors, among other examples. Further, the control circuitry 208 can be configured to perform other functions, such as cause display of information, process data, receive input, communicate with other components or devices, or any other function or operation described herein.

[0056] The control system 118 can further include one or more input or out (I / O) components 210 configured to assist a physician or others in performing a medical procedure. For example, the one or more I / O components 210 can be configured to receive input or provide output to enable a user to control or navigate the medical instrument 106, the robotic system 102, or other instruments or devices associated with the medical system 100. The control system 118 can include one or more displays 212 to provide, display or otherwise present various information regarding a procedure. For example, the one or more displaysAUR6338WOPCT1 PCT 212 can be used to present navigation information including a virtual anatomical model of anatomy with a virtual representation of a medical instrument, image data, or other information. The one or more I / O components 210 can include one or more user input control(s) 214, which can include any type of user input (or output) devices or device interfaces, such as one or more buttons, keys, joysticks, handheld controllers (such as video- game-type controllers), computer mice, trackpads, trackballs, control pads, sensors (such as motion sensors or cameras) that capture hand gestures and finger gestures, touchscreens, toggle (such as button) inputs, or interfaces or connectors therefore. In some implementations, such inputs can be used to generate commands for controlling one or more medical instruments, robotic arms, or other components.

[0057] The control system 118 can also include data storage 216 configured to store executable instruments (such as computer-readable instructions) that can be executed by the control circuitry 208 to cause the control circuitry 208 to perform various operations or functionality described herein. In some implementations, the data storage 216 also may store telemetry or runtime data (such as sensor data or image data) generated by the medical system 100 or otherwise captured or acquired during a medical procedure. In some implementations, two or more components of the control system 118 can be electrically or communicatively coupled to each other.

[0058] The robotic system 102 can include the one or more robotic arms 104 configured to engage with or control, for example, the medical instrument 106 or other elements or components to perform one or more aspects of a procedure. As shown in FIG.2, each robotic arm 104 can include multiple segments 220 coupled to joints 222, which can provide multiple degrees of movement or freedom. The robotic system 102 can be configured to receive control signals from the control system 118 to perform certain operations, such as to position one or more of the robotic arms 104 in a particular manner or manipulate an instrument, among other examples. In response, the robotic system 102 can control, using control circuitry 224 thereof, actuators 226 or other components of the robotic system 102 to perform the operations. For example, the control circuitry 224 can control insertion or retraction, articulation, or roll of a shaft of the medical instrument 106 or other instrument by actuating one or more drive outputs 228 of a manipulator 230 (or end-effector) coupled to a base of a robotically-controllable instrument. The drive outputs 228 can be coupled to a drive input on an associated instrument, such as an instrument base of anAUR6338WOPCT1 PCT instrument that is coupled to the associated robotic arm 104. The robotic system 102 also may include one or more power supply interfaces 232.

[0059] The robotic system 102 can include a support column 234, a base 236, or a console 238. The console 238 can provide one or more I / O components 240, such as a user interface for receiving user input or a display screen (or a dual-purpose device, such as a touchscreen) to provide the physician or user with preoperative or intraoperative data. The support column 234 can include an arm support 242 (also referred to as a “carriage”) for supporting the deployment of the one or more robotic arms 104. The arm support 242 can be configured to vertically translate along the support column 234. Vertical translation of the arm support 242 allows the robotic system 102 to adjust the reach of the robotic arms 104 to meet a variety of table heights, patient sizes, or physician preferences. The base 236 can include wheel-shaped casters 244 (also referred to as “wheels”) that allow the robotic system 102 to move around the operating room. After reaching the appropriate position, the casters 244 can be immobilized using wheel locks to hold the robotic system 102 in place during the procedure.

[0060] The joints 222 of each robotic arm 104 can each be independently-controllable or provide an independent degree of freedom available for instrument navigation. In some implementations, each robotic arm 104 may include seven joints that provide seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom can allow robotic arms 104 to be controlled to position their respective manipulators 230 at a specific position, orientation, or trajectory in space using different linkage positions and joint angles. This allows for the robotic system 102 to position or direct a medical instrument from a desired point in space while allowing the physician to move the joints 222 into a clinically advantageous position away from the patient to create greater access, while avoiding collisions.

[0061] The one or more manipulators 230 (or end-effectors) can be coupled to an instrument base or handle, which can be attached using a sterile adapter component. The combination of the manipulator 230 and instrument base, as well as any intervening mechanics or couplings (such as the sterile adapter), can be collectively referred to as the manipulator or a manipulator assembly. Manipulators or manipulator assemblies can provide power or control interfaces. Example interfaces may include connectors to transfer pneumatic pressure, electrical power, electrical signals, or optical signals from the robotic arm 104 to an instrument base. Manipulators or manipulator assemblies can be configured toAUR6338WOPCT1 PCT manipulate medical instruments (such as surgical tools) using techniques including, for example, direct drives, harmonic drives, geared drives, belts or pulleys, or magnetic drives, among other examples.

[0062] The robotic system 102 can also include data storage 246 configured to store executable instruments (such as computer-readable instructions) that can be executed by the control circuitry 224 to cause the control circuitry 224 to perform various operations or functionality described herein. In some implementations, the data storage 216 also may store telemetry or runtime data (such as sensor data or image data) generated by the medical system 100 or otherwise captured or acquired during a medical procedure. In some implementations, two or more of the components of the robotic system 102 can be electrically or communicatively coupled to each other.

[0063] Data storage (including the data storage 216, data storage 246, or other data storage or memory) can include any suitable or desirable type of computer-readable media. For example, computer-readable media can include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, or nonremovable data storage devices implemented using any technology, layout, or data structure(s) or protocol, including any suitable or desirable computer-readable instructions, data structures, program modules, or other types of data.

[0064] Computer-readable media that can include, but is not limited to, phase change memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer- readable media may not generally include communication media, such as modulated data signals and carrier waves. As such, computer-readable media should generally be understood to refer to non-transitory media.

[0065] Functionality described herein can be implemented by the control circuitry 208 of the control system 118 or the control circuitry 224 of the robotic system 102, such as by the control circuitry 208 or 224 executing instructions to cause the control circuitry 208 or 224 to perform the functionality. Control circuitry (including the control circuitry 208,AUR6338WOPCT1 PCT control circuitry 224, or other control circuitry) can include circuitry embodied in a robotic system, control system or tower, instrument, or any other component or device. Control circuitry can include any collection of processors, processing circuitry, processing modules or units, chips, dies (such as semiconductor dies including one or more active or passive devices or connectivity circuitry), microprocessors, micro-controllers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines (such as hardware state machines), logic circuitry, analog circuitry, digital circuitry, or any device that manipulates signals (analog or digital) based on hard coding of the circuitry or operational instructions.

[0066] Control circuitry referenced herein can further include one or more circuit substrates (such as printed circuit boards), conductive traces and vias, or mounting pads, connectors, or components. Control circuitry can further include one or more storage devices, which may be embodied in a single device, a plurality of devices, or embedded circuitry of a device. Such data storage can comprise read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, or any device that stores digital information. In examples in which control circuitry includes a hardware or software state machine, analog circuitry, digital circuitry, or logic circuitry, data storage device(s) or register(s) storing any associated operational instructions can be embedded within, or external to, the circuitry comprising the state machine, analog circuitry, digital circuitry, or logic circuitry.

[0067] FIG.3 shows a block diagram of an example localization system 300, according to some implementations. The localization system 300 includes various positioning or imaging systems or modalities 302–312 (also referred to as “subsystems”), which can be implemented to facilitate anatomical mapping, navigation, positioning, or visualization for procedures in accordance with one or more examples. For example, the various systems 302–312 can be configured to provide data for generating an anatomical map, determining a location of an instrument, determining a location of a target, or performing other techniques.

[0068] Each of the systems 302–312 can be associated with a respective coordinate space (also referred to as a “position coordinate frame”) or can provide data or information relating to instrument or anatomy locations, wherein registering the various coordinate spaces to one another can allow for integration of the various systems to provide mapping, navigation, or instrument visualization. For example, registering a first modality to a secondAUR6338WOPCT1 PCT modality can allow for determined positions in the first modality to be tracked or superimposed on or in a reference frame associated with the second modality, thereby providing layers of positional information that can be combined to provide a robust localization system.

[0069] In some aspects, the system 300 may be configured to perform one or more localization or localizing techniques. In some implementations, the anatomical space in which a medical instrument can be localized (such as where a pose or shape of the instrument is determined or estimated) may be a 2D or 3D portion of a patient’s tracheobronchial airways, vasculature, urinary tract, gastrointestinal tract, or any organ or space accessed via lumens. Various modalities can be implemented to provide images, representations, or models of the anatomical space. For example, an imaging modality can be implemented, which can include, for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography, CBCT, 3DRA, SPECT, MRI, OCT, or ultrasound, among other examples. In some implementations, the imaging modality may be used to capture or acquire images of a patient’s anatomy during a preoperative phase of a medical procedure. In some other implementations, the imaging modality may be used to capture or acquire images of a patient’s anatomy during an intraoperative phase of the medical procedure.

[0070] The systems 302–312 can provide information for generating a graphical interface 314 (also referred to as a “graphical interface (I / F)”) that includes navigation information for navigating an instrument to a target within an anatomy (such as the navigation data 126(A) or 126(B) of FIG.1). For example, the navigation information may include an anatomical map, an estimated position, orientation, and / or shape of the instrument. The navigation information also may include a shape, boundary, eccentricity, texture, and / or position of the target. In some implementations, the graphical user interface 314 or other localization information may be displayed to a user, such as a physician, during a medical procedure to assist the user in performing the procedure. For example, a visualization of a tracked instrument can be superimposed on an anatomical map depicted by the graphical user interface 314 based on position or sensor data associated with the tracked medical instrument.

[0071] As shown in FIG.3, the system 300 can include a support structure 302 (such as a surgical bed or other patient positioning or support platform). For example, the support structure 302 includes a planar surface that contacts and supports the patient. In some implementations, the position of the support structure 302 may be known based on data maintained relating to the position of the support structure 302 within the surgical orAUR6338WOPCT1 PCT procedure environment. In some other implementations, the position of the support structure 302 may be sensed or otherwise determined using one or more markers or an appropriate imaging or positioning modality.

[0072] The system 300 can further include a robotic system 304 (such as a robotic cart or other device or system including one or more robotic end effectors). In some implementations, the robotic system 304 may be one example of the robotic system 102 of FIGS.1 and 2. Data relating to the position or state of robotic arms, actuators, or other components of the robotic system 304 can be known or derived from robotic command data or other robotic data relative to a coordinate frame of the robotic system 304. In some examples, reference frame registration 316 occurs between the support structure 302 and the robotic system 304, which can be a relatively coarse registration, in some implementations, based on robotic system or cart-set-up procedure (which can have any suitable or desirable scheme).

[0073] The system 300 can further include an electromagnetic (EM) sensor system 306, which can include an EM field generator (such as the EM field generator 120 of FIG.1) and one or more EM sensors. An EM sensor can be associated with a portion of an instrument that is tracked or controlled, such as a distal end (or tip) of the instrument or along a length of the instrument or other elongate member (such as a working channel) disposed in a lumen of the instrument. In some implementations, the EM field generator can be mechanically coupled to the support structure 302 or the robotic system 304 such that registration or association 318 between such systems can be known or determined. In some implementations, the registration 318 between the EM sensor system 306 and the robotic system 304 can be determined through forward kinematics or field generator mount transform information. For example, the field generator can be mounted to the support structure 302 such that the position of the field generator can be known relative to the robotic system positioning frame based on a known relationship between the position of the support structure 302 and the robotic system 304. The EM sensor system 306 can provide instrument pose or path information based on sensor readings associated with the instrument.

[0074] The system 300 can further include an optical camera system 308 including one or more cameras or other imaging devices configured to generate images of patient anatomy within a visual field thereof (such as real-time image data) during a surgical procedure. In some implementations, registration 320 between the optical camera system 308 and the EM sensor system 306 can be achieved through identification of features having EMAUR6338WOPCT1 PCT sensor data associated therewith, such as a medical instrument tip, in images generated by the optical camera system 308. The registration 320 can further be based at least in part on hand- eye interaction of the physician when viewing real-time camera images while the EM-sensor- equipped endoscope is navigating in the patient anatomy.

[0075] The system 300 can further include a computed tomography (CT) imaging system 310 configured to generate CT images of the patient anatomy, which can be performed preoperatively or intraoperatively. The CT imaging system 310 is generally used for scanning a relatively large volume. In some implementations, image processing can be implemented for registration 322 of the CT image data with the camera image data generated by the optical camera system 308. For example, common features identified in both camera image data and CT image data can be identified to relate the CT image frame to the camera image frame in space. In some examples, the CT imaging system 310 can be used to generate preoperative imaging data for producing the graphical user interface 314 or for path navigation planning.

[0076] In some aspects, the CT imaging system 310 may be registered 326 to the EM sensor system 306 through various techniques. In some implementations, a mechanical structure of the CT imaging system 310 can have a known physical transform or relationship with respect to a mounting position of the EM field generator of the EM sensor system 306. Such known relationship can be used to register the CT image space to the EM sensor space. The connection 328 represents a mapping or relationship between the CT imaging system 310 and an anatomical map depicted by a graphical user interface 314.

[0077] The system 300 can further include a fluoroscopy imaging system 312 configured to generate tomographic images (such as real-time X-ray images) of the surgical site. The fluoroscopy imaging system 312 is generally used for scanning a smaller volume compared to the CT imaging system 310. In some implementations, the fluoroscopy imaging system 312 may be one example of the imaging system 122 of FIG.1. For example, the fluoroscopy imaging system 312 may include a CBCT scanner coupled to a C-arm. In some implementations, the fluoroscopy imaging system 312 may be used with a contrast agent introduced into the anatomy to generate image data representing patient anatomy or instrumentation. In some implementations, the fluoroscopy imaging system 312 may be registered 324 to the CT imaging system 310 using any image processing technique suitable for such registration.AUR6338WOPCT1 PCT

[0078] In some aspects, the fluoroscopy imaging system 312 may be registered 332 to the EM sensor system 306 through various techniques. In some implementations, a mechanical structure of the fluoroscopy imaging system 312 (such as the C-arm instrumentation) can have a known physical transform or relationship with respect to a mounting position of the EM field generator of the EM sensor system 306. Such known relationship can be used to register the fluoroscopy image space to the EM sensor space. The connection 330 represents a mapping or relationship between the fluoroscopy imaging system 312 and an anatomical map depicted by the graphical user interface 314.

[0079] In the example of FIG.3, the CT imaging system 310 and fluoroscopy imaging system 312 are illustrated as separated systems. However, in some other implementations, a single imaging system may perform the functions of both the CT imaging system 310 and fluoroscopy imaging system 312.

[0080] The position, shape, or orientation of an instrument, such as an endoscope, can be determined using any one or more of the systems 302–312, which can facilitate generation of graphical interface data representing the estimated position or shape of the instrument relative to an anatomical map depicted by the graphical user interface 314. The graphical user interface 314 can be displayed on a display device, such as via the control system 118 or robotic system 102, or another device. In some implementations, the graphical user interface 314 also may indicate a position of a target within the anatomy that has been designated for treatment.

[0081] Although the systems 302–312 have been described in a particular order, the operations or functions associated therewith can be performed in different orders. In some implementations, the systems 302–312 can be used in different ways. In some other implementations, registration can occur between different systems and modalities.

[0082] In some aspects, one or more of the systems 302–312 may be used to generate the graphical user interface 314 preoperatively or determine a location of one or more targets within an anatomical map depicted by the graphical user interface 314 during a preoperative phase of a medical procedure. As used herein, the terms “skeleton image data” and “skeleton image space” refer to images of an anatomy captured or acquired by an imaging system (such as the CT imaging system 310) during the preoperative phase of a medical procedure. However, a graphical user interface 314 generated using a preoperative CT scan may not accurately reflect the spatial relationship between a medical instrument and a target during an intraoperative phase. For example, changes in the patient’s anatomy or the medicalAUR6338WOPCT1 PCT environment can cause the spatial relationship between the instrument and the target to deviate from what is depicted in the graphical user interface 314. Example factors that may cause such deviations include EM distortion, poor registration (or mapping) between the sensor data and the image data, outdated preoperative scans, and anatomical deformations, among other examples. Thus, in some other aspects, one or more of the systems 302–312 may be used to determine a location of a medical instrument or position of a target relative to an anatomical map depicted by the graphical user interface 314 during an intraoperative phase of the medical procedure.

[0083] FIG.4 shows a block diagram of an example navigation system 400, according to some implementations. In some aspects, the navigation system 400 may be implemented by a medical system such as the medical system 100 of FIG.1. With reference to FIG.2, the navigation system 400 may be one example of any of the control circuitry 208 or 224. The navigation system 400 is configured to generate or update a graphical interface 404 that depicts or otherwise indicates a spatial relationship between a medical instrument and a target within an anatomy. For example, the graphical interface 404 may be used by the medical system or a user thereof (such as a physician) to navigate the instrument to the target. In some implementations, the graphical interface 404 may be one example of the graphical user interface 314 of FIG.3.

[0084] In some aspects, the navigation system may generate the graphical interface 404 based at least in part on image data 402 (also referred to as “skeleton image data”) acquired via an imaging system (such as the CT imaging system 310 of FIG.3) and sensor data 401 acquired via a sensor system (such as the EM sensor system 306 of FIG.3). For example, the skeleton image data 401 may include one or more tomograms captured as a result of a CT scan (such as described with reference to FIGS.1–3). In some implementations, the skeleton image data 401 may be captured during a preoperative phase of a medical procedure. The sensor data 401 may indicate a pose of the instrument and / or one or more EM sensors disposed in an EM field (such as described with reference to FIGS.1–3). As used herein, the term “pose” may refer to a position and / or orientation of a sensor or an instrument. In some implementations, the pose of the instrument may be determined based at least in part on sensor data 401 received from a sensor disposed on the instrument (such as at the distal end of an endoscope).

[0085] The navigation system 400 includes a skeleton registration component 410, a navigation component 420, an intraoperative registration component 430, and a navigationAUR6338WOPCT1 PCT update component 440. The skeleton registration component 410 is configured to register a coordinate space associated with the sensor data 401 (also referred to as the “sensor space”) to a coordinate space associated with the skeleton image data 402 (also referred to as the “skeleton image space”). As used herein, the term “registration” refers to a mapping between different coordinate spaces. In other words, the skeleton registration component 410 may determine a mapping 403, based at least in part on the sensor data 401 and the skeleton image data 402, that can transform any data point (including a point coordinate or vector) in the sensor space to a respective data point in the image space (such as a transformation matrix).

[0086] Various existing techniques can be used to register the skeleton image space with the sensor space, and vice-versa. For example, a number of EM sensors (such as 3 or more EM sensors) may be placed around a patient’s body at predetermined positions relative to the anatomy. The skeleton registration component 410 may determine the positions of the EM sensors in an EM field (which represents the sensor space) based on the received sensor data 401. The skeleton registration component 410 also may determine corresponding positions for the EM sensors in the skeleton image space based on their relative locations to known anatomical features depicted by the skeleton image data 402. The skeleton registration component 410 may further determine a mapping 403 (or transformation matrix) that transforms the positions of the EM sensors in the sensor space to their respective positions in the skeleton image space.

[0087] The navigation component 420 is configured to generate the graphical interface 404 based on the sensor data 401, the skeleton image data 402, and the mapping 403 between the sensor space and the skeleton image space. The graphical interface 404 may indicate a spatial relationship between the instrument and the target within the anatomy. For example, the graphical interface 404 may include a three-dimensional (3D) model of the anatomy for display on one or more displays associated with the medical system (such as the displays 212 of FIG.2). In some implementations, the navigation component 420 may render the 3D model based at least in part on the skeleton image data 402. Accordingly, the 3D model of the anatomy may provide a visual indicator of the location of the target within the anatomy. In some aspects, the graphical interface 404 may be used by the medical system or a user (such as a physician) to navigate the instrument to the target during the intraoperative phase of the medical procedure.

[0088] In some aspects, the navigation component 420 also may display a pose and / or path of the instrument on the 3D model of the anatomy. For example, the navigation component 420 may determine the pose of the instrument in relation to the anatomy based onAUR6338WOPCT1 PCT the sensor data 401 and the mapping 403 between the sensor space and the skeleton image space. More specifically, the navigation component 420 may use the mapping 403 to transform the pose of the instrument in the sensor space (as indicated by the sensor data 401) to a corresponding pose in the skeleton image space. In some implementations, the navigation component 420 may implement a “fusion” navigation model that combines multiple sensing modalities (such as EM, robotic telemetry, and vision) to determine the instrument pose and / or compensate for any distortions in the sensor data 401. The navigation component 420 may continuously track any changes in the pose or movement of the instrument based on the sensor data 401 and update the graphical interface 404 to reflect the real-time pose of the instrument in relation to the anatomy.

[0089] In some aspects, the navigation component 420 may analyze a spatial relationship between the instrument and the target to provide guidance for navigating the instrument within the anatomy. For example, the navigation component 420 may plan a navigation path through the patient’s anatomy from the instrument to the target. The navigation path may be rendered or displayed on the graphical interface 404 for the instrument to follow. In some implementations, the navigation component 420 may determine the position of the target in the skeleton image space based at least in part on the skeleton image data. In some other implementations, the navigation component 420 may determine the position of the target in the skeleton image space based at least in part on input provided by a user of the navigation system 400. For example, the user may specify or otherwise input the location of the target on the 3D map of the anatomy.

[0090] As described above, images acquired during the preoperative phase may not accurately reflect the spatial relationship between the instrument and the target during the intraoperative phase. For example, changes in the patient’s anatomy or the medical environment can cause the spatial relationship between the instrument and target to deviate from what is depicted by the graphical interface 404 at any given time. Example factors that may cause such deviations include EM distortion, poor registration (or mapping 403) between the sensor space and the skeleton image space, outdated skeleton image data 402, and anatomical deformations, among other examples. In some aspects, the medical system may capture updated images of the patient’s anatomy via an intraoperative imaging system (such as the fluoroscopy imaging system 312 of FIG.3) and use the updated image data 406 (also referred to as “intraoperative image data”) to improve or update the depiction of the spatial relationship between the instrument and the target in the graphical interface 404.AUR6338WOPCT1 PCT

[0091] In some aspects, the navigation component 420 may be configured to operate in a “navigation mode” or a “re-navigation mode” depending on whether a correction is applied to the graphical interface 404 based on the intraoperative image data 406. For example, the navigation component 420 may operate in the navigation mode while the instrument is initially navigated to a threshold proximity of the target prior to acquiring the intraoperative image data 406. While operating in the navigation mode, the navigation component 420 may use the fusion navigation model, which combines multiple sensing modalities (such as EM, robotic telemetry, and vision), to compensate for any distortions in the sensor data 401. When navigating the instrument past the threshold, the navigation component 420 may be configured to operate in the re-navigation mode. In some aspects, the operating mode of the navigation component 420 may be manually toggled by a user of the navigation system 400. For example, the user may dynamically switch between the navigation mode and the re-navigation mode at any given time during a medical procedure (provided certain preconditions are met). In some implementations, the navigation component 420 may pause 405 navigation or robot motion so that the intraoperative imaging system can be brought to a location proximate the anatomy to capture or acquire the intraoperative image data 406.

[0092] The intraoperative registration component 430 is configured to register a coordinate space associated with the intraoperative image data 406 (also referred to as the “intraoperative image space”) with the sensor space responsive to the navigation component 420 signaling a pause 405 in navigation. More specifically, the intraoperative registration component 430 may determine a mapping 407, based at least in part on the sensor data 401 and the intraoperative image data 406, that can transform any data point (including a point coordinate or vector) in the intraoperative image space to a respective data point in the sensor space. In some aspects, the intraoperative registration component 430 may determine the mapping 407 based at least in part on a pose of the instrument in the sensor space and a corresponding instrument pose in the intraoperative image space. In some other aspects, the intraoperative registration component 430 may determine the mapping 407 based on known locations of landmarks (such as fiducial markers or anatomical markers) in both the sensor space and the intraoperative image space.

[0093] In some implementations, the intraoperative registration component 430 may determine the pose of the instrument in the intraoperative image space through analysis of the intraoperative image data 406 using one or more image processing techniques. Example suitable image processing techniques include segmentation, machine learning, and statisticalAUR6338WOPCT1 PCT analysis, among other examples. As used herein, the term “segmentation” refers to various techniques for partitioning a digital image into groups of voxels (or “image segments”) based on related characteristics or identifying features. In some implementations, the intraoperative registration component 430 may segment the intraoperative image data 406 so that the pose of the instrument can be detected or estimated from the corresponding images (or tomograms).

[0094] The intraoperative registration component 430 may determine the pose of the instrument in the sensor space based at least in part on the sensor data 401. Aspects of the present disclosure recognize that some imaging technologies (such as CT or X-rays) may interfere with some sensor technologies (such as EM). Thus, in some implementations, the intraoperative registration component 430 may determine the pose of the instrument in the sensor space based on sensor data 401 captured or acquired while the intraoperative imaging system cannot interfere with the underlying sensor technology (such as while the fluoroscopy imaging system 312 is located outside the EM field of the EM sensor system 306).

[0095] The patient is generally breathing while the sensor data 401 is captured but may be forced into a breath-hold while capturing the intraoperative image data 406 (such as to avoid visual artifacts or other inaccuracies in the resulting images). The patient’s breathing may cause the instrument pose in the sensor space to differ from the pose in the image space at any given time. In some implementations, the intraoperative registration component 430 may accumulate sensor data 401 over a threshold duration so that time- varying changes in the instrument pose can be detected and compensated in the captured data. For example, the threshold duration may be a predetermined duration configured to span one or more respiratory cycles (such as 12 seconds). As a result, the mapping 407 generated by the intraoperative registration component 430 may compensate for any variations or deviations in the instrument pose due to sensor interference or changes in anatomy.

[0096] By compensating for deviations in instrument pose caused by sensor interference when registering the sensor space with the intraoperative image space, the intraoperative registration component 430 may rely on a single sensor modality (such as EM) having data available before any interference is introduced to compensate for any distortions in the sensor data 401. In other words, the intraoperative registration component 430 may not use any of the additional modalities (such as robotic telemetry or vision), that are otherwise used when operating in the navigation mode, to compensate for deviations in instrument pose when determining the mapping 407. In some implementations, the intraoperative registration component 430 may continue to collect data from one or more additional sensing modalitiesAUR6338WOPCT1 PCT (such as robotics or vision) while determining the mapping 407 and the interface updates 408. More specifically, the intraoperative registration component 430 may use the additional sensor data to detect any distortions in the sensor data 401, for example, to ensure that the sensor data 401 used for registration is undistorted.

[0097] The navigation update component 440 is configured to determine one or more interface updates 408 to be applied to the graphical interface 404 based on the intraoperative image data 406 and the mapping 407. In some implementations, the interface updates 408 may include a position of the target transformed from the intraoperative image space to the skeleton image space. In some other implementations, the interface updates 408 may include a pose of the instrument transformed from the intraoperative image space to the sensor space. For example, the navigation update component 440 may determine the target position or the instrument pose in the intraoperative image space through analysis of the intraoperative image data 406 using one or more image processing techniques (such as described above). The navigation update component 440 may further use the mapping 407 to transform the target position or instrument pose in the intraoperative image space to a corrected target position or instrument pose, respectively, in the sensor space. As used herein, the terms “corrected target position” and “corrected instrument pose” refer to the target position and instrument pose that are transformed from the intraoperative image space to the sensor space. The corrected target position and / or the corrected instrument pose in the sensor space may be provided, as the interface updates 408, to the navigation component 420.

[0098] Because deviations in instrument pose are compensated when registering the sensor space with the intraoperative image space, the navigation update component 440 also may rely on a single sensor modality (such as EM) to compensate for any distortions in the sensor data 401. In other words, the navigation update component 440 may not use any of the additional modalities (such as robotic telemetry or vision), that are otherwise used when operating in the navigation mode, to compensate for deviations in instrument pose when determining the interface updates 408. As described in greater detail below, any subsequent distortions in sensor data 401 introduced by the imaging system may be compensated by the navigation component 420 when applying the interface updates 408. As such, neither the intraoperative registration component 430 nor the navigation update component 440 may notify a user of any detected distortion when determining the mapping 407 or the interface updates 408.

[0099] In some implementations, the navigation component 420 may update the graphical interface 404 based at least in part on the interface updates 408. More specifically,AUR6338WOPCT1 PCT the navigation component 420 may update the spatial relationship between the instrument and the target depicted by the graphical interface 404. In some aspects, the navigation component 420 may update the spatial relationship in the sensor space. For example, the navigation component 420 may use the mapping 403 to transform the position of the target from the skeleton image space to the sensor space. Assuming the physical position of the target remains unchanged from the preoperative phase of the medical procedure to the intraoperative phase, the navigation component 420 may determine a transformation (^^^^^^) that maps the position of the target in the sensor space to the corrected target position indicated by the interface updates 408. The navigation component 420 may apply the same transformation ^^^^^^to the sensor data 401 to correct or adjust the spatial relationship between the instrument and the target while operating in the re-navigation mode.

[0100] In some other aspects, the navigation component 420 may update the spatial relationship between the instrument and the target in the skeleton space. In some implementations, the navigation component 420 may determine a correction to be applied to the position of the target in the skeleton image space based on the interface updates 408. In such implementations, the navigation component 420 may use the mapping 403 to transform the corrected target position in the sensor space to a corresponding position in the skeleton image space and determine a transformation (^^^^^^^ೄ^ಶ^) between the current position of the target in the skeleton image space and the target position. The navigationcomponent 420 may apply the transformation ^^^ೄ^ಶ^to the current position of the target (such as by modifying the skeleton image data 402 or the 3D model of the anatomy) to correct or adjust the position of the target relative to the current instrument pose in the graphical interface 404.

[0101] In some other implementations, the navigation component 420 may determine a correction to be applied to the instrument pose in the skeleton image space based at least in part on the interface updates 408. In such implementations, the navigation component 420 may apply a correction transformation (^^^^^^ ^^^^ௌೄ^ಶ^), based on the transformation ^^^ೄ^ಶ^associated with the corrected target position, to real-time sensor data 401 (such as by modifying the sensor data 401) to correct or adjust the pose of the instrument relative to the position of the target in the graphical interface 404. As described above, some imaging technologies may interfere with some sensor technologies. As a result, any sensor data 401 acquired after the intraoperative imaging system is brought within proximity of the anatomy may deviate due to interference caused by the imaging system or further changes to the anatomy. In someAUR6338WOPCT1 PCT implementations, while operating in the re-navigation mode, the navigation component 420 may accumulate the sensor data 401 over a threshold duration (such as 12 seconds) to detect and compensate for any changes or variations in instrument pose over one or more respiratory cycles

[0102] Still further, in some implementations, the navigation component 420 may determine corrections to be applied to the instrument pose and the target position in the skeleton image space. In such implementations, the mapping 407 may be determined based on landmarks in the sensor space and the intraoperative image space. For example, the navigation component 420 may determine a correction transformation (^^^^^^ௌೄ^^ೞ^^) that transforms the current instrument pose in the sensor space to the corrected instrument pose and may apply the transformation ^^^^^^ௌೄ^^ೞ^^to real-time sensor data 401 (such as by modifying the sensor data 401) to correct or adjust the pose of the instrument in the graphical interface 404. Furthermore, the corrected instrument pose in the sensor space may be mapped to the skeleton image space using the mapping 403. The navigation component 420 also may apply the transformation ^^^^^^^ೄ^ಶ^to the current position of the target (such as by modifying the skeleton image data 402 or the 3D model of the anatomy) to correct or adjust the position of the target in the graphical interface 404.

[0103] By compensating for deviations in instrument pose when applying the interface updates 408 and corrections to real-time sensor data 401, the navigation component 420 may rely on a single sensor modality (such as EM) while operating in the re-navigation mode. In other words, the navigation component 420 may not use any of the additional modalities (such as robotic telemetry or vision), that are otherwise used when operating in the navigation mode, to compensate for deviations in instrument pose when operating in the re- navigation mode. In some implementations, the navigation component 420 may continue to collect data from one or more additional sensing modalities while applying the interface updates 408 to the sensor data 401. More specifically, the navigation component 420 may use the additional sensor data to detect any distortions in the sensor data 401, for example, to notify the user of any detected distortions in the re-navigation mode (so that the user can remove any sources of distortion to improve the accuracy of the graphical interface 404).

[0104] The updated graphical interface 404 provided by the re-navigation mode may offer many advantages over the existing graphical interface 404 provided by the navigation mode. For example, by updating the spatial relationship between the instrument and the target based on the intraoperative image data 406 acquired during the intraoperative phase,AUR6338WOPCT1 PCT the navigation system 400 can provide more accurate navigation information to the medical system or user. In some aspects, the navigation system 400 may dynamically update the graphical interface 404 based on new intraoperative image data 406. For example, the intraoperative imaging system may be used to scan the anatomy multiple times throughout the intraoperative phase. Each time the anatomy is scanned, the intraoperative registration component 430 may determine a new mapping 407 based on the most recent intraoperative image data 406 and the sensor data 401, and the navigation update component 440 may use the updated mapping 407 to generate new interface updates 408.

[0105] The navigation component 420 also may provide additional features in the graphical interface 404 when operating in the re-navigation mode. In some aspects, the navigation component 420 may update the planned navigation path based on the updated patient anatomy depicted by the intraoperative image data 406 and / or the updated spatial relationship between the instrument and the target. For example, the updates may be depicted in contrast (such as with different colors) to the original navigation path. In some implementations, the navigation component 420 may revise the original navigation path (such as by creating new segments or modifying existing segments). In some other implementations, the navigation component 420 may create a new navigation path (also referred to as a “re-navigation path”) from the current instrument pose to the position of the target.

[0106] In some other aspects, the navigation component 420 may use the intraoperative image data 406 to supplement or replace at least a portion of the skeleton image data 402. For example, aspects of the present disclosure recognize that by correcting the instrument pose in the skeleton image space based on an instrument pose determined relative to the intraoperative image space, the corrected position of the instrument in the skeleton image space may lie outside of an airway on the 3D model of the anatomy generated using the skeleton image data 402. In some implementations, the navigation component 420 may recreate at least a portion of the 3D model using the intraoperative image data 406. For example, the new 3D model may supplant the original 3D model in the graphical interface 404 (as a whole or in overlapping regions) to provide contemporaneous imaging and anatomical information.

[0107] In some other implementations, the graphical interface 404 may display a 3D model of the anatomy based on the skeleton image data 402 and concurrently display another 3D model of the anatomy based on the intraoperative image data 406. For example, the 3D models may be displayed in contrasting colors or with varying degrees of translucencyAUR6338WOPCT1 PCT (which may be user-defined). This may allow the user to assess the degree of deviation or divergences between the 3D models and make commensurate adjustments to workflow or instrument navigation in accordance with their professional judgment. In some implementations, the divergence between the models may be quantified by a set of metrics (such as local deformation field magnitude, percentage overlap, or path angle difference, among other examples) that can be displayed or presented on the graphical interface 404 in various formats (such as heat maps or annotations, among other examples).

[0108] As described above, the navigation component 420 may dynamically switch between the navigation mode and the re-navigation mode at any given time. For example, if the user is unsatisfied with the spatial relationship depicted by the graphical interface 404 in the re-navigation mode, or if the user prefers the additional sensing modalities provided by the navigation mode, the user may toggle the navigation component 420 to revert back to the navigation mode at any time. The ability to update the graphical interface 404 with new 3D models of the anatomy may further enhance the functionality for toggling between the navigation mode and the re-navigation mode. For example, displaying multiple 3D models concurrently allows the user to determine which model better reflects the current instrument pose indicated by another sensing modality (such as a camera disposed on the distal end of the instrument). Accordingly, the updated 3D models also may provide a framework for using multiple sensing modalities in the re-navigation mode.

[0109] FIG.5 shows an example spatial relationship 500 between a skeleton image space 510, a sensor space 520, and an intraoperative image space 530. In some implementations, the skeleton image space 510 and the intraoperative image space 530 may be examples of the coordinate spaces associated with the skeleton image data 402 and the intraoperative image data 412, respectively, of FIG.4. More specifically, FIG.5 shows a position of a target (^^^^^^^ೌ^^) and a pose of an instrument (^^ௌ^^^^ೌ^^) in the intraoperative image space 530.

[0110] In some aspects, a navigation system (such as the navigation system 400 of FIG.4) may transform the target position ^^^^^^^ೌ^^ and instrument pose ^^ௌ^^^^ೌ^^ in the intraoperative image space 530 to a target position (^^^ೄೖ^^^^^^) and an instrument pose (^^ௌೄೖ^^^^^^), respectively, in the skeleton image space 510. As described with reference to FIG.4, the intraoperative registration component 430 may determine a mapping (^^ூௌ^^௧^^^^^^^^ ) that transforms any data point in the intraoperative image space 530 to apoint in the sensor space 520, and the skeleton registration component 410 may determine aAUR6338WOPCT1 PCT mapping (^^ௌௌ^^^^^^^^^௧^^) that transforms any data point in the sensor space 520 to a respective data point in the skeleton image space 510. In some implementations, the sensor space 520 may be one example of the coordinate space associated with the sensor data 401 of FIG.4. In some implementations, the mappings ^^ூௌ^^௧^^^^^^^^ and ^^ௌௌ^^^^^^^^^௧^^may be examples of the mappings 407 and 403, respectively, of FIG.4.

[0111] With reference for example to FIG.4, the navigation update component 440 may apply the mapping ^^ூௌ^^௧^^^^^^^^ to the target position ^^^^^^^ೌ^^ and the instrument pose ^^ௌ^^^^ೌ^^ in the intraoperative image space 530 to transfer the target position and instrument pose to the sensor space 520, and the navigation component 420 may apply the mapping ^^ௌௌ^^^^^^^^^௧^^to further transfer the resulting target position and instrument pose from the sensor space 520 to the skeleton image space 510. Accordingly, the target position ^^^ೄೖ^^^^^^ and the instrument pose ^^ௌೄೖ^^^^^^ in the skeleton image space 510 can be expressed as: ^^ ൌ ^^ ௌ^^^^௧^^ ∗ ௌ^^^^^^ೄೖ^^^^^^ ௌ^^^^^ ^^ூ^௧^^^^ ∗ ^^^^^^^ೌ^^As shown inthe intraoperative space 530 and the skeleton image space 510 that can transform any datapoint in the intraoperative image space 530 directly to a data point in the skeleton image space 510, where: ^^ௌ^^^^௧^^ ௌ^^^^௧^^ூ^௧^^^^ ൌ ^^ௌ^^^^^ ∗ ^^ூௌ^^௧^^^^^^^^

[0112] FIG.6 shows an example update 600 to target position relative to instrument pose. In some implementations, the update 600 may be performed in a skeleton image space (such as the coordinate space associated with the skeleton image data 402 of FIG.4). More specifically, FIG.6 shows a current position of a target (^^^ೄ^ಶ^) and a current pose of an instrument (^^ே^௪ௌೄ^ಶ^), as well as a corrected position of the target (^^^ೄ^ಶ^) and a corrected pose of the instrument (^^ே^௪ௌೄ^ಶ^), in the skeleton image space.

[0113] In some aspects, a navigation system (such as the navigation system 400 of FIG.4) may determine a correction (^^^^^^^ೄ^ಶ^) to be applied to the current target position ^^^ೄ^ಶ^based on the corrected target. In the example of FIG.6, the system may assume that the current instrument pose ^^ௌೄ^ಶ^ represents or otherwise coincides with the corrected instrument pose ^^ே^௪ ^^^^ௌೄ^ಶ^. Thus, the correction ^^^ೄ^ಶ^represents aAUR6338WOPCT1 PCT transformation between the current target position ^^^ೄ^ಶ^ and the corrected target position ^^ே^௪^ೄ^ಶ^, where: ^^^^^^ ே^௪^ೄ^ಶ^ ൌ ^^^ೄ^ಶ^ െ ^^^ೄ^ಶ^ ^1^As described with component 420 may apply the rrection ^^^^co^^^ೄ^ಶ^to the or of the anatomy to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. Among other advantages, the correction ^^^^^^^ೄ^ಶ^can be applied once and paired with subsequent sensor data (even in the absence of sensor data, such as due to EM dropout) without requiring continuous updates.

[0114] FIG.7 shows another example update 700 to instrument pose relative to target position. In some implementations, the update 700 may be performed in a skeleton image space (such as the coordinate space associated with the skeleton image data 402 of FIG.4). More specifically, FIG.7 shows a current position of a target (^^^ೄ^ಶ^) and a current pose of an instrument (^^ௌೄ^ಶ^), as well as a corrected position of the target (^^ே^௪^ೄ^ಶ^) and a corrected pose of the instrument (^^ே^௪ௌೄ^ಶ^), in the skeleton image space.

[0115] In some aspects, a navigation system (such as the navigation system 400 of FIG.4) may determine a correction (^^^^^^ௌೄ^ಶ^) to be applied to the current instrument pose ^^ based on the transform^^^^ௌೄ^ಶ^ ation ^^^ೄ^ಶ^, between the current target position ^^^ೄ^ಶ^ and the corrected target position ^^ே^௪^ೄ^ಶ^, shown in Equation 1 (as described with reference to FIG.6). In the example of FIG.7, the navigation system may assume that the current target position ^^ represents or otherwiseே^௪^ೄ^ಶ^ coincides with the corrected target position ^^^ೄ^ಶ^. Thus, the correction ^^^^^^ௌೄ^ಶ^represents a transformation matrix between the current instrument pose ^^ௌೄ^ಶ^ and the corrected instrument pose ^^ே^௪ௌೄ^ಶ^, where: ^^^^ ^^^^^^ ൌ ^ ^^ െ^^^ೄ^ಶ^ ^2^where ^^ is the identity matrix.

[0116] As described with reference to FIG.4, the navigation component 420 may apply the correction ^^^^^^to the sensor data 401 or the 3D model of the anatomy to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. Among other advantages, applying the correction ^^^^^^to the sensor data 401 allows the navigation system to compensate for deviations in instrument pose that may beAUR6338WOPCT1 PCT caused by sensor interference or changes in anatomy (such as described with reference to FIG.4).

[0117] FIG.8 shows example corrections 800 to instrument pose and target position in their respective coordinate spaces and updates to a spatial relationship between the instrument and the target in a common coordinate space (such as a skeleton image space). In the example of FIG.8, the update to the instrument pose is depicted in the sensor space but can be mapped to the skeleton image space using any of the techniques described with reference to FIGS.4 and 5 (such as via the mapping 403). In some implementations, the skeleton image space and the sensor space may be examples of the coordinate spaces associated with the skeleton image data 402 and sensor data 401, respectively, of FIG.4. More specifically, FIG.8 shows a current position of a target (^^^ೄ^ಶ^) and a corrected position of the target (^^ே^௪^ೄ^ಶ^) in the skeleton image space, as well as a current pose of an instrument (^^ ) and a corrected pose of the inே^௪ௌೄ^^ೞ^^ strument (^^ௌೄ^^ೞ^^) in the sensor space.

[0118] In some aspects, a navigation system (such as the navigation system 400 of FIG.4) may determine a correction (^^^^^^ௌೄ^^ೞ^^) to be applied to the current instrument pose ^^ and a^^^^ௌೄ^^ೞ^^ correction (^^^ೄ^ಶ^) to be applied to the current target position ^^^ೄ^ಶ^. As described with the corrections ^^^ೄ^^^^^ೞ^^and ^^^^ೄ^^^ಶ^rely on a mapping 407ௌ ^between the intraoperative image space and the sensor space that is determined based on landmarks in both coordinate spaces (rather the instrument pose alone). The correction ^^^^^^^ೄ^ಶ^can be determined based on the current target position ^^^ೄ^ಶ^ and the corrected target position ^^ே^௪^ೄ^ಶ^, according to Equation 1 (as described with reference to FIG.6). Similarly, the correction ^^^^^^ௌೄ^^ೞ^^can be determined as a transformation between the current instrument pose ^^ௌೄ^^ೞ^^ and the corrected instrument pose ^^ேௌೄ^^௪^ೞ^^, where: ^ௌೄ^^^^ೞ^^ ൌ ^^^ି^ ^^^ௌೄ^^ೞ^^൧ ∗ ^^ே^௪ௌೄ^^ೞ^^ ^3^As described with420 may apply the corrections ^^^^^^ௌೄ^^ೞ^^and ^^^^^^^ೄ^ಶ^to the sensor data 401 and the skeleton image data 402, respectively, or the 3D model of the anatomy, to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. For example, the navigation component 420 may convert the current instrument pose ^^ௌೄ^^ೞ^^ to a respective instrument pose (^^ே^௪ ௌ^ா^ௌೄ^ಶ^) in the skeleton image space by applying a mapping (^^ௌ^^^^^) thatAUR6338WOPCT1 PCT transforms any data point in the sensor space to a respective data point in the skeleton image space (such as the mapping 403), where: ^^ே^௪ ൌ ^^ ௌ^ா^ ∗ ^ ^^^^ௌೄ^ಶ^ ௌ^^^^^ ^ௌೄ^^ೞ^^ ∗ ^^ௌೄ^^ೞ^^

[0119] FIG.9 shows a block diagram of an example registration system 900, according to some implementations. In some implementations, the registration system 900 may be one example of the skeleton registration component 410 and / or the intraoperative registration component 430 of FIG.4. The registration system 900 is configured to register a sensor system with an imaging system during an intraoperative phase of a medical procedure to facilitate real-time navigation of a medical instrument within an anatomy. In some implementations, the sensor system and the imaging system may be examples of the EM sensor system 306 and the fluoroscopy imaging system 312, respectively, of FIG.3.

[0120] In some aspects, the registration system 900 may determine a mapping 906 between a coordinate space associated with the sensor system and a coordinate space associated with the imaging system based, at least in part, on sensor data 901 and image data 902 received via the sensor system and the imaging system, respectively. For example, the mapping 906 may be used to transform any data point (such as a coordinate or a vector) in the image space to a respective data point in the sensor space. In some implementations, the sensor data 901 may include position or heading information about one or more EM sensors disposed in an EM field (such as described with reference to FIGS.1–3). In some implementations, the image data 902 may include one or more tomograms captured by a CBCT scanner (such as described with reference to FIGS.1–3). Aspects of the present disclosure recognize that some imaging technologies (such as CT or X-rays) may interfere with some sensor technologies (such as EM). In some aspects, the registration system 900 may determine the mapping 906 based on sensor data 901 that is captured or acquired while the imaging system cannot interfere with the underlying sensor technology (such as when a CBCT system is located outside an EM field).

[0121] The registration system 900 includes a first reference point extraction component 910, a second reference point extraction component 920, and a spatial mapping determination component 930. The first reference point extraction component 910 is configured to determine one or more reference data points 903 based on the sensor data 901 and the second reference point extraction component 920 is configured to determine one or more reference data points 904 based on the image data 902. As used herein, the term “reference data point” may refer to any object, feature, point, or vector that can be identifiedAUR6338WOPCT1 PCT or found in both the sensor space and the image space. In some implementations, the reference data points 903 and 904 may include a position of a medical instrument (such as a position of the tip of an endoscope). In some other implementations, the reference data points 903 and 904 may include a heading of the medical instrument (such as an orientation of the tip of the endoscope).

[0122] For example, the first reference point extraction component 910 may determine the position and heading of an endoscope (in the sensor space) based on sensor data 901 associated with an EM sensor disposed on or coupled to the tip of the endoscope. The second reference point extraction component 920 may determine the position and heading of the same scope tip (in the image space) through segmentation of the image data 902. As used herein, the term “segmentation” refers to various techniques for partitioning a digital image into groups of voxels (or “image segments”) based on related characteristics or identifying features. Example suitable segmentation techniques include machine learning (ML) models, masking, thresholding, clustering, and edge detection, among other examples. In some implementations, the second reference point extraction component 920 may segment the image data 902 so that the position and heading of the scope tip can be detected or estimated from the corresponding tomograms.

[0123] The spatial mapping determination component 930 is configured to determine the mapping 906 based, at least in part, on the reference data points 903 in the sensor space and the reference data points 904 in the image space. In some implementations, the mapping 906 may be a transformation matrix. Aspects of the present disclosure recognize that at least 3 unique reference data points (which may include any combination of coordinates or vectors) are needed to define a unique transformation matrix in 3D space. For example, given 3 unique reference data points, the spatial mapping determination component 930 can determine a transformation matrix that transforms the 3 reference data points in the image space to the 3 reference data points in the sensor space using various known techniques or algorithms. Example suitable algorithms for determining the spatial transformation matrix include singular value decomposition (SVD) and the iterative closest point algorithm (ICP), among other examples.

[0124] In some implementations, each of the reference point extraction components 910 and 920 may be configured to extract at least 3 unique reference data points 903 and 904 from the sensor data 901 and the image data 902, respectively. In such implementations, the spatial mapping determination component 930 may determine the mapping 906 using only the reference data points 903 and 904. However, in some other implementations, at least oneAUR6338WOPCT1 PCT of the reference point extraction components 910 or 920 may provide fewer than 3 unique reference data points 903 or 904, respectively. For example, the reference data points 903 may include only the position or heading of the instrument in the sensor space and the reference data points 904 may include only the position or heading of the instrument in the image space. In such implementations, the spatial mapping determination component 930 may rely on one or more known data points 905 in the image space and the sensor space to determine the mapping 906.

[0125] In some aspects, the known data points 905 may be based on one or more assumptions or known characteristics about the medical system or the surrounding environment. With reference for example to FIG.1, aspects of the present disclosure recognize that the C-arm 124 of the imaging system 122 is positioned orthogonal to the table 112 (or bed) that supports the patient 110 and the table 112 is generally parallel to the floor. In some implementations, the EM field generator 120 may be a window field generator (WFG) positioned directly beneath (or above) the table 112 so that an open center (or “window”) of the WFG is parallel to, and overlaps, a planar surface of the table 112 (such as the surface of the table 112 on which the patient 110 lies). By positioning the EM field generator 120 and the imaging system 122 at known orientations or tilt angles in relation to the table 112, aspects of the present disclosure can control the relationship between the sensor space and the image space. For example, by positioning the EM field generator 120 parallel to the table 112 and positioning the imaging system 122 orthogonal to the table 112 (such as at a 0º cranial / caudal tilt angle), the image space may be described by a 90- or 180-degree rotation relative to the sensor space. Any discrepancies or offsets in the angle of the table 112 or the tilt-angle of the C-arm 124 can be accounted for by rotating the data set in the sensor space or the image space, for example, to ensure that the two data sets have high correspondence.

[0126] Accordingly, various known data points 905 can be inferred or extrapolated from a known spatial relationship between the imaging system and the sensors system. In some implementations, the known data points 905 may include a vector pointing towards the floor of the medical environment. For example, such vector may be orthogonal to the planar surface of the patient bed and the window of the WFG. In some other implementations, the known data points 905 may include a vector pointing towards the head (or toes) of the patient. For example, such vector may be parallel to the planar surface of the patient bend and the window of the WFG. In some implementations, the mapping determination component 930 may determine the mapping 906 based on one or more reference data pointsAUR6338WOPCT1 PCT 903 and 904 and one or more known data points 905. More specifically, the mapping determination component 930 may determine the mapping 906 based on any combination of 3 (or more) unique data points 903–905.

[0127] FIG.10 shows another example medical system 1000, according to some implementations. The medical system 1000 includes an imaging system 1010, a WFG 1020, a support structure 1030 configured to support a patient anatomy 1040, and an endoscope 1050 inserted into the patient anatomy 1040. In some implementations, the medical system 1000 may be one example of the medical system 100 of FIG.1. With reference for example to FIG.1, the imaging system 1010 may be one example of the imaging system 122, the WFG 1020 may be one example of the EM field generator 120, the support structure 1030 may be one example of the table 112, and the endoscope 1050 may be one example of the medical instrument 106.

[0128] In the example of FIG.10, the WFG 1020 is disposed beneath the support structure 1030 (as shown in the exploded view) so that a window of the WFG 1020 is parallel to a planar surface of the support structure 1030 on which the anatomy 1040 lies. The distal end (or tip) of the endoscope 1050 includes an EM sensor that can interact with an EM field generated by the WFG 1020 to produce sensor data indicating a position 1021 and a heading 1022 of the scope tip in the sensor space (such as the sensor data 901 of FIG.9). Due to the density of its materials, the endoscope 1050 can be segmented from image data captured by the imaging system 1010 (such as the image data 902 of FIG.9). In some implementations, the medical system 1000 may determine a position 1011 and a heading 1012 of the scope tip in the image space based on the segmented image data. In the example of FIG.10, the data points 1011 and 1012 describe the same pose of the endoscope 1050, but in different coordinate spaces, as the data points 1021 and 1022. Accordingly, the position 1011 of the scope tip in the image space can be mapped to the position 1021 of the scope tip in the sensor space, and the heading 1012 of the scope tip in the image space can be mapped to the heading 1022 of the scope tip in the sensor space.

[0129] In some aspects, the medical system 1000 may assume that a vector 1013 pointing from the tip of the endoscope 1050 towards the floor in the image space can be mapped to a vector 1023 pointing from the tip of the endoscope 1050 towards the floor in the sensor space. As shown in FIG.10, the vectors 1013 and 1023 (also referred to as “floor vectors”) are orthogonal to the window of the WFG 1020 and the planar surface of the support structure 1030. For example, the floor vector 1023 can be defined as the vector^0 0 1^் according to a cartesian coordinate system associated with the sensor space.AUR6338WOPCT1 PCT Based on the relative positioning of the imaging system 1010 and the WFG 1020, the medical system 1000 may determine that the floor vector 1013 in the image space is rotated 90 degrees relative to the floor vector 1023 in the sensor space. Accordingly, the floor vector1013 can be described by the vector ^0 1 0^் according to a left-posterior-superior (LPS)coordinate system associated with the image space. Alternatively, the floor vector 1013 canbe described by the vector ^0 െ1 0^் according to a right-anterior-inferior (RAI)coordinate system.

[0130] In some aspects, the medical system 1000 may assume that a vector 1014 pointing from the tip of the endoscope 1050 towards the head (or toes) of the patient in the image space can be mapped to a vector 1024 pointing from the tip of the endoscope 1050 towards the head (or toes) of the patient in the sensor space. As shown in FIG.10, the vectors 1014 and 1024 (also referred to as “head-to-toe vectors”) are parallel to the window of the WFG 1020 and the planar surface of the support structure 1030. The head-to-toe vectors 1014 and 1024 are also orthogonal to the floor vectors 1013 and 1023, respectively.Thus, the head-to-toe vector 1024 can be defined as the vector ^0 1 0^் or ^0 െ1 0^்according to a cartesian coordinate system associated with the sensor space (depending on the orientation of the WFG 1020). In some implementations, the medical system 1000 maydetermine whether the head-to-toe vector 1024 is ^0 1 0^் or ^0 െ1 0^் based on howthe WFG 1020 is registered to an image space associated with preoperative CT scans (such as the registration 326 of FIG.3). The medical system 1000 may further assume that the head- to-toe vector 1014 in the current image space is rotated 90 degrees relative to the head-to-toe vector 1024 in the sensor space. Thus, the head-to-toe vector 1014 can be described by thevector ^0 0 1^் according to the LPS coordinate system associated with the image space.Alternatively, the head-to-toe vector 1014 can be described by the vector ^0 0 െ1^்according to the RAI coordinate system.

[0131] In some implementations, the floor vectors 1013 and 1023 and head-to-toe vectors 1014 and 1024 may be examples of the known data points 905 of FIG.9. As described with reference to FIG.9, only 3 unique data points are needed to determine a transformation matrix that maps any data points in the image space to respective data points in the sensor space. In some instances, the headings 1012 and 1022 of the endoscope 1050 may overlap with the floor vectors 1013 and 1023 or the head-to-toe vectors 1014 and 1024. In such instances, the registration system 900 may still use the floor vectors 1013 and 1023 and the head-to-toe vectors 1014 and 1024, together with the positions 1011 and 1021 of theAUR6338WOPCT1 PCT endoscope 1050, to determine the mapping 906. In other words, the floor vectors 1013 and 1023, head-to-toe vectors 1014 and 1024, and positions 1011 and 1021 of the endoscope 1050 can always provide a set of 3 unique reference data points even if the headings 1012 and 1022 overlap with one of the other vectors.

[0132] FIG.11 shows an example mapping 1101 between a CT coordinate space 1110 and an EM coordinate space 1120, according to some implementations. In some implementations, the CT space 1110 and the EM space 1120 may be examples of the image space and the sensor space, respectively, associated with the imaging system 1010 and the WFG 1020 of FIG.10.

[0133] The CT space 1110 is shown to include a point coordinate ^^்_^்which forms the origin for a set of vectors ^^்_^், ^^ி_^், and ^^ு_^். With reference for example to FIG. 10, the point ^^்_^்may represent the position 1011 of the endoscope 1050 in the image space, the vector ^^்_^்may represent the heading 1012 of the endoscope 1050 in the image space, the vector ^^ி_^்may represent the floor vector 1013 in the image space, and the vector ^^ு_^்may represent the head-to-toe vector 1014 in the image space. In the example of FIG.11, the heading vector ^^்_^் is described by the cartesian coordinates ^െ1 0 0^், thefloor vector ^^ி_^் is described by the cartesian coordinates ^0 1 0^், and the head-to-toevector ^^ு_^் is described by the cartesian coordinates ^0 0 1^் in the CT space 1110.

[0134] The EM space 1120 is shown to include a point coordinate ^^்_ாெwhich forms the origin for a set of vectors ^^்_ாெ, ^^ி_ாெ, and ^^ு_ாெ. With reference for example to FIG. 10, the point ^^்_ாெmay represent the position 1021 of the endoscope 1050 in the sensor space, the vector ^^்_ாெmay represent the heading 1022 of the endoscope 1050 in the sensor space, the vector ^^ி_ாெmay represent the floor vector 1023 in the sensor space, and the vector ^^ு_ாெmay represent the head-to-toe vector 1024 in the sensor space. In the exampleof FIG. 11, the heading vector ^^ is descri ^ ^்்_ாெ bed by the cartesian coordinates െ1 0 0 ,the floor vector ^^ is described by ^ ^்ி_ாெ the cartesian coordinates 0 0 1 , and the head-to-toe vector ^^ு_ாெ is described by the cartesian coordinates ^0 െ1 0^் in the EM space1120.

[0135] As shown in FIG.11, the spatial mapping 1101 transforms the point coordinate ^^்_^்and the set of vectors ^^்_^், ^^ி_^், and ^^ு_^்from the CT space 1110 to the point coordinate ^^்_ாெand the set of vectors ^^்_ாெ, ^^ி_ாெ, and ^^ு_ாெ, respectively, in the EM space 1120. In some implementations, the spatial mapping 1101 may be one example ofAUR6338WOPCT1 PCT the mapping 906 of FIG.9. With reference for example to FIG.9, the registration system 900 may determine the spatial mapping 1101 based on the datapoints ^^்_^், ^^்_^், ^^ி_^், ^^ு_^், ^^்_ாெ, ^^்_ாெ, ^^ி_ாெ, and ^^ு_ாெ. More specifically, the registration system 900 may calculate a transformation matrix ^^ாெ^் that transforms the set of data^^^்_^் െ ^^்_^் ^^்_^் ^^ி_^் ^^ு_^்^ into the set of data points ^^ ൌ^^^்_ாெെ ^^்_ாெ^^்_ாெ^^ி_ாெ^^ு_ாெ^, where:^^ ൌ ^^^ா்ெ^^ where ^^்_^்denotes the centroid of points in the CT space 1110 and ^^்_ாெdenotes the centroid of points in the EM space 1120. Because there is only one known coordinate pointin each of the CT space 1110 (^^்_^்) and the EM space 1120 (^^்_ாெ), ^^்_^் ൌ ^^்_^் and^^்_ாெ ൌ ^^்_ாெ.

[0136] The transformation matrix ^^^ா்ெincludes a rotation matrix ^^^^^and a translation vector ^^^^^, where ^^ாெ^் ൌ ^^^^^^ ^^^^^^. In some implementations, theregistration system 900 may solve for the rotation matrix ^^^^^using singular value decomposition (SVD). For example, the registration system 900 may determine an orthogonal matrix U, a diagonal matrix S, and an orthogonal matrix V, where: ^^^^^^^^^ ^^்^ ൌ ^^ ^^ ^^்^^ ൌ ^^ ்^^^ ^^

[0137] The registration system 900 may further calculate the translation vector ^^^^^based on the point coordinates ^^்_^்and ^^்_ாெ, where: ^^^^^ ൌ ^^்_ாெ െ ^^^^^^^்_^்

[0138] As shown in FIG.11, the rotation matrix ^^^^^merely rotates the vectors ^^்_^், ^^ி_^், and ^^ு_^்90 degrees around the x-axis in the CT space 1110. Thus, as an alternative to SVD, the registration matrix 900 may determine a first rotation matrix ^^^^^^that transforms the heading vector ^^்_^்in the CT space 1110 to the heading vector ^^்_ாெin the EM space 1120 (assuming the vectors have the same origin). The registration matrix 900may apply the rotation matrix ^^^^^^ to the set of data points ^^ ൌ^^^்_^்^^்_^்^^ி_^்^^ு_^்^ and determine a second rotation matrix ^^^ଶ^^ that rotates theremaining vectors ^^^^^^^^ி_^்and ^^^^^^^^ு_^்around the heading vector ^^^^^^^^்_^்toAUR6338WOPCT1 PCT transform the vectors ^^^^^^^^ி_^்and ^^^^^^^^ு_^்into the vectors ^^ி_^்and ^^ு_^், respectively,in the EM space 1120, where ^^^^^ ൌ ^^^^^^ ^^^ଶ^^ .

[0139] FIG.12 shows another example medical system 1200, according to some implementations. The medical system 1200 includes an imaging system 1210, an EM field generator (FG) 1220, a support structure 1230 configured to support a patient anatomy 1240, and an endoscope 1250 inserted into the patient anatomy 1240. In some implementations, the medical system 1200 may be one example of the medical system 100 of FIG.1. With reference for example to FIG.1, the imaging system 1210 may be one example of the imaging system 122, the EM field generator 1220 may be one example of the EM field generator 120, the support structure 1230 may be one example of the table 112, and the endoscope 1250 may be one example of the medical instrument 106.

[0140] In the example of FIG.12, the EM field generator 1220 is depicted as a WFG disposed beneath the support structure 1230 (as shown in the exploded view) so that a window of the WFG is parallel to a planar surface of the support structure 1230 on which the anatomy 1240 lies. Alternatively, the EM field generator 1220 may be disposed above the support structure 1230 (such as in the form of a tabletop field generator). However, in some other implementations, the EM field generator 1220 may be any suitable EM field generator capable of producing one or more EM fields in or proximate to the anatomy 1240. The distal end of the endoscope 1250 includes an EM sensor that can interact with an EM field generated by the EM field generator 1220 to produce sensor data indicating a position 1221 and / or a heading 1222 of the scope tip in the sensor space. In some implementations, the endoscope 1250 may further include one or more additional EM sensors disposed at predetermined distances along the length of the shaft. For example, as shown in FIG.12, the additional EM sensors can produce sensor data indicating positions of predetermined points 1223 and 1224 along the shaft of the endoscope 1250. The sensor data also may indicate headings of the additional EM sensors (not shown for simplicity). In some implementations, the sensor data produced by the EM sensors may be examples of the sensor data 901 of FIG. 9.

[0141] Due to the density of its materials, the endoscope 1250 can be segmented from image data captured by the imaging system 1210 (such as the image data 902 of FIG.9). In some implementations, the medical system 1200 may determine a position 1211 and / or a heading 1212 of the scope tip in the image space based on the segmented image data. Because the additional EM sensors are disposed at predetermined distances (from the scopeAUR6338WOPCT1 PCT tip) along the length of the endoscope 1250, the medical system 1200 also may determine positions 1213 and 1214 and / or headings (not shown for simplicity) of the additional EM sensors from the segmented image data. In the example of FIG.12, the data points 1221– 1224 describe the same pose (and shape) of the endoscope 1250, but in different coordinate spaces, as the data points 1211–1214. Accordingly, the positions 1211, 1213, and 1214 and / or headings of the EM sensors in the image space can be mapped to the positions 1221, 1223, and 1224 and / or headings of the EM sensors, respectively, in the sensor space. Similarly, the heading 1212 of the scope tip in the image space can be mapped to the heading 1222 of the scope tip in the sensor space.

[0142] As described with reference to FIGS.9 and 11, the medical system 1200 may determine a transformation matrix that maps any data point in the image space to a respective data point in the sensor space based on 3 unique reference data points (which can include any combination of point coordinates or vectors). Thus, when the positions and / or headings of at least 3 EM sensors can be detected in each of the sensor space and the image space, the medical system 1200 may not require the heading vector of the endoscope 1250 to determine the transformation matrix. In some implementations, the medical system 1200 may guide a user or physician to collect sensor and image data that captures at least 2 EM sensors. However, if fewer than 2 EM sensors are detected in at least one of the sensor space or the image space, the medical system 1200 may rely on one or more known data points (such as the known data points 905 of FIG.9) to determine the transformation matrix. In some implementations, the known data points may include a floor vector pointing from the tip of the endoscope 1250 towards the floor of the medical environment or a head-to-toe vector pointing from the tip of the endoscope 1250 towards the head (or toes) of the patient (such as described with reference to FIG.10).

[0143] In some other implementations, the medical system 1200 may determine the transformation matrix based on a shape of the endoscope 1250. For example, the medical system 1200 may estimate the shape of the endoscope 1250 in the sensor space based on EM sensor data (such as position and / or heading information) collected from multiple EM sensors disposed along the length of the endoscope 1250. The medical system 1200 may further estimate the shape of the endoscope 1250 in the image space based on segmented image data. In such implementations, the medical system 1200 may determine a transformation matrix that transforms multiple points along the estimated shape of the instrument in the image space to corresponding points along the estimated shape of the instrument in the sensor space.AUR6338WOPCT1 PCT

[0144] FIG.13 shows another example medical system 1300, according to some implementations. The medical system 1300 includes an imaging system 1310, an EM field generator (FG) 1320, a support structure 1330 configured to support a patient anatomy 1340, and an endoscope 1350 inserted into the patient anatomy 1340. In some implementations, the medical system 1300 may be one example of the medical system 100 of FIG.1. With reference for example to FIG.1, the imaging system 1310 may be one example of the imaging system 122, the EM field generator 1320 may be one example of the EM field generator 120, the support structure 1330 may be one example of the table 112, and the endoscope 1350 may be one example of the medical instrument 106.

[0145] In the example of FIG.13, the EM field generator 1320 is depicted as a WFG disposed beneath the support structure 1330 (as shown in the exploded view) so that a window of the WFG is parallel to a planar surface of the support structure 1330 on which the anatomy 1340 lies. However, in some other implementations, the EM field generator 1320 may be any suitable EM field generator capable of producing one or more EM fields in or proximate to the anatomy 1340. The distal end of the endoscope 1350 includes an EM sensor that can interact with an EM field generated by the EM field generator 1320 to produce sensor data indicating a position 1321 and / or a heading 1322 of the scope tip in the sensor space. In some implementations, the medical system 1300 may further include one or more additional EM sensors disposed at predetermined positions proximate to the anatomy 1340. For example, as shown in FIG.13, the additional EM sensors can be disposed on the support structure 1330 to produce sensor data indicating positions of predetermined points 1323 and 1324 on either side of the anatomy 1340. In some implementations, the sensor data produced by the EM sensors may be examples of the sensor data 901 of FIG.9.

[0146] Due to the density of its materials, the endoscope 1350 can be segmented from image data captured by the imaging system 1310 (such as the image data 902 of FIG.9). In some implementations, the medical system 1300 may determine a position 1311 and / or a heading 1312 of the scope tip in the image space based on the segmented image data. In some implementations, the additional EM sensors may be radio-opaque so that positions 1313 and 1314 of the additional EM sensors also can be identified or detected in the segmented image data. In the example of FIG.13, the data points 1311 and 1312 describe the same pose of the endoscope 1350, but in different coordinate spaces, as the data points 1321 and 1322. Accordingly, the position 1311 of the scope tip in the image space can be mapped to the position 1321 of the scope tip in the sensor space, and the heading 1312 of the scope tip in the image space can be mapped to the heading 1322 of the scope tip in the sensor space.AUR6338WOPCT1 PCT Because the additional EM sensors can be detected in both image space and the sensor space, the positions 1313 and 1314 of the additional EM sensors in the image space also can be mapped to the positions 1323 and 1324 of the additional EM sensors, respectively, in the sensor space.

[0147] As described with reference to FIGS.9 and 11, the medical system 1300 may determine a transformation matrix that maps any data point in the image space to a respective data point in the sensor space based on 3 unique reference data points (which can include any combination of point coordinates or vectors). Thus, when the positions of at least 3 EM sensors can be detected in each of the sensor space and the image space, the medical system 1300 may not require the heading vector of the endoscope 1350 to determine the transformation matrix. In some implementations, the positions 1313 and 1314 of the additional EM sensors may be determined (in the image space) based on two or more 2D images (rather than 3D images) captured by the imaging system 1310 at different C-arm angles (such as at an anterior-posterior angle and at a 90 degree rotation therefrom). More specifically, the positions 1313 and 1314 of the additional EM sensors may be determined based on a rigid transformation from the 2D images to a 3D space (which may be known or predetermined).

[0148] FIG.14 shows another example medical system 1400, according to some implementations. The medical system 1400 includes an imaging system 1410, an EM field generator (FG) 1420, a support structure 1430 configured to support a patient anatomy 1440, and an endoscope 1450 inserted into the patient anatomy 1440. In some implementations, the medical system 1400 may be one example of the medical system 100 of FIG.1. With reference for example to FIG.1, the imaging system 1410 may be one example of the imaging system 122, the EM field generator 1420 may be one example of the EM field generator 120, the support structure 1430 may be one example of the table 112, and the endoscope 1450 may be one example of the medical instrument 106.

[0149] In the example of FIG.14, the EM field generator 1420 is a WFG disposed beneath the support structure 1430 (as shown in the exploded view) so that a window of the WFG is parallel to a planar surface of the support structure 1430 on which the anatomy 1440 lies. In some implementations, the WFG 1420 may be coupled to a fiducial board 1460 having a predetermined orientation or position relative to the WFG 1420. The fiducial board 1460 may include one or more radio-opaque fiducial markers disposed thereon. For example, as shown in FIG.14, the fiducial board 1460 includes two fiducial markers 1423 and 1424 disposed at predetermined positions relative to the WFG 1420. In some implementations, theAUR6338WOPCT1 PCT positions of the fiducial markers 1423 and 1424 in the sensor space may be examples of the known data points 905 of FIG.9. The distal end of the endoscope 1450 includes an EM sensor that can interact with an EM field generated by the EM field generator 1420 to produce sensor data indicating a position 1421 and / or a heading 1422 of the scope tip in the sensor space. In some implementations, the sensor data produced by the EM sensors may be examples of the sensor data 901 of FIG.9.

[0150] Due to the density of its materials, the endoscope 1450 can be segmented from image data captured by the imaging system 1410 (such as the image data 902 of FIG.9). In some implementations, the medical system 1400 may determine a position 1411 and / or a heading 1412 of the scope tip in the image space based on the segmented image data. Because the fiducial markers 1423 and 1424 are radio-opaque, the positions 1413 and 1414 of the fiducial markers 1423 and 1424, respectively, also can be identified or detected in the segmented image data. In the example of FIG.14, the data points 1411 and 1412 describe the same pose of the endoscope 1450, but in different coordinate spaces, as the data points 1421 and 1422. Accordingly, the position 1411 of the scope tip in the image space can be mapped to the position 1421 of the scope tip in the sensor space, and the heading 1412 of the scope tip in the image space can be mapped to the heading 1422 of the scope tip in the sensor space. Because the relative placement of the fiducial markers 1423 and 1424 is known in the sensor space, and can be detected in the image space, the positions 1413 and 1414 of the fiducial markers in the image space also can be mapped to known positions of the fiducial markers 1423 and 1424, respectively, in the sensor space.

[0151] As described with reference to FIGS.9 and 11, the medical system 1400 may determine a transformation matrix that maps any data point in the image space to a respective data point in the sensor space based on 3 unique reference data points (which can include any combination of point coordinates or vectors). However, when multiple fiducial markers lie in the same plane (such as the fiducial markers 1423 and 1424), the medical system 1400 may additionally rely on the position 1411 or the heading 1412 of the scope tip to determine the transformation matrix. In some implementations, the positions 1413 and 1414 of the fiducial markers may be determined (in the image space) based on two or more 2D images (rather than 3D images) captured by the imaging system 1410 at different C-arm angles (such as at an anterior-posterior angle and at a 90 degree rotation therefrom). More specifically, the positions 1413 and 1414 of the fiducial markers may be determined based on a rigid transformation from the 2D images to a 3D space (which may be known or predetermined).AUR6338WOPCT1 PCT

[0152] In the examples described with reference to FIGS.10–14, a medical system (or registration system) may register the image space with the sensor space based on the pose of an endoscope having 5 mechanical degrees of freedom (including translation in three orthogonal axes plus yaw and pitch of the scope tip). In actual implementations, the endoscope may have 6 mechanical degrees of freedom (including translation in three orthogonal axes plus yaw, pitch, and roll of the scope tip). However, because the endoscope is a cylindrical instrument, the roll of the scope tip may be difficult to detect in the image space due to the limited resolution of existing imaging technologies. Aspects of the present disclosure recognize that, if the roll of the scope tip can be determined in both the EM space and the image space, a medical system may determine the transformation matrix using only the pose of the scope tip itself. In some implementations, a medical system may determine the roll of the scope tip when a working channel is inserted therein.

[0153] FIG.15 shows an example medical instrument 1500, according to some implementations. The medical instrument 1500 includes an endoscope 1510 having an inner diameter (or lumen) in which a working channel 1520 is disposed. As shown in FIG.15, a sensor load 1502 is coupled to the working channel 1520 so that the center of the working channel 1520 is offset relative to the center of the endoscope 1510. As used herein, the term “sensor load” may refer to any electrical wires, circuitry, or other sensor components associated with a sensing system.

[0154] As described with reference to FIGS.1–14, a medical system (such as any of the medical systems 100, 1000, 1200, 1300, or 1400 of FIGS.1, 10, 12, 13, and 14, respectively) may determine the position and heading of the endoscope 1510 based on an EM sensor disposed on the scope tip (not shown for simplicity). In some implementations, the medical system may further detect the offset between the working channel 1520 and the endoscope 1510, and thus the roll of the endoscope 1510, in the sensor space based on one or more sensors disposed on the endoscope 1510 or the working channel 1520. The medical system also may detect the offset between the working channel 1520 and the endoscope 1510 in the image space by segmenting image data captured via an imaging system. As a result, the medical system may determine a transformation matrix between the image space and the sensor space using only the position and orientation (including the yaw, pitch, and roll) of the scope tip in each of the coordinate spaces.

[0155] FIG.16 shows a block diagram of an example controller 1600 for a medical system, according to some implementations. In some implementations, the controller 1600 may be one example of the navigation system 400 of FIG.4. More specifically, theAUR6338WOPCT1 PCT controller 1600 is configured to generate and / or update a graphical interface that depicts or otherwise indicates a spatial relationship between an instrument and a target within an anatomy.

[0156] The controller 1600 includes a communication interface 1610, a processing system 1620, and a memory 1630. The communication interface 1610 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1610 includes an image source interface (I / F) 1612 for communicating with one or more image sources (such as the CT imaging system 310 and / or the fluoroscopy imaging system 312 of FIG.3) and a sensor interface (I / F) 1614 for communicating with one or more sensors (such as the EM sensor system 306 of FIG.3). In some implementations, the image source I / F 1612 may receive image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy. In some implementations, the sensor I / F 1614 may receive sensor data via one or more sensors associated with a sensor system, where the one or more sensors include at least a first sensor disposed on the instrument.

[0157] The memory 1630 may include a non-transitory computer-readable medium (including one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, or a hard drive, among other examples) that may store the following software (SW) modules: a graphical interface SW module 1632 to generate a graphical interface depicting a spatial relationship between the instrument and the target; a registration SW module 1634 to determine a mapping between a first coordinate space associated with the imaging system and a second coordinate space associated with the sensor system based at least in part on the image data and the sensor data; and an interface update SW module 1636 to update the graphical interface to depict an updated spatial relationship between the instrument and the target based at least in part on the mapping between the first coordinate space and the second coordinate space. Each of the software modules 1632–1636 includes instructions that, when executed by the processing system 1620, causes the controller 1600 to perform the corresponding functions.

[0158] The processing system 1620 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the controller 1600 (such as in the memory 1630). For example, the processing system 1620 may execute the graphical interface SW module 1632 to generate a graphical interface depicting a spatial relationship between the instrument and the target. The processing system 1620 also may execute the registration SW module 1634 to determine a mapping between a firstAUR6338WOPCT1 PCT coordinate space associated with the imaging system and a second coordinate space associated with the sensor system based at least in part on the image data and the sensor data. The processing system 1620 may further execute the interface update SW module 1636 to update the graphical interface to depict an updated spatial relationship between the instrument and the target based at least in part on the mapping between the first coordinate space and the second coordinate space.

[0159] FIG.17 shows another block diagram of an example controller 1700 for a medical system, according to some implementations. In some implementations, the controller 1700 may be one example of the registration system 900 of FIG.9. More specifically, the controller 1700 is configured to register a sensor system with an imaging system during an intraoperative phase of a medical procedure to facilitate real-time navigation of an instrument within an anatomy.

[0160] The controller 1700 includes a communication interface 1710, a processing system 1720, and a memory 1730. The communication interface 1710 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1710 includes an image source interface (I / F) 1712 for communicating with one or more image sources (such as the CT imaging system 310 and / or the fluoroscopy imaging system 312 of FIG.3) and a sensor interface (I / F) 1714 for communicating with one or more sensors (such as the EM sensor system 306 of FIG.3). In some implementations, the image source I / F 1712 may receive image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy. In some implementations, the sensor I / F 1714 may receive sensor data from a sensor disposed on the instrument.

[0161] The memory 1730 may include a non-transitory computer-readable medium (including one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, or a hard drive, among other examples) that may store the following software (SW) modules: a sensor space positioning SW module 1732 to determine a position of the instrument in a first coordinate space based on the received sensor data; an image space positioning SW module 1734 to determine a position of the instrument in a second coordinate space based on the received image data; and a registration SW module 1736 to determine a mapping between the first coordinate space and the second coordinate space based at least in part on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space. Each of the software modules 1732–1736AUR6338WOPCT1 PCT includes instructions that, when executed by the processing system 1720, causes the controller 1700 to perform the corresponding functions.

[0162] The processing system 1720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the controller 1700 (such as in the memory 1730). For example, the processing system 1720 may execute the sensor space positioning SW module 1732 to determine a position of the instrument in a first coordinate space based on the received sensor data. The processing system 1720 also may execute the image space positioning SW module 1734 to determine a position of the instrument in a second coordinate space based on the received image data. The processing system 1720 may further execute the registration SW module 1736 to determine a mapping between the first coordinate space and the second coordinate space based at least in part on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space.

[0163] FIG.18 shows an illustrative flowchart depicting an example operation 1800 for navigating an instrument within an anatomy, according to some implementations. In some implementations, the example operation 1800 may be performed by a controller for a medical system such as the controller 1600 of FIG.16 or the navigation system 400 of FIG. 4.

[0164] The controller may generate a graphical interface depicting a spatial relationship between the instrument and a target within the anatomy (1802). The controller may receive first image data captured by a first imaging system external to the anatomy while the instrument is disposed within the anatomy (1804). In some implementations, the first imaging system may be a cone beam computed tomography (CBCT) system. The controller may receive first sensor data via one or more sensors associated with a sensor system, where the one or more sensors include at least a first sensor disposed on the instrument (1806). The controller may determine a mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system based at least in part on the first image data and the first sensor data (1808). The controller may further update the graphical interface to depict an updated spatial relationship between the instrument and the target based at least in part on the mapping between the first coordinate space and the second coordinate space (1810).

[0165] FIG.19 shows an illustrative flowchart depicting an example operation 1900 for registering an image space with a sensor space, according to some implementations. In some implementations, the example operation 1900 may be performed by a controller for aAUR6338WOPCT1 PCT medical system such as the controller 1700 of FIG.17 or the registration system 900 of FIG. 9.

[0166] The controller may receive sensor data from a sensor disposed on an instrument within an anatomy (1902). The controller may determine a position of the instrument in a first coordinate space based on the received sensor data (1904). The controller may receive image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy (1906). In some implementations, the first imaging system may be a CBCT system. The controller may determine a position of the instrument in a second coordinate space based on the received image data (1908). The controller may determine a mapping between the first coordinate space and the second coordinate space based at least in part on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space (1910).

[0167] Those of skill in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0168] The various illustrative logics, logical blocks, modules, circuits and algorithm processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. The interchangeability of hardware and software has been described generally, in terms of functionality, and illustrated in the various illustrative components, blocks, modules, circuits and processes described herein. Whether such functionality is implemented in hardware or software depends upon the particular application and design constraints imposed on the overall system.

[0169] In the foregoing specification, implementations have been described with reference to specific examples thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader scope of the disclosure as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.

[0170] As used herein, a phrase referring to “at least one of” a list of items refers to any combination of those items, including single members. As an example, “at least one of: a, b, or c” is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.AUR6338WOPCT1 PCT

[0171] Various modifications to the implementations described in this disclosure may be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the spirit or scope of this disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but are to be accorded the widest scope consistent with this disclosure, the principles and the novel features disclosed herein.

Claims

AUR6338WOPCT1 PCT CLAIMS What is claimed is:

1. A controller for a medical system, comprising: a processing system; and a memory storing instructions that, when executed by the processing system, cause the controller to: generate a graphical interface depicting a spatial relationship between the instrument and a target within the anatomy; receive first image data captured by a first imaging system external to the anatomy while the instrument is disposed within the anatomy; receive first sensor data via one or more sensors associated with a sensor system, the one or more sensors including at least a first sensor disposed on the instrument; determine a mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system based at least in part on the first image data and the first sensor data; and update the graphical interface to depict an updated spatial relationship between the instrument and the target based at least in part on the mapping between the first coordinate space and the second coordinate space.

2. The controller of claim 1, wherein the generating of the graphical interface comprises: receiving second image data via a second imaging system external to the anatomy; receiving second sensor data via the first sensor disposed on the instrument; determining a mapping between the second coordinate space and a third coordinate space associated with the second imaging system based at least in part on the second image data and the second sensor data; and determining the spatial relationship between the instrument and the target based at least in part on the mapping between the second coordinate space and the third coordinate space.

3. The controller of claim 2, wherein the updating of the graphical interface comprises:AUR6338WOPCT1 PCT determining a position of the target in the third coordinate space based on the second image data; determining a position of the target in the first coordinate space based on the received first image data; determining a transformation that transforms the position of the target in the first coordinate space to the position of the target in the third coordinate space based on the mapping between the first coordinate space and the second coordinate space and the mapping between the second coordinate space and the third coordinate space; and applying a correction associated with the transformation to subsequent sensor data received via the first sensor disposed on the instrument.

4. The controller of claim 2, wherein execution of the instructions further causes the controller to: determine a position of the target in the first coordinate space based on the first image data; transform the position of the target from the first coordinate space to the third coordinate space based on the mapping between the first coordinate space and the second coordinate space and the mapping between the second coordinate space and the third coordinate space; and determine the updated spatial relationship between the instrument and the target based at least in part on the position of the target in the third coordinate space.

5. The controller of claim 2, wherein the generating of the graphical interface further comprises: generating a first model of the anatomy based on the second image data; and displaying the spatial relationship between the instrument and the target on the first model of the anatomy.

6. The controller of claim 5, wherein the updating of the graphical interface comprises: reconstructing at least a portion of the first model of the anatomy based on the first image data.AUR6338WOPCT1 PCT 7. The controller of claim 5, wherein the updating of the graphical interface comprises: generating a second model of the anatomy based on the first image data; and displaying the updated spatial relationship between the instrument and the target on the second model of the anatomy.

8. The controller of claim 2, wherein the generating of the graphical interface comprises: determining a path for navigating the instrument within the anatomy based at least in part on the second image data.

9. The controller of claim 8, wherein the updating of the graphical interface comprises: updating the path for navigating the instrument within the anatomy based at least in part on the first image data.

10. The controller of claim 1, wherein the updating of the graphical interface comprises: determining a pose of the instrument in the first coordinate space based on the first sensor data received via the first sensor; determining a corrected pose of the instrument in the second coordinate space based on the first image data; transforming the corrected pose of the instrument from the second coordinate space to the first coordinate space based on the mapping between the first coordinate space and the second coordinate space; determining a correction transformation that transforms the pose of the instrument in the first coordinate space to the corrected pose of the instrument in the first coordinate space; and applying the correction transformation to subsequent sensor data received via the first sensor disposed on the instrument.

11. The controller of claim 10, wherein the first sensor is an electromagnetic (EM) sensor disposed within an EM field produced by a field generator coupled to a supportAUR6338WOPCT1 PCT structure at a known angle, the support structure having a planar surface which supports the anatomy.

12. The controller of claim 11, wherein the determining of the mapping between the first coordinate space and the second coordinate space comprises: estimating, in each of the first coordinate space and the second coordinate space, a first vector orthogonal to the planar surface of the support structure; estimating, in each of the first coordinate space and the second coordinate space, a second vector parallel to the planar surface of the support structure; and determining a transformation matrix that transforms the pose of the instrument and the estimated first and second vectors in the first coordinate space to the corrected pose of the instrument and the estimated first and second vectors, respectively, in the second coordinate space.

13. The controller of claim 11, wherein the field generator is further coupled to a fiducial board including one or more fiducial markers disposed at predetermined positions relative to the field generator, execution of the instructions further causing the controller to: determine a position of each of the one or more fiducial markers in the first coordinate space based on the first image data, the mapping between the first coordinate space and the second coordinate space further being determined based at least in part on the positions of the one or more fiducial markers in the first coordinate space and the predetermined positions of the one or more fiducial markers relative to the field generator.

14. The controller of claim 1, further comprising: receiving additional sensor data from one or more additional sensors proximate to the anatomy; determining positions of the one or more additional sensors in the second coordinate space based on the additional sensor data; and determining positions of the one or more additional sensors in the first coordinate space based on the first image data, the mapping between the first coordinate space and the second coordinate space further being determined based at least in part on the positions of the one or more additional sensors in the first coordinate space and the positions of the one or more additional sensors in the second coordinate space.AUR6338WOPCT1 PCT 15. The controller of claim 14, wherein at least one of the one or more additional sensors is disposed on the instrument.

16. The controller of claim 14, wherein at least one of the one or more additional sensors is positioned at a predetermined location external to the anatomy.

17. A controller for a medical system, comprising: a processing system; and a memory storing instructions that, when executed by the processing system, cause the controller to: receive sensor data from a sensor disposed on an instrument within an anatomy; determine a position of the instrument in a first coordinate space based on the received sensor data; receive image data captured by an imaging system external to the anatomy while the instrument is disposed within the anatomy; determine a position of the instrument in a second coordinate space based on the received image data; and determine a mapping between the first coordinate space and the second coordinate space based at least in part on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space.

18. The controller of claim 17, wherein the sensor is an electromagnetic (EM) sensor disposed within an EM field produced by a field generator coupled to a support structure at a known angle, the support structure having a planar surface which supports the anatomy.

19. The controller of claim 18, wherein the determining of the mapping between the first coordinate space and the second coordinate space comprises: estimating, in each of the first coordinate space and the second coordinate space, a first vector orthogonal to the planar surface of the support structure; estimating, in each of the first coordinate space and the second coordinate space, a second vector parallel to the planar surface of the support structure; andAUR6338WOPCT1 PCT determining a transformation matrix that transforms the position of the instrument and the estimated first and second vectors in the second coordinate space to the position of the instrument and the estimated first and second vectors, respectively, in the first coordinate space.

20. The controller of claim 19, wherein the determining of the transformation matrix comprises: determining a heading of the instrument in the first coordinate space based on the received sensor data; determining a heading of the instrument in the second coordinate space based on the received image data; and determining a rotation matrix and a translation matrix that transform the position of the instrument, the heading of the instrument, and the first and second vectors in the second coordinate space to the position of the instrument, the heading of the instrument, and the first and second vectors, respectively, in the first coordinate space.

Images