Navigation update for medical systems

Abstract

The present disclosure provides methods, devices, and systems for planning and performing medical procedures. Embodiments of the invention more specifically relate to techniques for navigating an instrument to a target within an anatomical structure. In some aspects, a controller for a medical system can generate a graphical interface depicting a spatial relationship between an instrument and a target, and update the graphical interface to depict an updated spatial relationship between the instrument and the target based on sensor data received via a sensor disposed on the instrument and image data captured by an imaging system external to the anatomical structure while the instrument is disposed within the anatomical structure. More specifically, the controller can determine a mapping between a sensor space and an image space based on the sensor data and the image data, and can determine the updated spatial relationship based on the mapping between the sensor space and the image space.

Description

Cross-references to related applications

[0001] This patent application claims priority to U.S. Non-Provisional Patent Application No. 19 / 006,086, filed December 30, 2024, entitled “REGISTRATION OF IMAGING SYSTEM WITH SENSOR SYSTEM FOR INSTRUMENT NAVIGATION,” which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 571,768, filed March 29, 2024, entitled “REGISTRATION OF IMAGING SYSTEM WITH SENSOR SYSTEM FOR INSTRUMENT NAVIGATION,” pursuant to 35 USC § 119(e). This patent application also claims priority to U.S. Nonprovisional Patent Application No. 19 / 034,431, filed January 22, 2025, entitled “NAVIGATION UPDATES FOR MEDICAL SYSTEMS,” which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 571,909, filed March 29, 2024, entitled “INTRAOPERATIVE NAVIGATION UPDATES FOR MEDICAL SYSTEMS,” pursuant to 35 USC § 119(e). The disclosures of these earlier applications are considered part of this patent application and are incorporated herein by reference. Technical Field

[0002] This disclosure relates in general to medical systems, and more specifically to intraoperative navigation updates for medical systems. Background Technology

[0003] Many medical procedures include steps that can be performed preoperatively (also known as the “preoperative phase”), intraoperatively (also known as the “intraoperative phase”), or postoperatively (also known as the “postoperative phase”). For example, during the preoperative phase, imaging systems can be used to scan or otherwise capture images or videos of the patient’s anatomy. Suitable example imaging techniques include computed tomography (CT), X-ray, fluoroscopy, positron emission tomography (PET), PET-CT, CT angiography, cone-beam computed tomography (CBCT), three-dimensional rotational angiography (3DRA), single-photon emission computed tomography (SPECT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, among others. During the intraoperative phase, images can be used to help guide or navigate medical devices to a target within the patient’s anatomy (also known as the “treatment site”). However, images acquired during the preoperative phase may not accurately reflect the spatial relationship between the medical device and the target during the intraoperative phase. For example, among various other factors, preoperative images are typically acquired days (or even weeks) before the intraoperative phase, so changes in the patient’s anatomy can lead to deviations in the spatial positioning of the target. Therefore, more accurate information about the spatial relationship between the medical device and the target is needed during the intraoperative phase. Summary of the Invention

[0004] The present invention is presented in a simplified form to introduce some concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to limit the scope of the subject matter protected by the claims.

[0005] One innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to: generate a graphical interface depicting the spatial relationship between an instrument and a target within an anatomical structure; receive first image data captured by a first imaging system outside the anatomical structure when the instrument is positioned within the anatomical structure; receive first sensor data via one or more sensors associated with a sensor system, wherein the one or more sensors include at least a first sensor mounted on the instrument; determine, at least in part, a mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system based on the first image data and the first sensor data; and update the graphical interface, at least in part, based on the mapping between the first and second coordinate spaces, to depict the updated spatial relationship between the instrument and the target.

[0006] Another innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to: receive sensor data from sensors on an instrument disposed within an anatomical structure; determine the position of the instrument in a first coordinate space based on the received sensor data; receive image data captured by an imaging system outside the anatomical structure while the instrument is disposed within the anatomical structure; determine the position of the instrument in a second coordinate space based on the received image data; and determine a mapping between the first and second coordinate spaces based at least in part on the positions of the instrument in the first and second coordinate spaces. Attached Figure Description

[0007] Embodiments of the present invention are shown by way of example and are not intended to be limited to the figures in the accompanying drawings.

[0008] Figure 1 An example medical system according to some implementation methods is shown.

[0009] Figure 2 The following are illustrated according to some embodiments. Figure 1 Example components of control systems and robotic systems.

[0010] Figure 3 A block diagram of an example positioning system according to some implementations is shown.

[0011] Figure 4 A block diagram of an example navigation system according to some implementations is shown.

[0012] Figure 5 An example spatial relationship between the skeleton image space, sensor space, and intraoperative image space is shown.

[0013] Figure 6 An example update of the target position relative to the instrument posture is shown.

[0014] Figure 7 An example update of the device posture relative to the target position is shown.

[0015] Figure 8 Example corrections to the instrument pose and target position in their respective coordinate spaces are shown, as well as updates to the spatial relationship between the instrument and the target in a common coordinate space.

[0016] Figure 9 A block diagram of an example registration system according to some implementations is shown.

[0017] Figure 10 Another example medical system according to some implementation methods is shown.

[0018] Figure 11 An example mapping between imaging space and sensor space according to some implementations is shown.

[0019] Figure 12 Another example medical system according to some implementation methods is shown.

[0020] Figure 13 Another example medical system according to some implementation methods is shown.

[0021] Figure 14 Another example medical system according to some implementation methods is shown.

[0022] Figure 15 An example medical device according to some implementation methods is shown.

[0023] Figure 16 A block diagram of an example controller for a medical system according to some implementations is shown.

[0024] Figure 17 Another block diagram of an example controller for a medical system according to some implementations is shown.

[0025] Figure 18 An exemplary flowchart depicting example operations of a navigation instrument within an anatomical structure according to some embodiments is shown.

[0026] Figure 19 An exemplary flowchart depicting example operations for registering image space with sensor space according to some embodiments is shown. Detailed Implementation

[0027] In the following description, numerous specific details, such as examples of particular components, circuits, and processes, are set forth to provide a full understanding of this disclosure. As used herein, the term “coupled” means directly connected to or connected via one or more intermediate components or circuits. The terms “electronic system” and “electronic device” are used interchangeably to refer to any system capable of processing information electronically. Furthermore, specific terminology is set forth in the following description and for purposes of explanation to provide a full understanding of various aspects of this disclosure. However, it will be apparent to those skilled in the art that practical exemplary embodiments may not require these specific details. In other instances, well-known circuits and apparatuses are shown in block diagram form to avoid obscuring this disclosure. Some portions of the following detailed description are presented in terms of processes, logic blocks, handling, and other symbolic representations of operations on data bits within computer memory.

[0028] These descriptions and representations are used by those skilled in the art of data processing to most effectively communicate the essence of their work to others skilled in the art. In this disclosure, processes, logic blocks, procedures, etc., are considered as a self-consistent sequence of steps or instructions that produce a desired result. A step is a physical operation requiring physical quantities. Typically, but not necessarily, these quantities are in the form of electrical or magnetic signals that can be stored, transmitted, combined, compared, and otherwise manipulated in a computer system. However, it should be remembered that these and similar terms will be associated with appropriate physical quantities and are merely convenient labels applied to those quantities.

[0029] Unless otherwise expressly indicated in the following discussion, it should be understood that throughout this application, discussions using terms such as “access,” “receive,” “send,” “use,” “select,” “determine,” “normalize,” “multiply,” “average,” “monitor,” “compare,” “apply,” “update,” “measure,” and “derive” refer to the actions and processes of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in the registers and memories of the computer system and transforms that data into other data similarly represented as physical quantities in the computer system’s memory or registers or other such information storage, transmission, or display devices.

[0030] Regarding the example implementations, certain standard positional anatomical terms may be used herein to refer to anatomical structures of animals (i.e., humans). Although certain spatially relative terms (such as “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” “top,” “bottom,” and similar terms) are used herein to describe the spatial relationship of one element, device, or anatomical structure to another, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements and structures, as illustrated in the accompanying figures. It should be understood that spatially relative terms are intended to cover different orientations of an element or structure in use or operation other than those depicted in the figures. For example, an element or structure described as “above” another element or structure may indicate a position below or beside such other element or structure relative to an alternative orientation of the subject patient, element, or structure, and vice versa. As used herein, the term “patient” can generally refer to a human, anatomical model, simulator, cadaver, and other living or non-living object.

[0031] In the accompanying drawings, a single block may be described as performing one or more functions; however, in practice, the one or more functions performed by that block may be performed in a single component or across multiple components, or may be performed using hardware, software, or a combination of hardware and software. To clearly illustrate this interchangeability between hardware and software, various exemplary components, blocks, modules, circuits, and steps are generally described below in relation to their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure. Furthermore, the example system or apparatus may include components other than those shown, including well-known components such as processors and memory.

[0032] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any feature described as a module or component may also be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques may be implemented at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed, perform one or more of the methods described herein. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include encapsulation material.

[0033] Non-transitory processor-readable storage media may include random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, and other known storage media. Alternatively or additionally, these techniques may be implemented at least in part by a processor-readable communication medium that carries or transmits code in the form of instructions or data structures and can be accessed, read, or executed by a computer or other processor.

[0034] The various exemplary logic blocks, modules, circuits, and instructions described in conjunction with the embodiments disclosed herein can be executed by one or more processors (or processing systems). As used herein, the term "processor" can refer to any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

[0035] As mentioned above, many medical procedures include a preoperative phase preceding the intraoperative phase. During the preoperative phase, for some medical procedures, imaging systems may be used to scan or otherwise capture images or videos of at least a portion of the patient's anatomy. For example, a computed tomography (CT) scanner may be used to acquire tomographic images (also known as "tomograms" or "CT scans") of the patient's lungs during the preoperative phase of bronchoscopy. A tomogram is a cross-section or slice of a three-dimensional (3D) volume. For example, multiple tomograms may be stacked or combined to recreate a 3D volume (such as a 3D model of the patient's lungs). Thus, tomograms can be used to detect the precise location or localization (in 3D space) of nodules or targets in the patient's lungs. During the intraoperative phase, for some medical procedures, medical systems may use preoperative images to generate a graphical interface for navigating medical instruments within the patient's anatomy. For example, during a bronchoscopy, the medical system can detect the endoscope's posture (such as its position and orientation in 3D space) based on sensor data received via an electromagnetic (EM) sensor located at the end of the endoscope, and map the endoscope's posture onto a 3D model of the patient's lungs depicted by tomographic images.

[0036] Therefore, a graphical interface can depict the spatial relationship between a medical device and a target within an anatomical structure based on sensor and image data. However, images acquired during the preoperative phase may not accurately reflect the spatial relationship between the medical device and the target during the intraoperative phase. For example, changes in patient anatomy or the medical environment can cause the spatial relationship between the endoscope and the target to deviate from the spatial relationship depicted by the graphical interface at any given time, potentially leading to inaccurate navigation. Example factors include EM distortion, poor registration (or mapping) between sensor space and the image space associated with the preoperative scan (also referred to as the "preoperative image space"), outdated preoperative scans, anatomical distortions, and others. Various aspects of this disclosure recognize that some modern imaging techniques, such as cone-beam CT, can be used to scan a patient's anatomy during the intraoperative phase. In some aspects, medical systems can capture updated images of the patient's anatomy during the intraoperative phase and use the updated image data to improve the representation of the spatial relationship between the medical device and the target.

[0037] Updated image data and sensor data are typically associated with different coordinate spaces. Therefore, in some implementations, medical systems can “register” the updated image space with the sensor space to facilitate real-time navigation. As used herein, the term “registration” refers to a mapping or transformation between different coordinate spaces. For example, a medical system can register an imaging system (such as a cone-beam CT scanner) for capturing images of a patient’s anatomy with a sensor system (such as an EM field generator) for tracking the posture of a medical device within an anatomy by determining a mapping or spatial transformation (such as a transformation matrix) that maps any point or vector in the image space to a corresponding point or vector in the sensor space. The terms “mapping,” “transformation,” “spatial transformation,” and “registration matrix” are used interchangeably herein. The terms “corresponding” and “corresponding” are also used interchangeably herein.

[0038] Although certain aspects of this disclosure have been described in detail herein in the context of bronchoscopy, it should be understood that the systems and techniques of this disclosure can be applied to any medical procedure. Example medical procedures may include minimally invasive procedures (such as laparoscopy), non-invasive procedures (such as endoscopy), therapeutic procedures, diagnostic procedures, percutaneous procedures, and non-percutaneous procedures, among others. Example endoscopic procedures include bronchoscopy, ureteroscopy, gastroscopy, nephroscopy, and nephrolithotomy, among others. The terms “scope,” “endoscope,” “catheter,” and “instrument” are used interchangeably herein.

[0039] The aspects of this disclosure can be used to perform robot-assisted medical procedures, such as endoscopic access, percutaneous access, or treatment of target anatomical sites. For example, a robotic tool can engage or control one or more medical instruments (such as an endoscope) to access a target site within a patient's anatomy or to perform treatment at that site. In some embodiments, the robotic tool may be guided or controlled by a physician. In some other embodiments, the robotic tool may operate autonomously or semi-autonomously. Although systems and techniques are described herein in the context of robot-assisted medical procedures, these systems and techniques can be applied to other types of medical procedures (such as those that do not rely on robotic tools or utilize robotic tools only with very limited capabilities). For example, the systems and techniques described herein can be applied to medical procedures that rely on manually operated medical instruments (such as endoscopes that are specifically controlled and operated by a physician). The systems and techniques described herein can also be applied outside the context of medical procedures (such as in simulated environments or laboratory settings, such as using models or simulators, and other examples).

[0040] Figure 1 An example medical system 100 (also referred to as a "surgical medical system" or "robotic medical system") according to some implementations is shown. Figure 1As shown, medical system 100 can be arranged for diagnostic or therapeutic bronchoscopy. Medical system 100 may include and utilize robotic system 102, which may be implemented as, for example, a robotic cart. Although medical system 100 is shown as including various cart-based systems or devices, the concepts disclosed herein can be implemented in any type of robotic system or arrangement, such as robotic systems employing track-based components, table-based robotic end effectors or manipulators, and other examples. Robotic system 102 may include one or more robotic arms 104 (also referred to as “robotic locators”) configured to locate or otherwise manipulate medical device 106 (such as a manipulatory endoscope or another elongated instrument). For example, medical device 106 may be advanced through a natural orifice entry point (such as the oral cavity 108 of patient 110 positioned on table 112) to deliver diagnostic or therapeutic treatment. Although described in the context of bronchoscopy procedures, medical system 100 can also be used to perform other types of medical procedures. Suitable example protocols include gastrointestinal (GI) protocols, renal protocols, urological protocols, and nephrology protocols, as well as other examples.

[0041] With the robotic system 102 correctly positioned, the medical device 106 can be inserted into the patient 110 by robot, by manual means, or a combination thereof. For example, one or more robotic arms 104, or instrument actuators 114 coupled thereto, can control the medical device 106. In some embodiments, the medical device 106 can be advanced within a sheath 116. For example, the sheath 116 can be coupled to or controlled by the robotic arm 104. In some embodiments, the medical device 106 and the sheath 116 can each be coupled to a corresponding instrument actuator from a set of instrument actuators 114. The instrument actuators 114 can be repositioned in space by manipulating one or more robotic arms 104 to different angles or positions.

[0042] exist Figure 1 In some examples, the medical device 106 can be guided downwards along the patient's trachea and lungs after insertion, or advanced to a target destination or surgical site. In some embodiments, to enhance navigation through the patient's lung network or to reach a desired target, the medical device 106 can be manipulated to extend telescopically from the external sheath 116 to obtain enhanced joint movement and a larger bending radius. Using a separate device actuator 114 allows the medical device 106 and the sheath 116 to be driven independently of each other.

[0043] In some embodiments, the medical device 106 may include an elongated member or shaft configured to insert or retract within an anatomical structure, perform articulation, or otherwise move. Additionally, in some embodiments, the medical device 106 may include one or more imaging devices (such as cameras) positioned at the distal end of the elongated shaft or deployed through a working channel of the elongated shaft. The imaging devices may be configured to generate or capture image (or video) data or transmit image data to another device or component. In some embodiments, the medical device 106 may include an instrument base or one or more handles positioned at the proximal end of the medical device 106. The instrument base may be coupled to a manipulator (such as the end of a robotic arm 104). The instrument base may include one or more drive inputs coupled to one or more drive outputs of the manipulator, wherein the drive inputs or drive outputs act as interfaces.

[0044] In some embodiments, the medical device 106 may include a working channel configured to receive one or more other instruments or elements or provide other functions therein. The working channel may extend axially, such as along the length of the medical device 106. Furthermore, the medical device 106 may include or be associated with one or more elongated moving members (such as a drawstring) extending from a proximal end through an elongated shaft to a distal end of the elongated shaft. The elongated moving members may be manipulated (such as by manipulators on one or more robotic arms 104) to control actuation of the elongated moving members.

[0045] In some embodiments, medical device 106 may include one or more sensors, such as electromagnetic (EM) sensors, shape sensors (such as shape-sensing optical fibers), accelerometers, gyroscopes, satellite-based positioning sensors (such as Global Positioning System (GPS) sensors), or radio frequency (RF) transceivers, among other examples. Sensors may be configured to generate or produce sensor data or to provide sensor data to another device or component. Sensors may be located at the distal end of an elongated shaft or along its length. In some embodiments, medical device 106 may be configured to receive an elongated member or device via a working channel, wherein the elongated member includes one or more sensors along its length. One or more sensors on medical device 106 may provide sensor data to control circuitry of medical system 100, which is then used to determine the position, orientation, or shape of medical device 106.

[0046] The medical system 100 may also include a control system 118 (also referred to as a “control tower” or “mobile tower”). The control system 118 may be communicatively coupled (e.g., via a wired or wireless connection) to the robotic system 102 to control various aspects of the robotic system 102 (such as electronics, optics, sensors, or power) or one or more subsystems associated with the robotic system 102 (such as a fluid management system (not shown)). Placing such functionality within the control system 118 allows for a smaller form factor of the robotic system 102, which can be more easily adjusted or repositioned by an operator or user. Furthermore, the functional separation between the robotic system 102 and the control system 118 can reduce operating room clutter and facilitate efficient clinical workflows.

[0047] Medical system 100 may include an electromagnetic (EM) field generator 120 configured to broadcast or transmit an EM field detectable by various EM sensors, such as sensors disposed on medical device 106. The EM field can induce a small current in the coil of the EM sensor, which can be analyzed to determine the position, angle, or orientation of the EM sensor relative to the EM field generator 120. Although EM fields and EM sensors are described in many examples herein, position sensing systems or sensors may include various other types of position sensing systems or sensors, such as optical position sensing systems or sensors, image-based position sensing systems or sensors, and other examples.

[0048] Medical system 100 may also include imaging system 122 (also referred to as an “imaging device”) configured to generate image data (also referred to as an “image”), provide that image data to, or transmit it to, another device or system. For example, imaging system 122 may generate image data depicting the anatomical structures of patient 110 and provide the image data to control system 118, robotic system 102, or another device. Imaging system 122 may include a transmitter or energy source (such as an X-ray source) or detector (such as an X-ray detector) mounted on C-arm support 124, allowing flexibility in positioning around patient 110 to capture images from various angles without moving patient 110. The use of imaging system 122 can provide visualization of internal structures or anatomy, which can be used for a variety of purposes, including navigation of medical device 106 (such as by providing the user with images of internal anatomy) and positioning of medical device 106 (based on image data analysis), among others. In some aspects, imaging system 122 can enhance the effectiveness or safety of medical procedures (such as bronchoscopy) by providing clear, continuous visual feedback to the operating physician or team.

[0049] In some implementations, the imaging system 122 may be a mobile device configured to move around within the environment. For example, the imaging system 122 may be positioned next to the patient 110 during a specific phase of the procedure (e.g., Figure 1 As shown), and is removed when imaging system 122 is no longer needed. In some other embodiments, imaging system 122 may be part of workbench 112 or other equipment in the operating environment. Imaging system 122 may be implemented as a computed tomography (CT) machine or system, X-ray machine or system, fluoroscopy machine or system, positron emission tomography (PET) machine or system, PET-CT machine or system, CT angiography machine or system, cone-beam computed tomography (CBCT) machine or system, three-dimensional rotational angiography (3DRA) machine or system, single-photon emission computed tomography (SPECT) machine or system, magnetic resonance imaging (MRI) machine or system, optical coherence tomography (OCT) machine or system, or ultrasound machine or system, and other examples. In some embodiments, medical system 100 may include different types of imaging systems that may be used or positioned on patient 110 at different stages or portions of the procedure as needed.

[0050] In some implementations, the imaging system 122 may be configured to process multiple images (also referred to as “image data”) to generate a three-dimensional (3D) view or model. For example, the imaging device 122 may be implemented as a CT scanner configured to capture or generate a series of images (also referred to as “tomograms”) or image data (which represent two-dimensional (2D) cross-sections or slices of a 3D volume) from different angles around the patient 110, and then use one or more algorithms to reconstruct a 3D model from these images or image data. The 3D model may be provided to the control system 118, the robotic system 102, or another device, such as for processing or display.

[0051] In some embodiments, image data from imaging system 122 can be used to locate various elements, such as medical device 106, targets within an anatomical structure, or specific anatomical features, among others. As used herein, the term "localize, localization, localizing" refers to any process for determining or estimating the position (or orientation) and / or orientation (or heading) (collectively, "posture") of an instrument or target (or any other element) within a given space or environment. For example, control system 118 may be configured to provide navigation information during procedures to assist the user in navigating medical device 106 within an anatomical structure to reach a target (such as a desired treatment site or location). In some embodiments, such as in the context of certain bronchoscopy procedures, the target may include a nodule. For illustration, control system 118 may display navigation views or graphical data 126, including instrument indicators 128 representing medical device 106, target indicators 130 representing the target, and an anatomical map. Navigation data 126(A) (such as initial navigation data) may be determined based on sensor data from sensors of medical device 106 (such as EM sensor data associated with EM field generator 120), mappings of anatomical structures, or the location of the target. In some embodiments, the mapping or location of the target may be determined based on preoperative data (such as data obtained during preoperative procedures) to locate the target location or map the anatomical structure.

[0052] In some implementations, navigation data 126(A) may be dynamically updated based on image data 132 from imaging system 122. For example, control system 118 may receive and analyze image data 132 to determine the current or actual spatial relationship between medical device 106 and a target. In some implementations, control system 118 may display image data 132 to a user, receive user input indicating the position of medical device 106 or the position of a target in image data 132, and analyze image data 132 based on user input to determine the current spatial relationship. If control system 118 determines that navigation data 126(A) does not correctly depict the position of medical device 106 (e.g., the spatial relationship associated with image data 132 differs from the spatial relationship associated with navigation data 126(A)), control system 118 may update navigation data 126(A) at 134 and provide updated navigation data 126(B) reflecting the current or near-real-time position of medical device 106 relative to a target or map.

[0053] The various components of the medical system 100 can be communicatively coupled to each other via a network, which may include a wireless network or a wired network. Example networks include one or more Personal Area Networks (PANs), Local Area Networks (LANs), Wide Area Networks (WANs), Internet Local Area Networks (IANs), cellular networks, the Internet, Body Area Networks (BANs), etc. In some examples, various communication interfaces may include wireless technologies such as Bluetooth, Wi-Fi, Near Field Communication (NFC), etc. Furthermore, in some examples, the various components of the medical system 100 can be connected via one or more support cables, pipes, connectors, etc., for data communication, fluid exchange, power exchange, etc.

[0054] Figure 2 The following are illustrated according to some embodiments. Figure 1 Example components of the control system 118 and the robot system 102. Figure 2 In the example, control system 118 and robot system 102 are implemented as a tower and a robotic cart, respectively. However, control system 118 and robot system 102 can be implemented in other ways. Control system 118 can be coupled to and cooperate with robot system 102 to perform medical procedures. For example, control system 118 may include a communication interface 202 for communicating with robot system 102 via a wireless or wired connection (such as to control robot system 102). In some embodiments, control system 118 can communicate with robot system 102 to receive from it location or sensor data relating to the location of sensors associated with and controlled by instruments or components of the robot system 102. For example, control system 118 can communicate with EM field generator 120 to control the generation of EM fields in the area surrounding the patient. Control system 118 may also include one or more power interfaces 206.

[0055] The control system 118 may include control circuitry 208 configured to actuate or otherwise control any of a variety of system components of the medical system 100, such as brackets, mounts, arms or positioners, medical devices, imaging devices, position sensing devices or sensors, and other examples. Furthermore, control circuitry 208 may be configured to perform other functions, such as inducing the display of information, processing data, receiving input, communicating with other components or devices, or any other function or operation described herein.

[0056] The control system 118 may also include one or more input or output (I / O) components 210 configured to assist a physician or other person in performing medical procedures. For example, one or more I / O components 210 may be configured to receive input or provide output to enable a user to control or navigate the medical device 106, the robotic system 102, or other instruments or devices associated with the medical system 100. The control system 118 may include one or more displays 212 to provide, display, or otherwise present various information regarding the procedures. For example, one or more displays 212 may be used to present navigation information, including virtual anatomical models of anatomical structures with virtual representations of medical devices, image data, or other information. One or more I / O components 210 may include one or more user input controls 214, which may include any type of user input (or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (such as video game type controllers), computer mice, touchpads, trackballs, control pads, sensors for capturing gestures and finger postures (such as motion sensors or cameras), touchscreens, toggle inputs (such as buttons), or interfaces or connectors for this purpose. In some embodiments, such input may be used to generate commands for controlling one or more medical devices, robotic arms, or other components.

[0057] The control system 118 may also include a data storage device 216 configured to store executable instructions (such as computer-executable instructions) that can be executed by the control circuitry 208 to cause the control circuitry 208 to perform the various operations or functions described herein. In some embodiments, the data storage device 216 may also store telemetry or runtime data (such as sensor data or image data) generated by the medical system 100 or otherwise captured or acquired during medical procedures. In some embodiments, two or more components of the control system 118 may be electrically or communicatively coupled to each other.

[0058] The robotic system 102 may include one or more robotic arms 104 configured to engage and / or control, for example, a medical device 106 or other elements or components, to perform one or more aspects of a procedure. Figure 2As shown, each robotic arm 104 may include multiple segments 220 coupled to joints 222, which may provide multiple degrees of motion or freedom. Robotic system 102 may be configured to receive control signals from control system 118 to perform certain operations, such as positioning one or more robotic arms 104 in a particular manner, manipulating instruments, and other examples. In response, robotic system 102 may use its control circuitry 224 to control actuators 226 or other components of robotic system 102 to perform operations. For example, control circuitry 224 may control the insertion or retraction of axes, joint movements, rolling, etc., of medical device 106 or another instrument by actuating one or more drive outputs 228 of a manipulator 230 (or end effector) coupled to the base of a robotically controllable instrument. Drive outputs 228 may be coupled to drive inputs on associated instruments, such as the instrument base of an instrument coupled to an associated robotic arm 104. Robotic system 102 may also include one or more power interfaces 232.

[0059] The robotic system 102 may include a support column 234, a base 236, or a console 238. The console 238 may provide one or more I / O components 240, such as a user interface or display screen (or dual-purpose device, such as a touchscreen) for receiving user input to provide preoperative or intraoperative data to a physician or user. The support column 234 may include arm supports 242 (also referred to as “brackets”) for supporting the deployment of one or more robotic arms 104. The arm supports 242 may be configured to translate vertically along the support column 234. Vertical translation of the arm supports 242 allows the robotic system 102 to adjust the range of the robotic arms 104 to accommodate various table heights, patient shapes, or physician preferences. The base 236 may include wheel-shaped casters 244 (also referred to as “wheels”) that allow the robotic system 102 to move around the operating room. Once in place, the casters 244 may be secured using wheel locks to hold the robotic system 102 in place during procedures.

[0060] Each joint 222 of the robotic arm 104 may be independently controllable or provide independent degrees of freedom that can be used for instrument navigation. In some embodiments, each robotic arm 104 may include seven joints that provide seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom can allow the robotic arm 104 to be controlled to position its respective manipulator 230 at a specific location, orientation, or trajectory in space using different link positions and joint angles. This allows the robotic system 102 to locate and guide medical instruments from desired points in space, while allowing physicians to move joints 222 to clinically advantageous positions away from the patient to create more operating space while avoiding collisions.

[0061] One or more manipulators 230 (or end effectors) may be coupled to an instrument base or handle, which may be attached using a sterile adapter component. The combination of manipulators 230 and the instrument base, along with any intermediate mechanical components or couplings (such as sterile adapters), may be collectively referred to as a manipulator or manipulator assembly. The manipulator or manipulator assembly may provide an electrical or control interface. Example interfaces may include connectors for transmitting pneumatic pressure, electrical, electronic, or optical signals from the robotic arm 104 to the instrument base. The manipulator or manipulator assembly may be configured to manipulate medical devices (such as surgical instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt or pulley drive, or magnetic drive.

[0062] The robot system 102 may also include a data storage device 246 configured to store executable instructions (such as computer-executable instructions) that can be executed by control circuitry 224 to cause control circuitry 224 to perform the various operations or functions described herein. In some embodiments, the data storage device 216 may also store telemetry or runtime data (such as sensor data or image data) generated by the medical system 100 or otherwise captured or acquired during medical procedures. In some embodiments, two or more components of the robot system 102 may be electrically or communicatively coupled to each other.

[0063] Data storage devices (including data storage device 216, data storage device 246, or other data storage devices or memories) may include any suitable or desired type of computer-readable medium. For example, computer-readable media may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, or non-removable data storage devices, which are implemented using any technology, layout, or data structure or protocol (including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data).

[0064] Computer-readable media may include, but is not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media, such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.

[0065] The functions described herein can be implemented by control circuitry 208 of control system 118 or control circuitry 224 of robot system 102, such as by executing instructions from control circuitry 208 or 224 to cause control circuitry 208 or 224 to perform functions. Control circuitry (including control circuitry 208, control circuitry 224, or other control circuitry) may include circuitry embodied in a robot system, control system, tower, machine, or any other component or device. Control circuitry may include any combination of the following: processor, processing circuitry, processing module or unit, chip, die (such as a semiconductor die comprising one or more active or passive devices or interconnecting circuits), microprocessor, microcontroller, digital signal processor, microcomputer, central processing unit, field-programmable gate array, programmable logic device, state machine (such as a hardware state machine), logic circuitry, analog circuitry, digital circuitry, or any device that manipulates signals (analog or digital) based on hard-coded circuitry or operational instructions.

[0066] The control circuitry mentioned herein may also include one or more circuit substrates (such as printed circuit boards), conductive traces and vias, or mounting pads, connectors, or components. The control circuitry may also include one or more storage devices, which may be embodied in a single device, multiple devices, or embedded circuitry of a device. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, or any device for storing digital information. In examples where the control circuitry includes a hardware or software state machine, analog circuitry, digital circuitry, or logic circuitry, the data storage device or register storing any associated operating instructions may be embedded within or outside the circuitry including the state machine, analog circuitry, digital circuitry, or logic circuitry.

[0067] Figure 3 A block diagram of an example positioning system 300 according to some embodiments is shown. Positioning system 300 includes various positioning or imaging systems or modalities 302-312 (also referred to as “subsystems”) that can be implemented, according to one or more examples, to facilitate anatomical mapping, navigation, positioning, or visualization for procedures. For example, the various systems 302-312 can be configured to provide data for generating anatomical maps, determining instrument placement, determining target placement, or performing other techniques.

[0068] Each of systems 302-312 can be associated with a corresponding coordinate space (also referred to as a “position coordinate system”), or can provide data or information related to the position of instruments or anatomical structures. Registering various coordinate spaces with each other allows for the integration of various systems to provide mapping, navigation, or instrument visualization. For example, registering a first modality to a second modality allows for tracking or overlaying of the determined position in the first modality on or within a reference frame associated with the second modality, thereby providing a layer of position information that can be combined to provide a robust positioning system.

[0069] In some aspects, system 300 can be configured to perform one or more positioning techniques. In some embodiments, the anatomical space in which a medical device can be positioned (such as the place where the posture or shape of the device is determined or estimated) can be a 2D or 3D portion of the patient's tracheobronchial airway, vascular system, urinary tract, gastrointestinal tract, or any organ or space accessed via a lumen. Various modalities can be implemented to provide images, representations, or models of the anatomical space. For example, imaging modalities can be implemented, which may include, for example, X-ray, fluoroscopy, CT, PET, PET-CT, CT angiography, CBCT, 3DRA, SPECT, MRI, OCT, or ultrasound, among other examples. In some embodiments, the imaging modalities can be used to capture or acquire images of the patient's anatomical structures during the preoperative phase of a medical procedure. In some other embodiments, the imaging modalities can be used to capture or acquire images of the patient's anatomical structures during the intraoperative phase of a medical procedure.

[0070] System 302-312 can provide information for generating graphical interface 314 (also referred to as "graphical interface (I / F)"), which includes navigation information (such as) for navigating instruments to targets within anatomical structures. Figure 1 The navigation information may include navigation data 126(A) or 126(B). For example, navigation information may include an anatomical map, the estimated location, orientation, and / or shape of the device. Navigation information may also include the shape, boundaries, eccentricity, texture, and / or location of the target. In some embodiments, a graphical user interface 314 or other location information may be displayed to a user (such as a physician) during a medical procedure to assist the user in performing the procedure. For example, visualization of the tracked device may be overlaid on an anatomical map depicted by the graphical user interface 314 based on location or sensor data associated with the tracked medical device.

[0071] like Figure 3As shown, system 300 may include a support structure 302 (such as an operating table or other patient positioning or support platform). For example, support structure 302 includes a flat surface that contacts and supports the patient. In some embodiments, the position of support structure 302 may be known based on maintained data relating to its location within a surgical or procedural environment. In some other embodiments, one or more markers or appropriate imaging or positioning modalities may be used to sense or otherwise determine the position of support structure 302.

[0072] System 300 may also include robot system 304 (such as a robot cart or other device or system including one or more robotic end effectors). In some embodiments, robot system 304 may be Figure 1 and Figure 2 An example of robot system 102. Data relating to the position or state of the robot arm, actuators, or other components of robot system 304 may be known or derived from robot command data or other robot data relative to the coordinate system of robot system 304. In some examples, reference system registration 316 occurs between support structure 302 and robot system 304, and in some implementations, it may be a relatively coarse registration based on robot system or cart setup procedures (which may have any suitable or desired scheme).

[0073] System 300 may also include an electromagnetic (EM) sensor system 306, which may include an EM field generator (such as...) Figure 1 The system includes an EM field generator 120 and one or more EM sensors. The EM sensors may be associated with a portion of the tracked or controlled apparatus, such as the distal end (or distal end) of the apparatus or other elongated members (such as working channels) along the length of the apparatus or disposed within the lumen of the apparatus. In some embodiments, the EM field generator may be mechanically coupled to the support structure 302 or the robot system 304, allowing registration or association 318 between such systems to be known or determined. In some embodiments, registration 318 between the EM sensor system 306 and the robot system 304 may be determined using forward kinematics or field generator mounting transformation information. For example, the field generator may be mounted to the support structure 302, allowing the position of the field generator relative to the robot system positioning frame to be known based on a known relationship between the positions of the support structure 302 and the robot system 304. The EM sensor system 306 may provide apparatus posture or path information based on sensor readings associated with the apparatus.

[0074] System 300 may also include an optical camera system 308 comprising one or more cameras or other imaging devices configured to generate images (such as real-time image data) of patient anatomy within its field of view during surgical procedures. In some embodiments, registration 320 between the optical camera system 308 and the EM sensor system 306 may be achieved by identifying features (such as medical device ends) in the images generated by the optical camera system 308 that have associated EM sensor data. Registration 320 may also be based, at least in part, on hand-eye interactions as a physician views real-time camera images while an endoscope equipped with EM sensors navigates through patient anatomy.

[0075] System 300 may also include a computed tomography (CT) imaging system 310 configured to generate CT images of the patient's anatomy, which may be performed preoperatively or intraoperatively. The CT imaging system 310 is typically used to scan relatively large volumes. In some embodiments, image processing may be implemented for registration 322 of the CT image data with camera image data generated by the optical camera system 308. For example, common features identified in both the camera image data and the CT image data may be recognized to spatially correlate CT image frames with camera image frames. In some examples, the CT imaging system 310 may be used to generate preoperative imaging data for producing a graphical user interface 314 or for path navigation planning.

[0076] In some aspects, the CT imaging system 310 can be registered 326 to the EM sensor system 306 using various techniques. In some embodiments, the mechanical structure of the CT imaging system 310 may have known physical transformations or relationships relative to the mounting location of the EM field generator of the EM sensor system 306. Such known relationships can be used to register the CT image space to the EM sensor space. Connection 328 represents a mapping or relationship between the CT imaging system 310 and the anatomical map depicted by the graphical user interface 314.

[0077] System 300 may also include a fluorescence fluoroscopy imaging system 312 configured to generate tomographic images (such as real-time X-ray images) of the surgical site. Compared to CT imaging system 310, fluorescence fluoroscopy imaging system 312 is typically used to scan smaller volumes. In some embodiments, fluorescence fluoroscopy imaging system 312 may be… Figure 1An example of imaging system 122. For example, fluoroscopic imaging system 312 may include a CBCT scanner coupled to a C-arm. In some embodiments, fluoroscopic imaging system 312 may be used with a contrast agent introduced into anatomical structures to generate image data representing patient anatomy or instruments. In some embodiments, any image processing technique suitable for such registration may be used to register fluoroscopic imaging system 312 324 to CT imaging system 310.

[0078] In some aspects, the fluorescence fluoroscopy imaging system 312 can be registered 332 to the EM sensor system 306 using various techniques. In some embodiments, the mechanical structure of the fluorescence fluoroscopy imaging system 312 (such as a C-arm instrument) may have known physical transformations or relationships relative to the mounting location of the EM field generator of the EM sensor system 306. Such known relationships can be used to register the fluorescence fluoroscopy image space to the EM sensor space. Connection 330 represents a mapping or relationship between the fluorescence fluoroscopy imaging system 312 and the anatomical map depicted by the graphical user interface 314.

[0079] exist Figure 3 In the example, the CT imaging system 310 and the fluoroscopy imaging system 312 are illustrated as separate systems. However, in some other embodiments, a single imaging system may perform the functions of both the CT imaging system 310 and the fluoroscopy imaging system 312.

[0080] The position, shape, or orientation of an instrument (such as an endoscope) can be determined using any one or more of systems 302-312, which can facilitate the generation of graphical interface data representing the estimated position or shape of the instrument relative to an anatomical map depicted by the graphical user interface 314. The graphical user interface 314 can be displayed on a display device, such as via control system 118 or robotic system 102 or another means. In some embodiments, the graphical user interface 314 can also indicate the location of a target within an anatomical structure that has been designated for treatment.

[0081] Although systems 302-312 have been described in a specific order, their associated operations or functions may be performed in a different order. In some embodiments, systems 302-312 may be used in different ways. In some other embodiments, registration may occur between different systems and modes.

[0082] In some respects, one or more of the systems 302-312 can be used to generate a graphical user interface 314 preoperatively, or to determine the location of one or more targets within an anatomical map depicted by the graphical user interface 314 during the preoperative phase of a medical procedure. As used herein, the terms “skeleton image data” and “skeleton image space” refer to images of anatomical structures captured or acquired by an imaging system, such as a CT imaging system 310, during the preoperative phase of a medical procedure. However, a graphical user interface 314 generated using a preoperative CT scan may not accurately reflect the spatial relationship between the medical device and the target during the intraoperative phase. For example, changes in patient anatomy or the medical environment may cause the spatial relationship between the device and the target to deviate from the spatial relationship depicted in the graphical user interface 314. Examples of factors that may cause such deviations include EM distortion, poor registration (or mapping) between sensor data and image data, outdated preoperative scans, and anatomical distortions, among others. Therefore, in some other respects, one or more of the systems 302-312 can be used to determine the location of a medical device or target relative to an anatomical map depicted by the graphical user interface 314 during the intraoperative phase of a medical procedure.

[0083] Figure 4 A block diagram of an example navigation system 400 according to some embodiments is shown. In some aspects, the navigation system 400 may be provided by a medical system (such as...) Figure 1 This can be achieved through a medical system (100). (Refer to...) Figure 2 The navigation system 400 may be an example of either control circuitry 208 or 224. The navigation system 400 is configured to generate or update a graphical interface 404 that depicts or otherwise indicates the spatial relationship between a medical device and a target within an anatomical structure. For example, the graphical interface 404 may be used by the medical system or its user (such as a physician) to navigate the device to a target. In some embodiments, the graphical interface 404 may be… Figure 3 An example of a graphical user interface 314.

[0084] In some respects, navigation systems can be at least partially based on imaging systems (such as...) Figure 3 Image data 402 (also known as “skeleton image data”) acquired by a CT imaging system 310 and transmitted via a sensor system (such as...) Figure 3 The EM sensor system 306 acquires sensor data 401 to generate a graphical interface 404. For example, the skeleton image data 401 may include one or more tomographic images (such as reference images) captured as a result of a CT scan. Figures 1 to 3(As described). In some embodiments, skeletal image data 401 may be captured during the preoperative phase of a medical procedure. Sensor data 401 may indicate the orientation of the instrument and / or one or more EM sensors positioned in the EM field (such as reference). Figures 1 to 3 As described herein, the term "posture" can refer to the position and / or orientation of a sensor or instrument. In some embodiments, the posture of the instrument can be determined at least in part based on sensor data 401 received from sensors disposed on the instrument (such as at the distal end of an endoscope).

[0085] Navigation system 400 includes a skeleton registration component 410, a navigation component 420, an intraoperative registration component 430, and a navigation update component 440. The skeleton registration component 410 is configured to register a coordinate space (also referred to as “sensor space”) associated with sensor data 401 to a coordinate space (also referred to as “skeleton image space”) associated with skeleton image data 402. As used herein, the term “registration” refers to a mapping between different coordinate spaces. In other words, the skeleton registration component 410 can determine a mapping 403, at least in part, based on the sensor data 401 and the skeleton image data 402, that can transform any data point (including point coordinates or vectors) in the sensor space to a corresponding data point (such as a transformation matrix) in the image space.

[0086] Various existing techniques can be used to register skeletal image space with sensor space and vice versa. For example, multiple EM sensors (such as three or more EM sensors) can be positioned around the patient's body at predetermined locations relative to anatomical structures. Skeletal registration component 410 can determine the positions of the EM sensors in the EM field (which represents sensor space) based on received sensor data 401. Skeletal registration component 410 can also determine the corresponding positions of the EM sensors in skeletal image space based on the relative positions of the EM sensors with known anatomical features depicted by skeletal image data 402. Skeletal registration component 410 can also determine a mapping 403 (or transformation matrix) that transforms the positions of the EM sensors in sensor space to their corresponding positions in skeletal image space.

[0087] The navigation component 420 is configured to generate a graphical interface 404 based on sensor data 401, skeletal image data 402, and a mapping 403 between sensor space and skeletal image space. The graphical interface 404 can indicate the spatial relationship between instruments and targets within anatomical structures. For example, the graphical interface 404 may include a three-dimensional (3D) model of the anatomical structure for display on one or more displays associated with the medical system (such as...). Figure 2The navigation component 420 may be displayed on a display 212. In some embodiments, the navigation component 420 may render a 3D model based at least in part on the skeletal image data 402. Thus, the 3D model of the anatomical structure can provide a visual indicator of the location of a target within the anatomical structure. In some aspects, the graphical interface 404 may be used by a medical system or a user (such as a physician) to navigate instruments to a target during the intraoperative phase of a medical procedure.

[0088] In some aspects, navigation component 420 can also display the instrument's posture and / or path on a 3D model of the anatomical structure. For example, navigation component 420 can determine the instrument's posture relative to the anatomical structure based on sensor data 401 and a mapping 403 between sensor space and skeletal image space. More specifically, navigation component 420 can use mapping 403 to transform the instrument's posture in sensor space (as indicated by sensor data 401) to a corresponding posture in skeletal image space. In some embodiments, navigation component 420 can implement a "fusion" navigation model that combines multiple sensing modalities (such as EM, robot telemetry, and vision) to determine the instrument's posture and / or compensate for any distortions in the sensor data 401. Navigation component 420 can continuously track any changes in the instrument's posture or movement based on sensor data 401 and update the graphical interface 404 to reflect the instrument's real-time posture relative to the anatomical structure.

[0089] In some aspects, navigation component 420 can analyze the spatial relationship between the instrument and the target to provide guidance for navigating the instrument within the anatomical structure. For example, navigation component 420 can plan a navigation path from the instrument to the target through the patient's anatomy. The navigation path can be rendered or displayed on the graphical interface 404 for the instrument to travel along. In some embodiments, navigation component 420 can determine the target's position in the skeletal image space based at least in part on skeletal image data. In some other embodiments, navigation component 420 can determine the target's position in the skeletal image space based at least in part on input provided by the user of navigation system 400. For example, the user can specify or otherwise input the target's position on a 3D map of the anatomical structure.

[0090] As mentioned above, images acquired during the preoperative phase may not accurately reflect the spatial relationship between instruments and targets during the intraoperative phase. For example, changes in patient anatomy or the medical environment may cause the spatial relationship between instruments and targets to deviate from the spatial relationship depicted by the graphical interface 404 at any given time. Examples of factors that may cause such deviations include EM distortion, poor registration (or mapping 403) between sensor space and skeletal image space, outdated skeletal image data 402, and anatomical distortions, among others. In some respects, the medical system may be affected by intraoperative imaging systems (such as...) Figure 3 The fluorescence fluoroscopic imaging system 312) captures updated images of the patient's anatomy and uses the updated image data 406 (also known as "intraoperative image data") to improve or update the depiction of the spatial relationship between instruments and targets in the graphical interface 404.

[0091] In some aspects, navigation component 420 can be configured to operate in either a "navigation mode" or a "renavigation mode," depending on whether corrections are applied to the graphical interface 404 based on intraoperative image data 406. For example, navigation component 420 can operate in navigation mode while the instrument is initially navigated to a threshold near the target before acquiring intraoperative image data 406. When operating in navigation mode, navigation component 420 can use a fusion navigation model that combines multiple sensing modalities (such as EM, robot telemetry, and vision) to compensate for any distortions in the sensor data 401. When the instrument is navigated beyond the threshold, navigation component 420 can be configured to operate in renavigation mode. In some aspects, the operating mode of navigation component 420 can be manually switched by the user of navigation system 400. For example, the user can dynamically switch between navigation mode and renavigation mode at any given time during the medical procedure (assuming certain prerequisites are met). In some embodiments, navigation component 420 can pause 405 navigation or robot movement, allowing the intraoperative imaging system to be brought to a position near the anatomical structure to capture or acquire intraoperative image data 406.

[0092] The intraoperative registration component 430 is configured to register the coordinate space (also referred to as the "intraoperative image space") associated with the intraoperative image data 406 with the sensor space in response to a signal from the navigation component 420 notifying a pause 405 in navigation. More specifically, the intraoperative registration component 430 may determine a mapping 407, at least in part, based on the sensor data 401 and the intraoperative image data 406, which can transform any data point (including point coordinates or vectors) in the intraoperative image space into a corresponding data point in the sensor space. In some aspects, the intraoperative registration component 430 may determine the mapping 407, at least in part, based on the instrument's pose in the sensor space and the corresponding instrument pose in the intraoperative image space. In some other aspects, the intraoperative registration component 430 may determine the mapping 407 based on the known locations of landmarks (such as reference markers or anatomical markers) in both the sensor space and the intraoperative image space.

[0093] In some embodiments, the intraoperative registration component 430 can determine the instrument's pose in the intraoperative image space by analyzing the intraoperative image data 406 using one or more image processing techniques. Suitable example image processing techniques include segmentation, machine learning, and statistical analysis, among others. As used herein, the term "segmentation" refers to various techniques used to divide a digital image into multiple groups of voxels (or "image segments") based on relevant or recognizable features. In some embodiments, the intraoperative registration component 430 can segment the intraoperative image data 406 so that the instrument's pose can be detected or estimated from the corresponding image (or tomographic photograph).

[0094] The intraoperative registration component 430 can determine the instrument's orientation in the sensor space based at least in part on sensor data 401. Various aspects of this disclosure recognize that some imaging techniques (such as CT or X-ray) can interfere with some sensor techniques (such as EM). Therefore, in some embodiments, the intraoperative registration component 430 can determine the instrument's orientation in the sensor space based on sensor data 401 captured or acquired when the intraoperative imaging system cannot interfere with the underlying sensor technology (such as when the fluorescence fluoroscopy imaging system 312 is located outside the EM field of the EM sensor system 306).

[0095] While the patient is typically breathing during sensor data capture 401, they may be forced into a breath-holding state during intraoperative image data capture 406 (e.g., to avoid visual artifacts or other inaccuracies in the resulting images). The patient's breathing can cause the instrument posture in sensor space to differ from its posture in image space at any given time. In some embodiments, the intraoperative registration component 430 can accumulate sensor data 401 over a threshold duration, allowing time-varying changes in instrument posture to be detected and compensated for in the captured data. For example, the threshold duration could be a predetermined duration (e.g., 12 seconds) configured to span one or more respiratory cycles. Therefore, the mapping 407 generated by the intraoperative registration component 430 can compensate for any changes or deviations in instrument posture due to sensor interference or anatomical variations.

[0096] By compensating for deviations in instrument posture caused by sensor interference during registration of the sensor space with the intraoperative image space, the intraoperative registration component 430 can rely on a single sensor modality (such as an EM) that has available data before any interference is introduced to compensate for any distortions in the sensor data 401. In other words, the intraoperative registration component 430 can compensate for deviations in instrument posture when determining the mapping 407 without using any additional modalities (such as robotic telemetry or vision) otherwise used when operating in navigation mode. In some embodiments, the intraoperative registration component 430 can continue to collect data from one or more additional sensing modalities (such as robotic or vision) while determining the mapping 407 and interface update 408. More specifically, the intraoperative registration component 430 can use additional sensor data to detect any distortions in the sensor data 401, for example, to ensure that the sensor data 401 used for registration is not distorted.

[0097] The navigation update component 440 is configured to determine one or more interface updates 408 to be applied to the graphical interface 404 based on intraoperative image data 406 and mapping 407. In some embodiments, interface updates 408 may include the location of a target transformed from intraoperative image space to skeleton image space. In some other embodiments, interface updates 408 may include the posture of an instrument transformed from intraoperative image space to sensor space. For example, the navigation update component 440 may determine the target location or instrument posture in intraoperative image space by analyzing the intraoperative image data 406 using one or more image processing techniques, such as those described above. The navigation update component 440 may also use mapping 407 to transform the target location or instrument posture in intraoperative image space into a corrected target location or instrument posture in sensor space, respectively. As used herein, the terms “corrected target location” and “corrected instrument posture” refer to the target location and instrument posture transformed from intraoperative image space to sensor space. The corrected target location and / or corrected instrument posture in sensor space may be provided to the navigation component 420 as interface updates 408.

[0098] Because instrument posture deviations are compensated for when the sensor space is registered with the intraoperative image space, the navigation update unit 440 can also rely on a single sensor modality (such as EM) to compensate for any distortions in the sensor data 401. In other words, the navigation update unit 440 can compensate for instrument posture deviations when determining the interface update 408 without using any additional modalities (such as robotic telemetry or vision) otherwise used when operating in navigation mode. As described in more detail below, any subsequent distortions in the sensor data 401 introduced by the imaging system when the interface update 408 is applied can be compensated by the navigation unit 420. Therefore, neither the intraoperative registration unit 430 nor the navigation update unit 440 notifies the user of any detected distortions when determining the mapping 407 or the interface update 408.

[0099] In some implementations, navigation component 420 may update graphical interface 404 at least in part based on interface update 408. More specifically, navigation component 420 may update the spatial relationships between the instrument and the target depicted by graphical interface 404. In some aspects, navigation component 420 may update spatial relationships in sensor space. For example, navigation component 420 may use mapping 403 to transform the target's position from skeleton image space to sensor space. Assuming the target's physical position remains constant from the preoperative to the intraoperative phase of the medical procedure, navigation component 420 may determine the transformation ( This transformation maps the target's position in sensor space to the corrected target position indicated by interface update 408. Navigation unit 420 can apply the same transformation to sensor data 401. This is to correct or adjust the spatial relationship between the instrument and the target when operating in renavigation mode.

[0100] In some other aspects, navigation component 420 can update the spatial relationship between the instrument and the target in the skeleton space. In some embodiments, navigation component 420 can determine the correction to be applied to the target's position in the skeleton image space based on interface update 408. In such embodiments, navigation component 420 can use mapping 403 to transform the corrected target position in sensor space to the corresponding position in skeleton image space, and determine the transformation between the target's current position in skeleton image space and the corrected target position. The navigation component 420 can transform... Apply to the target’s current position (such as by modifying the skeletal image data 402 or the 3D model of the anatomical structure) to correct or adjust the target’s position relative to the current instrument posture in the graphical interface 404.

[0101] In some other embodiments, navigation component 420 may determine the correction to be applied to the device pose in the skeleton image space based at least in part on interface update 408. In such embodiments, navigation component 420 may be based on a transformation associated with the corrected target position. To perform the correction transformation ( The navigation component 420 applies real-time sensor data 401 (e.g., by modifying sensor data 401) to correct or adjust the instrument's posture relative to the target position in the graphical interface 404. As mentioned above, some imaging techniques can interfere with some sensor techniques. Therefore, any sensor data 401 acquired after the intraoperative imaging system has been brought near the anatomical structure may be deviated due to interference caused by the imaging system or further changes in the anatomical structure. In some embodiments, when operating in renavigation mode, the navigation component 420 may accumulate sensor data 401 for a threshold duration (e.g., 12 seconds) to detect and compensate for any changes or alterations in instrument posture during one or more respiratory cycles.

[0102] Furthermore, in some embodiments, the navigation unit 420 can determine corrections to be applied to the instrument posture and target position in the skeletal image space. In such embodiments, mapping 407 can be determined based on landmarks in the sensor space and the intraoperative image space. For example, the navigation unit 420 can determine a correction transformation that transforms the current instrument posture in the sensor space to the corrected instrument posture. ), and can transform this The device's posture in the graphical interface 404 is corrected or adjusted by modifying real-time sensor data 401 (e.g., by modifying sensor data 401). Furthermore, mapping 403 can be used to map the corrected device posture in sensor space to skeleton image space. The navigation component 420 can also transform... Apply to the target’s current position (such as by modifying the skeleton image data 402 or the 3D model of the anatomical structure) to correct or adjust the target’s position in the graphical interface 404.

[0103] By compensating for deviations in the instrument's posture when applying interface updates 408 and corrections to real-time sensor data 401, the navigation component 420 can rely on a single sensor modality (such as EM) when operating in renavigation mode. In other words, the navigation component 420 can compensate for deviations in the instrument's posture when operating in renavigation mode without using any additional modalities (such as robot telemetry or vision) that are otherwise used when operating in navigation mode. In some implementations, the navigation component 420 can continue to collect data from one or more additional sensing modalities while applying interface updates 408 to sensor data 401. More specifically, the navigation component 420 can use the additional sensor data to detect any distortions in the sensor data 401, for example, to notify the user of any detected distortions in renavigation mode (allowing the user to remove any sources of distortion to improve the accuracy of the graphical interface 404).

[0104] The updated graphical interface 404 provided by the renavigation mode can offer many advantages over the existing graphical interface 404 provided by the navigation mode. For example, by updating the spatial relationship between the instrument and the target based on intraoperative image data 406 acquired during the intraoperative phase, the navigation system 400 can provide more accurate navigation information to the medical system or the user. In some aspects, the navigation system 400 can dynamically update the graphical interface 404 based on new intraoperative image data 406. For example, the intraoperative imaging system can be used to scan anatomical structures multiple times throughout the intraoperative phase. Each time an anatomical structure is scanned, the intraoperative registration component 430 can determine a new mapping 407 based on the most recent intraoperative image data 406 and sensor data 401, and the navigation update component 440 can use the updated mapping 407 to generate a new interface update 408.

[0105] When operating in renavigation mode, navigation component 420 may also provide additional features in the graphical interface 404. In some aspects, navigation component 420 may update the planned navigation path based on updated patient anatomy and / or updated spatial relationships between the instrument and the target as depicted by intraoperative image data 406. For example, the update may be depicted in contrast to the original navigation path (e.g., using a different color). In some embodiments, navigation component 420 may modify the original navigation path (e.g., by creating new segments or modifying existing segments). In some other embodiments, navigation component 420 may create a new navigation path (also referred to as a "renavigation path") from the current instrument posture to the target location.

[0106] In some other aspects, the navigation component 420 may use intraoperative image data 406 to supplement or replace at least a portion of the skeletal image data 402. For example, as recognized by various aspects of this disclosure, by correcting the instrument posture in the skeletal image space based on the instrument posture determined relative to the intraoperative image space, the corrected position of the instrument in the skeletal image space may be located outside the airway on the 3D model of the anatomical structure generated using the skeletal image data 402. In some embodiments, the navigation component 420 may use the intraoperative image data 406 to recreate at least a portion of the 3D model. For example, the new 3D model may replace the original 3D model in the graphical interface 404 (either as a whole or in an overlapping area) to provide simultaneous imaging and anatomical information.

[0107] In some other embodiments, the graphical interface 404 may display a 3D model of the anatomical structure based on skeletal image data 402, and simultaneously display another 3D model of the anatomical structure based on intraoperative image data 406. For example, the 3D models may be displayed in contrasting colors or with varying degrees of translucency (which may be user-defined). This allows the user to assess the degree of deviation or difference between these 3D models and make corresponding adjustments to the workflow or instrument navigation based on their professional judgment. In some embodiments, the differences between the models can be quantified by a set of metrics, such as local deformation field amplitude, overlap percentage, or path angle difference, and other examples, which may be displayed or presented on the graphical interface 404 in various formats, such as heatmaps or annotations, and other examples.

[0108] As described above, the navigation component 420 can dynamically switch between navigation mode and renavigation mode at any given time. For example, if the user is not satisfied with the spatial relationships depicted by the graphical interface 404 in renavigation mode, or if the user prefers the additional sensing modalities provided by the navigation mode, the user can switch the navigation component 420 to revert to navigation mode at any time. The ability to update the graphical interface 404 with new 3D models of the anatomical structures further enhances the functionality for switching between navigation and renavigation modes. For example, displaying multiple 3D models simultaneously allows the user to determine which model better reflects the current instrument posture indicated by another sensing modality, such as a camera positioned at the distal end of the instrument. Thus, the updated 3D model can also provide a framework for using multiple sensing modalities in renavigation mode.

[0109] Figure 5 An example spatial relationship 500 between the skeleton image space 510, the sensor space 520, and the intraoperative image space 530 is illustrated. In some embodiments, the skeleton image space 510 and the intraoperative image space 530 may be respectively associated with... Figure 4 An example of the coordinate space associated with the skeleton image data 402 and the intraoperative image data 412. More specifically, Figure 5 The location of the target in the intraoperative image space 530 is shown. ) and the posture of the equipment ( ).

[0110] In some aspects, navigation systems (such as Figure 4 The navigation system 400 can locate the target position in the intraoperative image space 530. and machine posture Transformed into target positions in skeleton image space 510 respectively ( ) and apparatus posture ( (See reference) Figure 4 As described, the intraoperative registration component 430 can determine the mapping ( This mapping transforms any data point in the intraoperative image space 530 into a corresponding data point in the sensor space 520, and the skeleton registration component 410 can determine the mapping ( This mapping transforms any data point in sensor space 520 into a corresponding data point in skeleton image space 510. In some embodiments, sensor space 520 can be... Figure 4 An example of the coordinate space associated with sensor data 401. In some implementations, mapping... and They can be respectively Figure 4 Examples of mappings 407 and 403.

[0111] For example, refer to Figure 4 The navigation update component 440 can map Target location applied in intraoperative image space 530 and machine posture To transfer the target position and instrument posture to the sensor space 520, and the navigation component 420 can apply mapping. This is to further transfer the obtained target position and device pose from sensor space 520 to skeleton image space 510. Therefore, the target position in skeleton image space 510... and machine posture It can be represented as: like Figure 5 As shown, the mapping ( (This exists between the intraoperative image space 530 and the skeleton image space 510, and can directly transform any data point in the intraoperative image space 530 into a data point in the skeleton image space 510, wherein:) Figure 6 An example update 600 relative to the instrument pose with respect to the target position is shown. In some implementations, the update 600 can be in the skeleton image space (such as with...). Figure 4 The execution is performed in the coordinate space associated with the skeleton image data 402. More specifically, Figure 6 This shows the current position of the target in the skeleton image space. ) and the current posture of the instrument ( ), and the corrected position of the target ( ) and the corrected posture of the instrument ( ).

[0112] In some aspects, navigation systems (such as Figure 4 The navigation system 400 can determine the position of the target to be calibrated. Applied to the current target location Correction ( ).exist Figure 6 In the example, the navigation system can assume the current instrument posture. Indicates the corrected posture of the apparatus. Or it may conform to it in other ways. Therefore, correction Indicates the current target location With the corrected target position The transformation between them, where: For reference Figure 4 As described, navigation component 420 can correct It is applied to skeletal image data 402 or 3D models of anatomical structures to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. Among other advantages, correction It can be applied once and paired with subsequent sensor data (even in cases where there is no sensor data due to EM loss), without the need for continuous updates.

[0113] Figure 7 Another example update 700 of the device pose relative to the target position is shown. In some implementations, update 700 can be in the skeleton image space (such as with...). Figure 4 The execution is performed in the coordinate space associated with the skeleton image data 402. More specifically, Figure 7 This shows the current position of the target in the skeleton image space. ) and the current posture of the instrument ( ), and the corrected position of the target ( ) and the corrected posture of the instrument ( ).

[0114] In some aspects, navigation systems (such as Figure 4 The navigation system 400 can determine the current target location. With the corrected target position Between, based on transformation Applied to the current instrument posture Correction ( As shown in Formula 1 (see reference) Figure 6 (As described). In Figure 7 In the example, the navigation system can assume the current target location. Indicates the corrected target position Or it may conform to it in other ways. Therefore, correction Indicates the current instrument posture With calibrated instrument posture The transformation matrix between them, where: in It is an identity matrix.

[0115] For reference Figure 4 As described, navigation component 420 can correct The 3D model of sensor data 401 or anatomical structure is applied to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. Among other advantages, this will correct... The application of sensor data 401 allows the navigation system to compensate for deviations in instrument posture that may be caused by sensor interference or changes in anatomical structure (such as reference). Figure 4 (as described).

[0116] Figure 8 Example corrections 800 to the instrument pose and target position in their respective coordinate spaces are shown, along with updates to the spatial relationship between the instrument and the target in a common coordinate space (such as the skeleton image space). Figure 8 In the example, the update of the machine's pose is depicted in sensor space, but a reference can be used. Figure 4 and Figure 5 Any described technique (such as via mapping 403) is mapped to the skeleton image space. In some implementations, the skeleton image space and the sensor space can be respectively mapped to... Figure 4 An example of the coordinate space associated with the skeleton image data 402 and the sensor data 401. More specifically, Figure 8 This shows the current position of the target in the skeleton image space. ) and the corrected position of the target ( ), and the current pose of the instrument in the sensor space ( ) and the corrected posture of the instrument ( ).

[0117] In some aspects, navigation systems (such as Figure 4 The navigation system 400 can determine the posture to be applied to the current apparatus. Correction ( ) and the target location to be applied Correction ( (See reference) Figure 4 As described, correction and This relies on a mapping 407 between the intraoperative image space and the sensor space, determined based on landmarks in both coordinate spaces (rather than just instrument pose). Correction Based on the current target location and the corrected target position According to Formula 1 (as shown in the reference) Figure 6 (As described) to determine. Similarly, correction It can be determined as the current apparatus posture With calibrated instrument posture The transformation between them, where: For reference Figure 4 As described, navigation component 420 can correct and These are respectively applied to sensor data 401 and skeletal image data 402, or a 3D model of the anatomical structure, to update the spatial relationship between the instrument and the target depicted by the graphical interface 404. For example, the navigation component 420 can apply a mapping ( To determine the current posture of the apparatus. Converted to the corresponding instrument pose in skeleton image space ( This mapping transforms any data point in the sensor space into a corresponding data point in the skeleton image space (such as mapping 403), where: Figure 9 A block diagram of an example registration system 900 according to some embodiments is shown. In some embodiments, the registration system 900 may be... Figure 4 An example of a skeletal registration component 410 and / or an intraoperative registration component 430. The registration system 900 is configured to register a sensor system with an imaging system during the intraoperative phase of a medical procedure to facilitate real-time navigation of the medical device within the anatomical structure. In some embodiments, the sensor system and the imaging system may be respectively... Figure 3 Examples of the EM sensor system 306 and the fluorescence imaging system 312.

[0118] In some aspects, the registration system 900 may determine a mapping 906 between the coordinate space associated with the sensor system and the coordinate space associated with the imaging system, at least in part, based on sensor data 901 and image data 902 received via the sensor system and imaging system, respectively. For example, the mapping 906 may be used to transform any data point (such as coordinates or vectors) in the image space into a corresponding data point in the sensor space. In some embodiments, the sensor data 901 may include position or heading information (such as references) about one or more EM sensors positioned in the EM field. Figures 1 to 3 (As described). In some implementations, image data 902 may include one or more tomographic images captured by a CBCT scanner (such as references). Figures 1 to 3 (As described). Various aspects of this disclosure recognize that some imaging techniques (such as CT or X-ray) may interfere with some sensor techniques (such as EM). In some aspects, the registration system 900 may determine the mapping 906 based on sensor data 901 captured or acquired when the imaging system cannot interfere with the underlying sensor technology (such as when the CBCT system is outside the EM field).

[0119] The registration system 900 includes a first reference point extraction component 910, a second reference point extraction component 920, and a spatial mapping determination component 930. The first reference point extraction component 910 is configured to determine one or more reference data points 903 based on sensor data 901, and the second reference point extraction component 920 is configured to determine one or more reference data points 904 based on image data 902. As used herein, the term "reference data point" can refer to any object, feature, point, or vector that can be identified or found in both sensor space and image space. In some embodiments, reference data points 903 and 904 may include the location of a medical device (such as the location of the endoscope tip). In some other embodiments, reference data points 903 and 904 may include the heading of the medical device (such as the orientation of the endoscope tip).

[0120] For example, the first reference point extraction unit 910 can determine the position and heading (in sensor space) of the endoscope based on sensor data 901 associated with an EM sensor disposed on or coupled to the endoscope's tip. The second reference point extraction unit 920 can determine the position and heading (in image space) of the same endoscope tip through segmentation of image data 902. As used herein, the term "segmentation" refers to various techniques used to divide a digital image into multiple groups of voxels (or "image segments") based on relevant or recognizable features. Suitable example segmentation techniques include machine learning (ML) models, masking, thresholding, clustering, and edge detection, among others. In some implementations, the second reference point extraction unit 920 can segment image data 902 such that the position and heading of the endoscope tip can be detected or estimated from the corresponding tomographic images.

[0121] The spatial mapping determination component 930 is configured to determine a mapping 906 based at least in part on reference data points 903 in sensor space and reference data points 904 in image space. In some embodiments, the mapping 906 may be a transformation matrix. Aspects of this disclosure recognize that at least three unique reference data points (which may include any combination of coordinates or vectors) are required to define a unique transformation matrix in 3D space. For example, given three unique reference data points, the spatial mapping determination component 930 may determine a transformation matrix that uses various known techniques or algorithms to transform the three reference data points in image space to three reference data points in sensor space. Suitable example algorithms for determining the spatial transformation matrix include Singular Value Decomposition (SVD) and Iterative Closest Point (ICP) algorithms, among others.

[0122] In some embodiments, each of the reference point extraction units 910 and 920 may be configured to extract at least three unique reference data points 903 and 904 from the sensor data 901 and image data 902, respectively. In such embodiments, the spatial mapping determination unit 930 may determine the mapping 906 using only the reference data points 903 and 904. However, in some other embodiments, at least one of the reference point extraction units 910 or 920 may provide fewer than three unique reference data points 903 or 904, respectively. For example, reference data point 903 may include only the position or heading of the instrument in the sensor space, and reference data point 904 may include only the position or heading of the instrument in the image space. In such embodiments, the spatial mapping determination unit 930 may rely on one or more known data points 905 in the image space and sensor space to determine the mapping 906.

[0123] In some respects, known data point 905 can be based on one or more assumptions or known characteristics about the medical system or its surrounding environment. For example, refer to... Figure 1 As recognized by various aspects of this disclosure, the C-arm 124 of the imaging system 122 is positioned orthogonally to the worktable 112 (or bed) supporting the patient 110, and the worktable 112 is generally parallel to the floor. In some embodiments, the EM field generator 120 may be a window field generator (WFG) positioned directly below (or above) the worktable 112, such that the open center (or “window”) of the WFG is parallel to and overlaps with a flat surface of the worktable 112 (such as the surface of the worktable 112 on which the patient 110 lies). By positioning the EM field generator 120 and the imaging system 122 relative to the worktable 112 at a known orientation or tilt angle, various aspects of this disclosure can control the relationship between the sensor space and the image space. For example, by positioning the EM field generator 120 parallel to the worktable 112 and positioning the imaging system 122 orthogonally to the worktable 112 (such as at a cranial / caudal tilt angle of 0°), the image space can be described by a rotation of 90 or 180 degrees relative to the sensor space. By rotating the dataset in the sensor space or image space, any differences or offsets in the angle of the stage 112 or the tilt angle of the C-arm 124 can be explained, for example, to ensure that the two datasets have a high correspondence.

[0124] Therefore, various known data points 905 can be inferred or extrapolated from the known spatial relationship between the imaging system and the sensor system. In some embodiments, known data points 905 may include vectors pointing to the floor of the medical environment. For example, such vectors may be orthogonal to the flat surface of the patient bed and the window of the WFG. In some other embodiments, known data points 905 may include vectors pointing to the patient's head (or toes). For example, such vectors may be parallel to the flat surface of the patient bed and the window of the WFG. In some embodiments, the mapping determination component 930 may determine a mapping 906 based on one or more reference data points 903 and 904 and one or more known data points 905. More specifically, the mapping determination component 930 may determine a mapping 906 based on any combination of three (or more) unique data points 903-905.

[0125] Figure 10 Another example medical system 1000 according to some embodiments is shown. Medical system 1000 includes an imaging system 1010, a WFG 1020, a support structure 1030 configured to support a patient's anatomy 1040, and an endoscope 1050 inserted into the patient's anatomy 1040. In some embodiments, medical system 1000 may be... Figure 1 An example of a healthcare system 100. For example, see [reference 100]. Figure 1 Imaging system 1010 may be an example of imaging system 122, WFG 1020 may be an example of EM field generator 120, support structure 1030 may be an example of workbench 112, and endoscope 1050 may be an example of medical device 106.

[0126] exist Figure 10 In the example, WFG 1020 is positioned below support structure 1030 (as shown in the exploded view) such that the window of WFG 1020 is parallel to the flat surface of support structure 1030 where anatomical structure 1040 is located. The distal end (or distal tip) of endoscope 1050 includes an EM sensor that can interact with the EM field generated by WFG 1020 to produce sensor data (such as...) indicating the position 1021 and heading 1022 of the endoscope distal tip in sensor space. Figure 9 Sensor data 901). Due to the density of its material, the endoscope 1050 can obtain image data (such as sensor data 901) captured by the imaging system 1010 from the sensor data. Figure 9 The image data (902) is segmented. In some embodiments, the medical system 1000 can determine the position 1011 and heading 1012 of the endoscope tip in the image space based on the segmented image data. Figure 10In the example, data points 1011 and 1012 describe the same pose of the endoscope 1050 as data points 1021 and 1022, but in different coordinate spaces. Therefore, the position 1011 of the endoscope tip in the image space can be mapped to the position 1021 of the endoscope tip in the sensor space, and the heading 1012 of the endoscope tip in the image space can be mapped to the heading 1022 of the endoscope tip in the sensor space.

[0127] In some respects, the medical system 1000 may assume that a vector 1013 in the image space pointing from the end of the endoscope 1050 to the floor can be mapped to a vector 1023 in the sensor space pointing from the end of the endoscope 1050 to the floor. For example... Figure 10 As shown, vectors 1013 and 1023 (also referred to as "floor vectors") are orthogonal to the window of WFG 1020 and the flat surface of support structure 1030. For example, floor vector 1023 can be defined as a vector according to a Cartesian coordinate system associated with the sensor space. Based on the relative positioning of the imaging system 1010 and the WFG 1020, the medical system 1000 can determine that the floor vector 1013 in the image space has been rotated by 90 degrees relative to the floor vector 1023 in the sensor space. Therefore, the floor vector 1013 can be determined from the left rear-upper (LPS) coordinate system associated with the image space. Description. Alternatively, floor vector 1013 can be derived from the right front-lower (RAI) coordinate system using vectors. describe.

[0128] In some respects, the medical system 1000 may assume that a vector 1014 in the image space pointing from the end of the endoscope 1050 to the patient's head (or toes) can be mapped to a vector 1024 in the sensor space pointing from the end of the endoscope 1050 to the patient's head (or toes). For example... Figure 10 As shown, vectors 1014 and 1024 (also referred to as "head-to-toe vectors") are parallel to the window of WFG 1020 and the flat surface of support structure 1030. Head-to-toe vectors 1014 and 1024 are also orthogonal to floor vectors 1013 and 1023, respectively. Therefore, head-to-toe vector 1024 can be defined as a vector according to the Cartesian coordinate system associated with the sensor space. or (Depending on the orientation of the WFG 1020). In some implementations, the medical system 1000 can be based on how the WFG 1020 is registered to the image space associated with the preoperative CT scan (such as...). Figure 3 The registration (326) determines the head-to-foot vector (1024). still The medical system 1000 can also assume that the head-to-foot vector 1014 in the current image space is rotated by 90 degrees relative to the head-to-foot vector 1024 in the sensor space. Therefore, the head-to-foot vector 1014 can be derived from the LPS coordinate system associated with the image space. Description. Alternatively, the head-to-foot vector 1014 can be derived from the vector in the RAI coordinate system. describe.

[0129] In some implementations, floor vectors 1013 and 1023 and head-to-toe vectors 1014 and 1024 can be Figure 9 Example of known data point 905. See reference... Figure 9 As described, only three unique data points are needed to determine the transformation matrix that maps any data point in the image space to the corresponding data point in the sensor space. In some cases, the headings 1012 and 1022 of the endoscope 1050 may overlap with the floor vectors 1013 and 1023 or the head-to-foot vectors 1014 and 1024. In this case, the registration system 900 can still use the floor vectors 1013 and 1023 and the head-to-foot vectors 1014 and 1024 together with the positions 1011 and 1021 of the endoscope 1050 to determine the mapping 906. In other words, even if the headings 1012 and 1022 overlap with one of the other vectors, the floor vectors 1013 and 1023, the head-to-foot vectors 1014 and 1024, and the positions 1011 and 1021 of the endoscope 1050 can always provide a set of three unique reference data points.

[0130] Figure 11 An example mapping 1101 between CT coordinate space 1110 and EM coordinate space 1120 according to some embodiments is shown. In some embodiments, CT space 1110 and EM space 1120 may be respectively... Figure 10 Examples of the image space and sensor space associated with the imaging systems 1010 and WFG 1020.

[0131] CT space 1110 is shown as including point coordinates The coordinates of this point form a set of vectors. , and The origin. For example, refer to... Figure 10 ,point This can represent the position 1011 of the endoscope 1050 in image space, a vector. This can represent the heading 1012 of the endoscope 1050 in the image space, a vector. It can represent the floor vector 1013 in image space, and the vector It can represent a head-to-foot vector of 10¹⁴ in image space. Figure 11 In the example, in CT space 1110, the heading vector Cartesian coordinates Description, floor vector Cartesian coordinates Description, and head-to-toe vector Cartesian coordinates describe.

[0132] EM space 1120 is shown as including point coordinates The coordinates of this point form a set of vectors. , and The origin. For example, refer to... Figure 10 ,point This can represent the position 1021 of the endoscope 1050 in the sensor space, a vector. This can represent the heading 1022 of the endoscope 1050 in the sensor space, a vector. The floor vector 1023 in the sensor space can be represented, and the vector... This can represent a head-to-foot vector of 1024 in sensor space. Figure 11 In the example, in EM space 1120, the heading vector Cartesian coordinates Description, floor vector Cartesian coordinates Description, and head-to-toe vector Cartesian coordinates describe.

[0133] like Figure 11 As shown, spatial mapping 1101 maps the point coordinates of CT space 1110. and vector sets , and Transform them into point coordinates in EM space 1120 respectively and vector sets , and In some implementations, spatial mapping 1101 can be... Figure 9 An example of mapping 906. For example, refer to... Figure 9 The registration system 900 can be based on data points , , , , , , and To determine the spatial mapping 1101. More specifically, the registration system 900 can calculate the transformation matrix. The transformation matrix will transform the data point set Transform into a set of data points ,in: in Let represent the centroid of the midpoint of CT space 1110, and This represents the centroid of the point at the midpoint of EM space 1120. Because CT space 1110 ( ) and EM space 1120 ( Each of the elements in the equation has only one known coordinate point, so and .

[0134] Transformation matrix Including rotation matrix Translation vector ,in In some implementations, the registration system 900 may use singular value decomposition (SVD) to solve for the rotation matrix. For example, registration system 900 can determine the orthogonal matrix U, the diagonal matrix S, and the orthogonal matrix V, where: Registration System 900 can also be based on point coordinates and To calculate the translation vector ,in: like Figure 11 As shown, rotation matrix Only make vector , and Rotate 90 degrees around the x-axis in CT space 1110. Therefore, as an alternative to SVD, registration matrix 900 can determine the first rotation matrix. The first rotation matrix will convert the heading vector in CT space 1110 Transformed into the heading vector in EM space 1120 (Assuming these vectors have the same origin). The registration matrix 900 can be used to rotate the matrix. Applied to data point sets And determine the second rotation matrix. The second rotation matrix makes the remaining vector and Around the heading vector Rotate to change the vector and Transformed into vectors in EM space 1120 respectively and ,in .

[0135] Figure 12 Another example medical system 1200 according to some embodiments is shown. Medical system 1200 includes an imaging system 1210, an EM field generator (FG) 1220, a support structure 1230 configured to support a patient's anatomy 1240, and an endoscope 1250 inserted into the patient's anatomy 1240. In some embodiments, medical system 1200 may be... Figure 1 An example of a healthcare system 100. For example, see [reference 100]. Figure 1 Imaging system 1210 may be an example of imaging system 122, EM field generator 1220 may be an example of EM field generator 120, support structure 1230 may be an example of workbench 112, and endoscope 1250 may be an example of medical device 106.

[0136] exist Figure 12 In the example, the EM field generator 1220 is depicted as a WFG (as shown in the exploded view) positioned below the support structure 1230, such that the window of the WFG is parallel to the flat surface of the support structure 1230 where the anatomical structure 1240 is located. Alternatively, the EM field generator 1220 may be positioned above the support structure 1230 (e.g., in the form of a desktop field generator). However, in some other embodiments, the EM field generator 1220 may be any suitable EM field generator capable of generating one or more EM fields in or near the anatomical structure 1240. The distal end of the endoscope 1250 includes an EM sensor that can interact with the EM field generated by the EM field generator 1220 to produce sensor data indicating the position 1221 and / or heading 1222 of the endoscope tip in the sensor space. In some embodiments, the endoscope 1250 may also include one or more additional EM sensors positioned at predetermined distances along the length of an axis. For example, as Figure 12 As shown, the additional EM sensor can generate sensor data indicating the position of predetermined points 1223 and 1224 along the axis of endoscope 1250. The sensor data can also indicate the heading of the additional EM sensor (not shown for simplicity). In some embodiments, the sensor data generated by the EM sensor can be... Figure 9 Example of sensor data 901.

[0137] Due to the density of its material, the endoscope 1250 can extract image data (such as...) captured by the imaging system 1210 from the image data (such as... Figure 9The image data (902) is segmented. In some embodiments, the medical system 1200 can determine the position 1211 and / or heading 1212 of the endoscope tip in the image space based on the segmented image data. Because the additional EM sensor is positioned at a predetermined distance (a predetermined distance from the endoscope tip) along the length of the endoscope 1250, the medical system 1200 can also determine the positions 1213 and 1214 and / or heading (not shown for simplicity) of the additional EM sensor based on the segmented image data. Figure 12 In the example, data points 1221-1224 describe the same pose (and shape) of the endoscope 1250 as data points 1211-1214, but in different coordinate spaces. Therefore, the positions 1211, 1213, and 1214 and / or headings of the EM sensor in image space can be mapped to the positions 1221, 1223, and 1224 and / or headings of the EM sensor in sensor space, respectively. Similarly, the heading 1212 of the endoscope tip in image space can be mapped to the heading 1222 of the endoscope tip in sensor space.

[0138] For reference Figure 9 and Figure 11 As described, the medical system 1200 can determine a transformation matrix that maps any data point in the image space to a corresponding data point in the sensor space based on three unique reference data points (which may include any combination of point coordinates or vectors). Therefore, when the positions and / or headings of at least three EM sensors can be detected in each of the sensor space and image space, the medical system 1200 may not require the heading vector of the endoscope 1250 to determine the transformation matrix. In some embodiments, the medical system 1200 can guide a user or physician to collect sensor and image data that captures at least two EM sensors. However, if fewer than two EM sensors are detected in at least one of the sensor space or image space, the medical system 1200 can rely on one or more known data points (such as...) Figure 9 The transformation matrix is ​​determined using known data points 905. In some embodiments, the known data points may include a floor vector pointing from the endoscope 1250 to the floor of the medical environment or a head-to-toe vector pointing from the endoscope 1250 to the patient's head (or toes) (such as a reference). Figure 10 (as described).

[0139] In some other embodiments, the medical system 1200 may determine a transformation matrix based on the shape of the endoscope 1250. For example, the medical system 1200 may estimate the shape of the endoscope 1250 in sensor space based on EM sensor data (such as position and / or heading information) collected from multiple EM sensors arranged along the length of the endoscope 1250. The medical system 1200 may also estimate the shape of the endoscope 1250 in image space based on segmented image data. In such embodiments, the medical system 1200 may determine a transformation matrix that transforms multiple points in image space along the estimated shape of the instrument to corresponding points in sensor space along the estimated shape of the instrument.

[0140] Figure 13 Another example medical system 1300 according to some embodiments is shown. Medical system 1300 includes an imaging system 1310, an EM field generator (FG) 1320, a support structure 1330 configured to support a patient's anatomy 1340, and an endoscope 1350 inserted into the patient's anatomy 1340. In some embodiments, medical system 1300 may be... Figure 1 An example of a healthcare system 100. For example, see [reference 100]. Figure 1 Imaging system 1310 may be an example of imaging system 122, EM field generator 1320 may be an example of EM field generator 120, support structure 1330 may be an example of workbench 112, and endoscope 1350 may be an example of medical device 106.

[0141] exist Figure 13 In the example, the EM field generator 1320 is depicted as a WFG (as shown in the exploded view) positioned below the support structure 1330, such that the window of the WFG is parallel to the flat surface of the support structure 1330 where the anatomical structure 1340 is located. However, in some other embodiments, the EM field generator 1320 can be any suitable EM field generator capable of generating one or more EM fields in or near the anatomical structure 1340. The distal end of the endoscope 1350 includes an EM sensor that can interact with the EM field generated by the EM field generator 1320 to produce sensor data indicating the position 1321 and / or heading 1322 of the endoscope tip in the sensor space. In some embodiments, the medical system 1300 may also include one or more additional EM sensors positioned at predetermined locations near the anatomical structure 1340. For example, as Figure 13 As shown, an additional EM sensor can be mounted on the support structure 1330 to generate sensor data indicating the positions of predetermined points 1323 and 1324 on either side of the anatomical structure 1340. In some embodiments, the sensor data generated by the EM sensor can be... Figure 9Example of sensor data 901.

[0142] Due to the density of its material, the endoscope 1350 can extract image data (such as...) captured by the imaging system 1310 from the image data (such as... Figure 9 The image data (902) is segmented. In some embodiments, the medical system 1300 can determine the position 1311 and / or heading 1312 of the endoscope tip in the image space based on the segmented image data. In some embodiments, the additional EM sensor may be radiopaque, such that the positions 1313 and 1314 of the additional EM sensor can also be identified or detected in the segmented image data. Figure 13 In the example, data points 1311 and 1312 describe the same orientation of the endoscope 1350 as data points 1321 and 1322, but in different coordinate spaces. Therefore, the position 1311 of the endoscope tip in image space can be mapped to the position 1321 of the endoscope tip in sensor space, and the heading 1312 of the endoscope tip in image space can be mapped to the heading 1322 of the endoscope tip in sensor space. Because the additional EM sensor can be detected in both image space and sensor space, the positions 1313 and 1314 of the additional EM sensor in image space can also be mapped to the positions 1323 and 1324 of the additional EM sensor in sensor space, respectively.

[0143] For reference Figure 9 and Figure 11 As described, the medical system 1300 can determine a transformation matrix that maps any data point in the image space to a corresponding data point in the sensor space based on three unique reference data points (which may include any combination of point coordinates or vectors). Therefore, the medical system 1300 may not require the heading vector of the endoscope 1350 to determine the transformation matrix when the positions of at least three EM sensors can be detected in each of the sensor space and the image space. In some embodiments, the positions 1313 and 1314 of the additional EM sensors (in the image space) can be determined based on two or more 2D images (instead of 3D images) captured by the imaging system 1310 at different C-arm angles (such as forward and backward angles and rotations of 90 degrees from them). More specifically, the positions 1313 and 1314 of the additional EM sensors can be determined based on a rigid transformation from the 2D image to 3D space (which may be known or predetermined).

[0144] Figure 14Another example medical system 1400 according to some embodiments is shown. Medical system 1400 includes an imaging system 1410, an EM field generator (FG) 1420, a support structure 1430 configured to support a patient's anatomy 1440, and an endoscope 1450 inserted into the patient's anatomy 1440. In some embodiments, medical system 1400 may be... Figure 1 An example of a healthcare system 100. For example, see [reference 100]. Figure 1 Imaging system 1410 may be an example of imaging system 122, EM field generator 1420 may be an example of EM field generator 120, support structure 1430 may be an example of workbench 112, and endoscope 1450 may be an example of medical device 106.

[0145] exist Figure 14 In the example, the EM field generator 1420 is a WFG (as shown in the exploded view) positioned below the support structure 1430, such that the window of the WFG is parallel to the flat surface of the support structure 1430 where the anatomical structure 1440 is located. In some embodiments, the WFG 1420 may be coupled to a reference plate 1460, which has a predetermined orientation or position relative to the WFG 1420. The reference plate 1460 may include one or more transmissive reference marks disposed thereon. For example, as... Figure 14 As shown, the reference plate 1460 includes two reference marks 1423 and 1424 disposed at predetermined positions relative to the WFG 1420. In some embodiments, the positions of the reference marks 1423 and 1424 in the sensor space can be... Figure 9 An example of known data point 905. The distal end of endoscope 1450 includes an EM sensor that can interact with an EM field generated by EM field generator 1420 to produce sensor data indicating the position 1421 and / or heading 1422 of the endoscope tip in sensor space. In some embodiments, the sensor data generated by the EM sensor may be... Figure 9 Example of sensor data 901.

[0146] Due to the density of its material, the endoscope 1450 can extract image data (such as...) captured by the imaging system 1410 from the image data (such as... Figure 9 The image data (902) is segmented. In some embodiments, the medical system 1400 can determine the position 1411 and / or heading 1412 of the endoscope tip in the image space based on the segmented image data. Because the reference marks 1423 and 1424 are radiopaque, the positions 1413 and 1414 of the reference marks 1423 and 1424 can also be identified or detected in the segmented image data, respectively. Figure 14In the example, data points 1411 and 1412 describe the same orientation of the endoscope 1450 as data points 1421 and 1422, but in different coordinate spaces. Therefore, the position 1411 of the endoscope tip in image space can be mapped to the position 1421 of the endoscope tip in sensor space, and the heading 1412 of the endoscope tip in image space can be mapped to the heading 1422 of the endoscope tip in sensor space. Because the relative placement of reference markers 1423 and 1424 is known in sensor space and can be detected in image space, the positions 1413 and 1414 of the reference markers in image space can also be mapped to the known positions of reference markers 1423 and 1424 in sensor space, respectively.

[0147] For reference Figure 9 and Figure 11 As described, the medical system 1400 can determine a transformation matrix that maps any data point in image space to a corresponding data point in sensor space based on three unique reference data points (which may include any combination of point coordinates or vectors). However, when multiple reference markers are located in the same plane (such as reference markers 1423 and 1424), the medical system 1400 can additionally rely on the position 1411 or heading 1412 of the endoscope tip to determine the transformation matrix. In some embodiments, the positions 1413 and 1414 of the reference markers (in image space) can be determined based on two or more 2D images (rather than 3D images) captured by the imaging system 1410 at different C-arm angles (such as front-to-back angles and 90 degrees rotation from them). More specifically, the positions 1413 and 1414 of the reference markers can be determined based on a rigid transformation from a 2D image to 3D space (which may be known or predetermined).

[0148] In reference Figures 10 to 14 In the example described, the medical system (or registration system) can register the image space with the sensor space based on the pose of an endoscope having five mechanical degrees of freedom (including translation on three orthogonal axes plus yaw and pitch of the endoscope tip). In practical implementations, the endoscope may have six mechanical degrees of freedom (including translation on three orthogonal axes plus yaw, pitch, and roll of the endoscope tip). However, because the endoscope is a cylindrical instrument, it may be difficult to detect the roll of the endoscope tip in the image space due to the limited resolution of existing imaging techniques. Various aspects of this disclosure recognize that if the roll of the endoscope tip can be determined in both the EM space and the image space, the medical system can determine the transformation matrix using only the pose of the endoscope tip itself. In some implementations, the medical system can determine the roll of the endoscope tip when the working channel is inserted therein.

[0149] Figure 15An example medical device 1500 according to some embodiments is shown. The medical device 1500 includes an endoscope 1510 having an inner diameter (or lumen) in which a working channel 1520 is provided. Figure 15 As shown, sensor load 1502 is coupled to working channel 1520 such that the center of working channel 1520 is offset relative to the center of endoscope 1510. As used herein, the term “sensor load” can refer to any wires, circuits or other sensor components associated with the sensing system.

[0150] For reference Figures 1 to 14 The described medical systems (such as respectively) Figure 1 , Figure 10 , Figure 12 , Figure 13 and Figure 14 Medical systems 100, 1000, 1200, 1300, or 1400 can determine the position and heading of endoscope 1510 based on EM sensors (not shown for simplicity) disposed on the endoscope tip. In some embodiments, the medical system can also detect the offset between the working channel 1520 and the endoscope 1510 in the sensor space, and thus the roll of the endoscope 1510, based on one or more sensors disposed on the endoscope 1510 or the working channel 1520. The medical system can also detect the offset between the working channel 1520 and the endoscope 1510 in the image space by segmenting image data captured via the imaging system. Therefore, the medical system can determine the transformation matrix between the image space and the sensor space using only the position and orientation (including yaw, pitch, and roll) of the endoscope tip in each coordinate space.

[0151] Figure 16 A block diagram of an example controller 1600 for a medical system according to some embodiments is shown. In some embodiments, the controller 1600 may be... Figure 4 An example of a navigation system 400. More specifically, the controller 1600 is configured to generate and / or update a graphical interface that depicts or otherwise indicates the spatial relationship between the instrument and a target within the anatomical structure.

[0152] The controller 1600 includes a communication interface 1610, a processing system 1620, and a memory 1630. The communication interface 1610 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1610 includes features for communicating with one or more image sources (such as...). Figure 3 The image source interface (I / F) 1612 for communicating with the CT imaging system 310 and / or the fluorescence fluoroscopy imaging system 312 and for communicating with one or more sensors (such as...) Figure 3The sensor interface (I / F) 1614 communicates with the EM sensor system 306. In some embodiments, the image source I / F 1612 can receive image data captured by an imaging system outside the anatomical structure when the instrument is positioned within the anatomical structure. In some embodiments, the sensor I / F 1614 can receive sensor data via one or more sensors associated with the sensor system, wherein the one or more sensors include at least a first sensor disposed on the instrument.

[0153] Memory 1630 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, and other examples) that may store the following software (SW) modules: a graphical interface SW module 1632 for generating a graphical interface depicting the spatial relationship between the instrument and the target; a registration SW module 1634 for determining, at least in part, a mapping between a first coordinate space associated with the imaging system and a second coordinate space associated with the sensor system based on image data and sensor data; and an interface update SW module 1636 for updating the graphical interface, at least in part, based on the mapping between the first and second coordinate spaces, to depict the updated spatial relationship between the instrument and the target. Each software module 1632-1636 includes instructions that, when executed by processing system 1620, cause controller 1600 to perform a corresponding function.

[0154] Processing system 1620 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1600 (such as in memory 1630). For example, processing system 1620 may execute graphical interface (SW) module 1632 to generate a graphical interface depicting the spatial relationship between the instrument and the target. Processing system 1620 may also execute registration (SW) module 1634 to determine, at least in part, a mapping between a first coordinate space associated with the imaging system and a second coordinate space associated with the sensor system, based on image data and sensor data. Processing system 1620 may further execute interface update (SW) module 1636 to update the graphical interface, at least in part, based on the mapping between the first and second coordinate spaces, to depict the updated spatial relationship between the instrument and the target.

[0155] Figure 17 Another block diagram of an example controller 1700 for a medical system according to some embodiments is shown. In some embodiments, the controller 1700 may be... Figure 9An example of the registration system 900. More specifically, the controller 1700 is configured to register the sensor system with the imaging system during the intraoperative phase of a medical procedure to facilitate real-time instrument navigation within anatomical structures.

[0156] The controller 1700 includes a communication interface 1710, a processing system 1720, and a memory 1730. The communication interface 1710 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1710 includes features for communicating with one or more image sources (such as...) Figure 3 The image source interface (I / F) 1712 for communicating with the CT imaging system 310 and / or the fluorescence fluoroscopy imaging system 312 and for communicating with one or more sensors (such as...) Figure 3 The sensor interface (I / F) 1714 communicates with the EM sensor system 306. In some embodiments, the image source I / F 1712 can receive image data captured by an imaging system outside the anatomical structure when the instrument is positioned within the anatomical structure. In some embodiments, the sensor I / F 1714 can receive sensor data from a sensor mounted on the instrument.

[0157] Memory 1730 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, and other examples) that may store the following software (SW) modules: a sensor spatial positioning SW module 1732 for determining the position of an instrument in a first coordinate space based on received sensor data; an image spatial positioning SW module 1734 for determining the position of an instrument in a second coordinate space based on received image data; and a registration SW module 1736 for determining a mapping between the first and second coordinate spaces based at least in part on the positions of the instrument in the first and second coordinate spaces. Each software module 1732-1736 includes instructions that, when executed by processing system 1720, cause controller 1700 to perform a corresponding function.

[0158] Processing system 1720 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1700 (such as in memory 1730). For example, processing system 1720 may execute sensor spatial positioning (SW) module 1732 to determine the position of the instrument in a first coordinate space based on received sensor data. Processing system 1720 may also execute image spatial positioning (SW) module 1734 to determine the position of the instrument in a second coordinate space based on received image data. Processing system 1720 may further execute registration (SW) module 1736 to determine a mapping between the first and second coordinate spaces, at least in part, based on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space.

[0159] Figure 18 An exemplary flowchart depicting example operation 1800 of a navigation instrument within an anatomical structure according to some embodiments is shown. In some embodiments, example operation 1800 may be controlled by a controller for a medical system (such as...). Figure 16 (controller 1600) or Figure 4 The navigation system 400 is executed.

[0160] The controller can generate a graphical interface depicting the spatial relationship between the instrument and the target within the anatomical structure (1802). The controller can receive first image data captured by a first imaging system outside the anatomical structure when the instrument is positioned within the anatomical structure (1804). In some embodiments, the first imaging system may be a cone-beam computed tomography (CBCT) system. The controller can receive first sensor data via one or more sensors associated with a sensor system, wherein the one or more sensors include at least a first sensor mounted on the instrument (1806). The controller can determine, at least in part, a mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system based on the first image data and the first sensor data (1808). The controller can also update the graphical interface to depict the updated spatial relationship between the instrument and the target based at least in part on the mapping between the first and second coordinate spaces (1810).

[0161] Figure 19 An exemplary flowchart depicting example operation 1900 for registering image space with sensor space according to some embodiments is shown. In some embodiments, example operation 1900 may be performed by a controller for a medical system (such as...) Figure 17 (Controller 1700) or Figure 9 The registration system 900 is executed.

[0162] The controller can receive sensor data from sensors on an instrument positioned within the anatomical structure (1902). The controller can determine the position of the instrument in a first coordinate space based on the received sensor data (1904). The controller can receive image data captured by an imaging system external to the anatomical structure while the instrument is positioned within the anatomical structure (1906). In some embodiments, the first imaging system may be a CBCT system. The controller can determine the position of the instrument in a second coordinate space based on the received image data (1908). The controller can determine a mapping between the first and second coordinate spaces, at least in part, based on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space (1910).

[0163] Those skilled in the art will understand that information and signals can be represented using any of a variety of different techniques and arts. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description can be represented by voltage, current, electromagnetic waves, magnetic fields or particles, light fields or particles, or any combination thereof.

[0164] The various exemplary logics, logic blocks, modules, circuits, and algorithmic processes described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. The interchangeability of hardware and software has been generally described in terms of functionality and illustrated in the various exemplary components, blocks, modules, circuits, and processes described herein. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the system as a whole.

[0165] In the foregoing specification, specific examples of various embodiments have been described with reference to them. However, it will be apparent that various modifications and changes can be made to these specific embodiments without departing from the broader scope of this disclosure as set forth in the appended claims. Therefore, the specification and drawings are to be regarded as illustrative rather than limiting.

[0166] As used in this article, the phrase “at least one of…” in a list of items refers to any combination of those items, including a single member. As an example, “at least one of a, b, or c” is intended to cover: a, b, c, ab, ac, bc, and abc.

[0167] Various modifications to the embodiments described in this disclosure will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of this disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but are to be given the broadest scope consistent with the disclosure, principles, and novel features disclosed herein.

Claims

1. A controller for a medical system, comprising: Processing system; and The memory stores instructions that, when executed by the processing system, cause the controller to: Generate a graphical interface that depicts the spatial relationship between instruments and targets within anatomical structures; Receive first image data captured by a first imaging system outside the anatomical structure when the instrument is placed inside the anatomical structure; First sensor data is received via one or more sensors associated with a sensor system, wherein the one or more sensors include at least the first sensor disposed on the instrument; The mapping between a first coordinate space associated with the first imaging system and a second coordinate space associated with the sensor system is determined, at least in part, based on the first image data and the first sensor data. as well as The graphical interface is updated at least in part based on the mapping between the first coordinate space and the second coordinate space to depict the updated spatial relationship between the instrument and the target.

2. The controller according to claim 1, wherein, The generation of the graphical interface includes: Second image data is received via a second imaging system outside the anatomical structure; The second sensor data is received via the first sensor mounted on the device. The mapping between the second coordinate space and the third coordinate space associated with the second imaging system is determined, at least in part, based on the second image data and the second sensor data; and The spatial relationship between the instrument and the target is determined at least in part based on the mapping between the second coordinate space and the third coordinate space.

3. The controller according to claim 2, wherein, The update to the graphical interface includes: The position of the target in the third coordinate space is determined based on the second image data; The position of the target in the first coordinate space is determined based on the received first image data; Based on the mapping between the first coordinate space and the second coordinate space, and the mapping between the second coordinate space and the third coordinate space, a transformation is determined to change the position of the target in the first coordinate space to the position of the target in the third coordinate space; and The correction associated with the transformation is applied to subsequent sensor data received via the first sensor mounted on the device.

4. The controller according to claim 2, wherein, The execution of the instruction further enables the controller to: The position of the target in the first coordinate space is determined based on the first image data; Based on the mapping between the first coordinate space and the second coordinate space and the mapping between the second coordinate space and the third coordinate space, the position of the target is transformed from the first coordinate space to the third coordinate space; as well as The updated spatial relationship between the instrument and the target is determined at least in part based on the target's position in the third coordinate space.

5. The controller according to claim 2, wherein, The generation of the graphical interface further includes: A first model of the anatomical structure is generated based on the second image data; and The spatial relationship between the instrument and the target is shown on the first model of the anatomical structure.

6. The controller according to claim 5, wherein, The update to the graphical interface includes: At least a portion of the first model is reconstructed based on the first image data to represent the anatomical structure.

7. The controller according to claim 5, wherein, The update to the graphical interface includes: A second model of the anatomical structure is generated based on the first image data; and The updated spatial relationship between the instrument and the target is displayed on the second model of the anatomical structure.

8. The controller according to claim 2, wherein, The generation of the graphical interface includes: The path for navigating the instrument within the anatomical structure is determined at least in part based on the second image data.

9. The controller according to claim 8, wherein, The update to the graphical interface includes: The path used for navigating the instrument within the anatomical structure is updated based at least in part on the first image data.

10. The controller according to claim 1, wherein, The update to the graphical interface includes: The posture of the device in the first coordinate space is determined based on the first sensor data received via the first sensor; The corrected posture of the instrument in the second coordinate space is determined based on the first image data; Based on the mapping between the first coordinate space and the second coordinate space, the corrected posture of the device is transformed from the second coordinate space to the first coordinate space; Determine the correction transformation that transforms the posture of the instrument in the first coordinate space into the corrected posture of the instrument in the first coordinate space; and The correction transformation is applied to subsequent sensor data received via the first sensor mounted on the device.

11. The controller according to claim 10, wherein, The first sensor is an electromagnetic (EM) sensor, which is positioned within an EM field generated by a field generator coupled at a known angle to a support structure having a flat surface supporting the anatomical structure.

12. The controller according to claim 11, wherein, The determination of the mapping between the first coordinate space and the second coordinate space includes: Estimate a first vector orthogonal to the flat surface of the support structure in each of the first coordinate space and the second coordinate space; Estimate a second vector parallel to the flat surface of the support structure in each of the first and second coordinate spaces; and A transformation matrix is ​​determined, which transforms the posture of the device in the first coordinate space and the estimated first and second vectors into the corrected posture of the device in the second coordinate space and the estimated first and second vectors, respectively.

13. The controller according to claim 11, wherein, The field generator is further coupled to a reference board, the reference board including one or more reference marks positioned relative to the field generator at a predetermined location, and the execution of the command further causes the controller to: The position of each of the one or more reference markers in the first coordinate space is determined based on the first image data, and the mapping between the first coordinate space and the second coordinate space is further determined at least in part based on the position of the one or more reference markers in the first coordinate space and the predetermined position of the one or more reference markers relative to the field generator.

14. The controller according to claim 1, further comprising: Receive additional sensor data from one or more additional sensors near the anatomical structure; The positions of the one or more additional sensors in the second coordinate space are determined based on the additional sensor data; as well as The positions of the one or more additional sensors in the first coordinate space are determined based on the first image data, and the mapping between the first coordinate space and the second coordinate space is further determined at least in part based on the positions of the one or more additional sensors in the first coordinate space and the positions of the one or more additional sensors in the second coordinate space.

15. The controller according to claim 14, wherein, At least one of the one or more additional sensors is mounted on the instrument.

16. The controller according to claim 14, wherein, At least one of the one or more additional sensors is positioned at a predetermined location outside the anatomical structure.

17. A controller for a medical system, comprising: Processing system; and The memory stores instructions that, when executed by the processing system, cause the controller to: Receive sensor data from sensors on instruments placed within the anatomical structure; The position of the device in the first coordinate space is determined based on the received sensor data; Receive image data captured by an imaging system outside the anatomical structure when the instrument is placed inside the anatomical structure; The position of the instrument in the second coordinate space is determined based on the received image data; as well as The mapping between the first coordinate space and the second coordinate space is determined at least in part based on the position of the instrument in the first coordinate space and the position of the instrument in the second coordinate space.

18. The controller according to claim 17, wherein, The sensor is an electromagnetic (EM) sensor, which is positioned within an EM field generated by a field generator coupled at a known angle to a support structure having a flat surface supporting the anatomical structure.

19. The controller according to claim 18, wherein, The determination of the mapping between the first coordinate space and the second coordinate space includes: Estimate a first vector orthogonal to the flat surface of the support structure in each of the first coordinate space and the second coordinate space; Estimate a second vector parallel to the flat surface of the support structure in each of the first and second coordinate spaces; and A transformation matrix is ​​determined, which transforms the position of the instrument in the second coordinate space and the estimated first vector and second vector into the position of the instrument in the first coordinate space and the estimated first vector and second vector, respectively.

20. The controller according to claim 19, wherein, The determination of the transformation matrix includes: The heading of the instrument in the first coordinate space is determined based on the received sensor data; The heading of the instrument in the second coordinate space is determined based on the received image data; and Determine the rotation matrix and the translation matrix, which transform the position of the instrument, the heading of the instrument, and the first vector and the second vector in the second coordinate space into the position of the instrument, the heading of the instrument, and the first vector and the second vector in the first coordinate space, respectively.

Images