Updating instrument navigation

Abstract

Techniques for updating navigation information associated with an instrument are discussed herein. For example, graphical interface data can be provided during a procedure to assist a user in navigating an instrument within an anatomical structure to a target and / or other location within the anatomical structure. The graphical interface data can indicate a first spatial relationship between the instrument and the target. Further, image data from an imaging system located externally can be used to determine a second spatial relationship between the instrument and the target. The graphical interface data can be updated based on the second spatial relationship.

Description

[0001] Cross-references to related applications This patent application claims priority to U.S. Non-Provisional Patent Application No. 19 / 090,297, filed March 25, 2025, entitled “UPDATING INSTRUMENTNAVIGATION,” which claims priority and benefit to U.S. Provisional Patent Application No. 63 / 572,135, filed March 29, 2024, entitled “UPDATING INSTRUMENTNAVIGATION,” pursuant to 35 USC § 119(e). The disclosures of these earlier applications are considered part of this patent application and are incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to medical systems, and more specifically to techniques for updating device navigation. Background Technology

[0003] This disclosure relates to the field of medical procedures. In the example, the medical procedure involves navigating an instrument within an anatomical structure to reach a treatment site. Certain navigation information may be provided to assist the user in navigating the instrument. However, in some cases, the navigation information may not accurately depict the position of the instrument within the anatomical structure. Summary of the Invention

[0004] This summary is provided to introduce some concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the subject matter protected by the claims, nor is it intended to limit the scope of the subject matter protected by the claims.

[0005] This disclosure relates to systems, apparatus, and methods for determining and / or updating the position of an instrument during a procedure. For example, navigation information can be provided during a procedure to assist a user in navigating an instrument within an anatomical structure to reach a target and / or other location within the anatomical structure. The navigation information can indicate the spatial relationship between the instrument and the target. To maintain / provide accurate navigation information, image data from an imaging system located outside the anatomical structure can be used to determine the spatial relationship between the instrument and the target. The navigation information can be updated based on the spatial relationship determined from the image data.

[0006] One innovative aspect of the subject matter of this disclosure can be implemented in a medical system including a processing system and control circuitry. The control circuitry is configured to: generate a graphical interface depicting a first spatial relationship between an instrument and a target within an anatomical structure; receive image data depicting an anatomical structure in which the instrument is disposed, wherein the image data is captured by an imaging system while positioned outside the anatomical structure; determine a second spatial relationship between the instrument and the target within the anatomical structure based at least in part on the image data; and update the graphical interface to depict the second spatial relationship between the instrument and the target.

[0007] Another innovative aspect of the subject matter of this disclosure can be implemented in a method executed by a controller for a medical system. The method includes the steps of: generating a graphical interface depicting a first spatial relationship between an instrument and a target within an anatomical structure; receiving image data depicting an anatomical structure in which the instrument is disposed, wherein the image data is captured by an imaging system positioned outside the anatomical structure; determining a second spatial relationship between the instrument and the target within the anatomical structure based at least in part on the image data; and updating the graphical interface to depict the second spatial relationship between the instrument and the target.

[0008] Another innovative aspect of the subject matter of this disclosure can be implemented in a controller for a medical system, the controller including a processing system and a memory. The memory stores instructions that, when executed by the processing system, cause the controller to: generate a graphical interface depicting a first spatial relationship between an instrument and a target within an anatomical structure; receive image data depicting an anatomical structure in which the instrument is disposed, wherein the image data is captured by an imaging system positioned outside the anatomical structure; determine a second spatial relationship between the instrument and the target within the anatomical structure, at least in part, based on the image data; and update the graphical interface to depict the second spatial relationship between the instrument and the target.

[0009] While certain aspects, advantages, and / or features have been described, it should be understood that not all such advantages can be implemented according to any particular example. Therefore, the disclosed features can be implemented or optimized in a manner that achieves or optimizes one or a set of advantages as taught herein, but not necessarily as other advantages that may be taught or presented herein. Attached Figure Description

[0010] The disclosed aspects will be described below in conjunction with the accompanying drawings, which are provided to illustrate rather than limit the disclosed aspects, wherein similar reference numerals denote similar elements.

[0011] Figure 1 Examples of medical systems based on one or more examples are provided.

[0012] Figure 2 Examples are provided based on one or more examples. Figure 1Example components of control systems and robotic systems.

[0013] Figure 3 This is a block diagram illustrating a positioning system based on one or more examples.

[0014] Figure 4A and Figure 4B Example flowcharts illustrating the process for updating navigation guidance based on one or more examples are provided.

[0015] Figure 5 An example flowchart illustrating a process for determining the spatial relationships between elements depicted within image data, based on one or more examples.

[0016] Figure 6 An example flowchart illustrating a process for providing navigation guidance for a medical device, based on one or more examples, is provided.

[0017] Figure 7 Example interfaces for providing intraoperative data and / or receiving input about the location of the target, based on one or more examples, are shown.

[0018] Figure 8 Example interfaces for providing intraoperative data and / or receiving input about the position of instruments, based on one or more examples, are illustrated.

[0019] Figure 9 Example interfaces for providing information and / or receiving input regarding the alignment of instrument indicators on image data, based on one or more examples, are illustrated.

[0020] Figure 10-1 Example diagrams illustrating elements associated with initial navigation information based on one or more examples are shown.

[0021] Figure 10-2 Example diagrams illustrating elements associated with updated navigation information based on one or more examples are shown.

[0022] Figure 11-1 Example interfaces are shown, based on one or more examples, for providing graphical interface data associated with initial navigation information.

[0023] Figure 11-2 Example interfaces are shown, based on one or more examples, for providing graphical interface data associated with updated navigation information.

[0024] Figure 12 A block diagram of an example controller for a medical system, based on some specific implementations, is shown.

[0025] Figure 13An exemplary flowchart depicting example operations for guiding instruments to navigate within anatomical structures, according to some specific implementations, is shown. Detailed Implementation

[0026] Some medical procedures are linked to preoperative and intraoperative phases, with anatomical mapping generated and targets designated in the preoperative phase, and the targets located in the intraoperative phase. For example, during the preoperative phase, anatomical mapping can be created, and locations within anatomical structures can be designated as targets. During the intraoperative phase, the endoscope can be controlled to attempt to reach that target. However, in some cases, differences in the characteristics of anatomical structures between the preoperative and intraoperative phases, distortions introduced into the environment and sensed by sensors, and / or other factors can make accurate navigation to the same preoperative target difficult. For example, in the context of bronchoscopy, the lungs may experience different levels of inspiration, deform due to interaction with the instruments, and / or shift / move between the preoperative and intraoperative phases. Furthermore, systems used to detect instrument position may experience distortion or other errors / problems, and / or the patient may be positioned differently between the preoperative and intraoperative phases. Such problems can lead to variations in anatomical mapping between the preoperative and intraoperative phases, potentially resulting in inaccuracies in determining the endoscope's position.

[0027] This disclosure relates to systems, apparatus, and methods for locating and / or updating the position of a medical device (such as the position of the medical device relative to a target and / or anatomical structure) during procedures. For example, a medical system may provide navigation information during procedures to assist a user in navigating a medical device within an anatomical structure to reach a target and / or other location within the anatomical structure. In some cases, the navigation information may include mapping of the anatomical structure, the position of the medical device, and / or the position of the target, wherein one or more pieces of navigation information may be based on preoperative data. For example, the navigation information may provide mapping and / or the position of the target based on preoperative data, and provide the position of the medical device obtained from real-time / intraoperative sensor data and / or image data.

[0028] During the procedure (e.g., intraoperative phase), image data from an external imaging system can be used to accurately determine the position of the medical device and update navigation information (if needed). For example, the imaging system may be positioned above an anatomical structure and configured to capture internal images of the anatomical structure from an external location, such as a computed tomography (CT) system positioned near the patient. The medical system can receive and analyze the image data to determine the spatial relationship between the medical device and the target. In an example, the medical system may display the image data to a user, receive user input indicating the position of the medical device and / or the target in the image data, and analyze the image data based on the user input and / or image processing techniques (e.g., including models associated with machine learning / artificial intelligence) to determine the spatial relationship between the medical device and the target. The medical system can update the navigation information based on the spatial relationships indicated / identified in the image data to reflect the current / intraoperative position of the medical device relative to the target and / or mapping. Thus, the navigation information can accurately depict the position of the medical device, the position of the target, and / or other information about the anatomical structure.

[0029] Although examples are discussed in the context of medical procedures with multiple phases, these techniques can be used during the same phase of the procedure, such as to accurately identify the current position of a medical device relative to a target that was previously identified in the same phase of the procedure.

[0030] In the examples discussed herein, the techniques utilize real-time / intraoperative image data from an imaging system located outside the anatomical site. In some cases, the imaging system provides 3D image data that allows the user to accurately identify the position of instruments and / or targets intraoperatively. This can allow for updates to navigation guidance, allow the user to view or confirm the proper positioning of instruments relative to the target / anatomical structure (which can be advantageous in procedures requiring a relatively high level of accuracy), etc. In the examples, using 3D image data may be more advantageous than 2D image data. For example, in some cases, certain target / anatomical features are not visible in 2D image data but are visible in 3D image data. By providing 3D image data, certain target / anatomical features can be identified (e.g., in multiple slices of the image data), identified more easily, or identified more accurately. However, in many cases, these techniques can be implemented with 2D image data and still offer various advantages. In any case, image data from an external imaging system can be used to update navigation guidance, view the position of instruments relative to the target / anatomical structure, or otherwise enhance navigation or other functions.

[0031] In some cases, the techniques discussed herein enable robot-assisted medical procedures, where robotic tools / components enable physicians, operators, or other users to perform the procedures. For example, a robotic tool may engage and / or control one or more medical devices (such as endoscopes) to access anatomical sites within a patient's body and / or perform diagnosis or treatment at those sites. In some cases, the robotic tool is guided / controlled by a physician, operator, or other user. In others, the robotic tool operates in an automated or semi-automatic manner. While many techniques are discussed in the context of robot-assisted medical procedures, these techniques are applicable to other types of medical procedures, such as those that do not implement robotic tools or implement them for a relatively small number of operations (e.g., less than a threshold number). For example, these techniques are applicable to procedures implementing manually operated medical devices, such as manual endoscopes that are entirely controlled by a physician, operator, or other user. In the examples discussed herein, the techniques / systems are implemented in the context of diagnosis, therapy / treatment, etc.

[0032] While certain examples are disclosed herein, the subject matter extends beyond the specifically disclosed examples to other alternative examples and / or uses, as well as modifications and equivalents thereof. Therefore, the scope of the appended claims is not limited to any particular example among the specific examples described herein. For instance, in any method or process disclosed herein, the actions or operations of the method or process can be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations can thus be described as multiple discrete operations in a manner that may aid in understanding certain examples; however, the order of description should not be construed as implying that these operations are sequentially related. Furthermore, the structures, systems, and / or apparatuses described herein may be embodied as integrated components or separate components.

[0033] Some aspects and advantages of these examples are described. Not all such aspects or advantages are achieved through any particular example. Thus, for example, various examples can be carried out in a way that achieves or optimizes one or a set of advantages as taught herein, without necessarily achieving other aspects or advantages as may also be taught or presented herein.

[0034] Example medical system Figure 1An example medical system 100 (also referred to as "surgical medical system 100" or "robotic medical system 100") according to one or more examples is illustrated. For example, medical system 100 may be configured for diagnostic and / or therapeutic bronchoscopy, as shown. Medical system 100 may include and utilize robotic system 102, which may be implemented as, for example, a robotic cart. Although medical system 100 is shown as including various cart-based systems / devices, the concepts disclosed herein can be implemented in any type of robotic system / arrangement, such as robotic systems employing track-based components, stage-based robotic end effectors / manipulators, etc. Robotic system 102 may include one or more robotic arms 104 (also referred to as "robotic locator 104") configured to locate or otherwise manipulate medical instruments, such as medical instrument 106 (e.g., a steerable endoscope or another elongated instrument). For example, medical device 106 can be advanced through a natural orifice access point (e.g., the mouth 108 of patient 110 positioned on stage 112 in this example) to deliver diagnostic and / or therapeutic treatments. Although described in the context of a bronchoscopy procedure, medical system 100 can be implemented for other types of procedures, such as gastrointestinal (GI) procedures, kidney / urinary tract / nephropathy procedures, etc.

[0035] With the robotic system 102 correctly positioned, the medical device 106 can be inserted into the patient 110 by robot, by manual means, or a combination thereof. In the example, one or more robotic arms 104 and / or their instrument actuators 114 can control the medical device 106. In some implementations, the medical device 106 can be advanced within a sheath 116, and in some cases, the sheath can be coupled to and / or controlled by a robotic arm. For example, the medical device 106 and the sheath 116 can each be coupled to a separate instrument actuator from a set of instrument actuators 114. The instrument actuators 114 can be repositioned in space by manipulating one or more robotic arms 104 to different angles and / or positions.

[0036] In the example shown, the medical device 106 can be guided downwards along the patient's trachea and lungs after insertion, and / or advanced to a target destination or surgical site. In the example, to enhance navigation through the patient's lung network and / or reach the desired target, the medical device 106 can be manipulated to extend telescopically from the external sheath 116 to obtain enhanced joint movement and / or a larger radius of bending. Using a separate device actuator 114 allows the medical device 106 and the sheath 116 to be driven independently of each other.

[0037] In the example, medical device 106 includes an elongated member / axis configured to insert / retract within an anatomical structure, perform articulation, or otherwise move. Furthermore, in the example, medical device 106 includes an imaging device (e.g., a camera) positioned at the distal end of the elongated axis or / or deployed via a working channel of the elongated axis. The imaging device may be configured to generate / capture image data and / or transmit image data to another device / component. Additionally, in the example, medical device 106 includes an instrument base / handle positioned at the proximal end of medical device 106. The instrument base may be configured to couple to a manipulator (e.g., the end effector of a robotic arm). The instrument base may include a drive input configured to couple to a drive output of the manipulator, wherein the drive input and / or drive output act as an interface.

[0038] Furthermore, in the example, the medical device 106 includes a working channel configured to receive one or more other instruments / components and / or provide other functions therein. The working channel may extend axially, such as along the length of the medical device 106. Additionally, the medical device 106 may include or be associated with an elongated moving member (e.g., a drawstring) that extends from a proximal end through an elongated shaft to a distal end of the elongated shaft. The elongated moving member may be manipulated (e.g., by manipulators on one or more robotic arms 104) to control actuation of the elongated moving member.

[0039] Furthermore, in the example, medical device 106 includes sensors such as electromagnetic (EM) sensors, shape sensors (e.g., shape-sensing fibers), accelerometers, gyroscopes, satellite-based positioning sensors (e.g., Global Positioning System (GPS) sensors), radio frequency transceivers, etc. Sensors (also referred to as “position sensors”) may be configured to generate sensor data and / or provide sensor data to another device / component. Sensors may be disposed at the distal end of an elongated shaft, along the length of the elongated shaft, etc. Additionally, in the example, medical device 106 is configured to receive an elongated member / device via a working channel, wherein the elongated member includes one or more sensors along the length of the elongated member. In some examples, sensors on medical device 106 may provide sensor data to control circuitry of medical system 100, which is then used to determine the position, orientation, and / or shape of medical device 106.

[0040] Medical system 100 may also include control system 118 (also known as “control tower 118” or “mobile tower 118”), which will be referred to below. Figure 2Detailed Description. The control system 118 is communicatively coupled (e.g., via wired and / or wireless connections) to the robotic system 102 to provide support for controls, electronics, fluid dynamics, optics, sensors, and / or power to the robotic system 102. Placing such functionality within the control system 118 allows for a smaller form factor to the robotic system 102, which can be more easily adjusted and / or repositioned by the operator. Furthermore, the separation of functions between the robotic system 102 and the control system 118 can reduce operating room clutter and / or facilitate efficient clinical workflows.

[0041] Medical system 100 may include an electromagnetic (EM) field generator 120 configured to broadcast / transmit an EM field detected by an EM sensor, such as a sensor associated with medical device 106. The EM field can induce a small current in the coil of the EM sensor (also referred to as a "position sensor"), which can be analyzed to determine the position and / or angle / or orientation of the EM sensor relative to the EM field generator 120. The EM field generator 120 may be positioned relative to the side of stage 112 (e.g., ...). Figure 1 (as shown), positioned below stage 112, positioned above stage 112, etc. Although EM fields and EM sensors are described in many examples in this paper, the position sensing system and / or sensor can be any type of position sensing system and / or sensor, such as optical position sensing system / sensor, image-based position sensing system / sensor, etc.

[0042] The medical system 100 may also include an imaging system 122 (also referred to as “imaging device 122”) configured to generate image data (also referred to as “images”) and / or provide / transmit such image data to / from another device / system. For example, the imaging system 122 may generate image data depicting the anatomical structures of the patient 110 and provide such image data to a control system 118, a robotic system 102, and / or another device. The imaging system 122 may include emitters / energy sources (e.g., X-ray sources) and / or detectors (e.g., X-ray detectors) mounted on a support 124 (which may include a C-shaped support or another shape), allowing flexibility in positioning around the patient 110 to capture images from various angles without moving the patient 110. The use of the imaging system 122 can provide visualization of internal structures / anatomical structures, which can be used for a variety of purposes, such as navigation of the medical device 106 (e.g., providing the user with images of internal anatomy), positioning of the medical device 106 (e.g., analysis based on image data), etc. In the example, the use of imaging system 122 can enhance the effectiveness and / or safety of medical procedures such as bronchoscopy by providing clear, continuous visual feedback to the surgical surgeon / team.

[0043] In some examples, the imaging system 122 is a mobile device configured to move around within the environment. For example, the imaging system 122 may be positioned next to the patient 110 during a specific phase of the procedure (as shown in the figure) and removed when the imaging system 122 is no longer needed. In other examples, the imaging system 122 may be part of table 112 or other equipment in the surgical environment. The imaging system 122 may be implemented as a computed tomography (CT) machine / system, an X-ray machine / system, a fluoroscopic examination machine / system, a positron emission tomography (PET) machine / system, a PET-CT machine / system, a CT angiography machine / system, a cone-beam CT machine / system, a 3DRA machine / system, a single-photon emission computed tomography (SPECT) machine / system, a magnetic resonance imaging (MRI) machine / system, an optical coherence tomography (OCT) machine / system, an ultrasound machine / system, etc. In some cases, medical system 100 includes multiple imaging systems, such as a first type of imaging system and a second type of imaging system, wherein different types of imaging systems may be used or positioned on the patient 110 as needed at different stages / parts of the procedure.

[0044] In some examples, imaging system 122 is configured to process / generate multiple images (also referred to in some cases as "image data") to generate a three-dimensional (3D) view / model. For example, imaging device 122 may be implemented as a CT scanner configured to capture / generate a series of images / image data (e.g., 2D images / slices) from different angles around patient 110, and then reconstruct a 3D model from these images / image data using one or more algorithms. The 3D model may be provided to control system 118, robotic system 102, and / or another device, such as for processing, display, or other purposes. In the example, imaging system 122 is configured to generate 2D or 3D image data.

[0045] In this example, image data from imaging system 122 is used to locate various elements, such as medical device 106, targets within an anatomical structure, specific anatomical features, etc. For example, control system 118 may be configured to provide navigation information (also referred to as “navigation data”) during a procedure to assist the user in navigating medical device 106 within an anatomical structure to reach a target (e.g., a desired anatomical site / location). In some examples, the target may include a nodule, such as in the context of certain bronchoscopy procedures. For illustration, control system 118 may display navigation view / graphic data 126, which includes an instrument indicator 128 representing medical device 106, a target indicator 130 representing the target, and an anatomical mapping. Navigation data 126(A) (e.g., initial navigation data) may be determined based on sensor data from sensors of medical device 106 (e.g., EM sensor data associated with EM field generator 120), mapping of the anatomical structure, and / or the location of the target. In the example, the target mapping and / or location is based on preoperative data (such as data obtained during the preoperative phase) to identify the target location and / or mapping anatomical structures.

[0046] In any case, navigation data 126(A) can be updated based on image data 132 from imaging system 122 as needed. For example, control system 118 can receive and analyze image data 132 to determine the current / actual spatial relationship between medical device 106 and target. In an example, control system 118 can display image data 132 to a user, receive user input indicating the position of medical device 106 and / or the position of target in image data 132, and analyze image data 132 based on user input to determine the current / preoperative spatial relationship of medical device 106 relative to target. In some cases, if control system 118 determines a difference between the position of device 106 as indicated in navigation data 126(A) (e.g., relative to preoperative mapping or target) and the position of device 106 as depicted in image data 132 (e.g., relative to target), an update operation is performed at 134 to update navigation. When an update operation is performed at 134, the control system 118 may update navigation data 126(A) and provide updated navigation data 126(B), which reflects the actual / real-time position of medical device 106 relative to a target and / or mapping. In this example, the position of device 106 as indicated in navigation data 126(A) and the position of device 106 as depicted in image data 132 are compared in the same coordinate system / space. Furthermore, in some cases, the update operation at 134 is performed based on the position of device 106 as depicted in image data 132, without determining whether there is a difference between the position of device 106 indicated in navigation data 126(A) (e.g., relative to a preoperative mapping or target) and the position of device 106 depicted in image data 132 (e.g., relative to a target). Here, even if an update operation is performed, navigation 126(A) may not actually change, such as in the absence of a difference.

[0047] The various components of the medical system 100 can be communicatively coupled to each other via a network, which may include wireless and / or wired networks. Example networks include one or more Personal Area Networks (PANs), Local Area Networks (LANs), Wide Area Networks (WANs), Internet Local Area Networks (IANs), cellular networks, the Internet, Body Area Networks (BANs), etc. In some examples, various communication interfaces may include wireless technologies such as Bluetooth, Wi-Fi, Near Field Communication (NFC), etc. Furthermore, in some examples, the various components of the medical system 100 can be connected via one or more support cables, pipes, connectors, etc., for data communication, fluid exchange, power exchange, etc.

[0048] Figure 2Example components of a control system 118 and a robotic system 102 according to one or more examples are illustrated. In various examples, the control system 118 and the robotic system 102 are implemented as a tower and a robotic cart, respectively. However, the control system 118 and the robotic system 102 can be implemented in other ways. The control system 118 may be coupled to and cooperate with the robotic system 102 to perform medical procedures. For example, the control system 118 may include a communication interface 202 for communicating with the robotic system 102 via a wireless or wired connection (e.g., to control the robotic system 102). Furthermore, in the examples, the control system 118 may communicate with the robotic system 102 to receive from it position / sensor data relating to the position of sensors associated with and controlled instruments / components. In some examples, the control system 118 may communicate with an EM field generator 120 to control the generation of an EM field in the area surrounding the patient. The control system 118 may also include a power interface 206.

[0049] The control system 118 may include control circuitry 208 configured to actuate one or more components of the medical system 100 and / or otherwise control any of the various system components, such as brackets, mounts, arms / positioners, medical devices, imaging devices, position sensing devices, sensors, etc. Furthermore, control circuitry 208 may be configured to perform other functions, such as inducing the display of information, processing data, receiving input, communicating with other components / devices, and / or any other functions / operations discussed herein.

[0050] The control system 118 may also include one or more input / output (I / O) components 210 configured to assist a physician or other person in performing medical procedures. For example, one or more I / O components 210 may be configured to receive input and / or provide output to enable a user to control / navigate the medical device 106, the robotic system 102, and / or other instruments / devices associated with the medical system 100. The control system 118 may include one or more displays 212 to provide / display / present various information regarding the procedures. For example, one or more displays 212 may be used to present navigation information, including virtual anatomical models of anatomical structures with virtual representations of the medical devices, image data, and / or other information. One or more I / O components 210 may include user input controls 214, which may include any type of user input (and / or output) device or device interface, such as one or more buttons, keys, joysticks, handheld controllers (e.g., video game type controllers), computer mice, touchpads, trackballs, control pads, sensors for capturing gestures and finger postures (e.g., motion sensors or cameras), touchscreens, toggle (e.g., button) inputs, and / or interfaces or connectors for this purpose. In the example, such input can be used to generate commands for controlling medical devices, robotic arms, and / or other components.

[0051] The control system 118 may also include a data storage device 216 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by the control circuitry 208 to cause the control circuitry 208 to perform the various operations / functions discussed herein. In the example, two or more components of the control system 118 may be electrically coupled and / or communicatively coupled to each other.

[0052] Robotic system 102 may include one or more robotic arms 104 configured to engage and / or control, for example, a medical device 106 and / or other elements / components, to perform one or more aspects of a procedure. As shown, each robotic arm 104 may include multiple segments 220 coupled to joints 222, which provide multiple degrees of motion / freedom. Robotic system 102 may be configured to receive control signals from control system 118 to perform certain operations, such as positioning one or more of the robotic arms 104 in a particular manner, manipulating instruments, etc. In response, robotic system 102 may use its control circuitry 224 to control actuators 226 and / or other components of robotic system 102 to perform operations. For example, control circuitry 224 may control the insertion / retraction, joint movement, rolling, etc., of the axis of medical device 106 or another instrument by actuating a drive output 228 of a manipulator 230 (e.g., an end effector) coupled to the base of the robotically controllable instrument. The drive output 228 can be coupled to a drive input on an associated instrument, such as the instrument base of an instrument coupled to an associated robotic arm 104. The robot system 102 may include one or more power interfaces 232.

[0053] The robotic system 102 may include a support column 234, a base 236, and / or a console 238. The console 238 may provide one or more I / O components 240, such as a user interface and / or display (or dual-purpose device, such as a touchscreen) for receiving user input to provide preoperative and / or intraoperative data to the physician / user. The support column 234 may include an arm support 242 (also referred to as “bracket 234”) for supporting the deployment of one or more robotic arms 104. The arm support 242 may be configured to translate vertically along the support column 234. This vertical translation of the arm support 242 allows the robotic system 102 to adjust the reach of the robotic arms 104 to accommodate various table heights, patient body shapes, and / or physician preferences. The base 236 may include wheel-shaped casters 244 (also referred to as “wheel 244”) that allow the robotic system 102 to move around the operating room prior to protocol. Once in the appropriate position, the casters 244 can be secured using wheel locks to keep the robot system 102 in place during the procedure.

[0054] Each joint 222 of each robotic arm 104 can be independently controllable and / or provide independent degrees of freedom for instrument navigation. In some examples, each robotic arm 104 has seven joints, thus providing seven degrees of freedom, including “redundant” degrees of freedom. Redundant degrees of freedom allow the robotic arms 104 to be controlled to position their respective manipulators 230 at specific locations, orientations, and / or trajectories in space using different link positions and joint angles. This allows the robotic system 102 to locate and guide medical instruments from desired points in space, while allowing physicians to move joints 222 to clinically advantageous positions away from the patient to achieve greater access while avoiding collisions.

[0055] One or more manipulators 230 (e.g., end effectors) may be coupled to an instrument base / handle, which in some cases may be attached using a sterile adapter component. The combination of the manipulator 230, the coupled instrument base, and any intermediate mechanisms or connectors (e.g., sterile adapters) may be referred to as a manipulator assembly, or simply as a manipulator. The manipulator / manipulator assembly may provide an electrical and / or control interface. For example, the interface may include connectors for transmitting pneumatic pressure, electrical, electrical signals, and / or optical signals from the robotic arm 104 to the coupled instrument base. The manipulator / manipulator assembly may be configured to manipulate medical devices (e.g., surgical tools / instruments) using techniques including, for example, direct drive, harmonic drive, gear drive, belt and / or pulley, magnetic drive, etc.

[0056] The robot system 102 may also include a data storage device 246 configured to store executable instruments (e.g., computer-executable instructions) that can be executed by control circuitry 224 to cause control circuitry 224 to perform the various operations / functions discussed herein. In the example, two or more components of the robot system 102 may be electrically coupled and / or communicatively coupled to each other.

[0057] Data storage devices (including data storage device 216, data storage device 246, and / or other data storage devices / memories) may include any suitable or desired type of computer-readable medium. For example, computer-readable medium may include one or more volatile data storage devices, non-volatile data storage devices, removable data storage devices, and / or non-removable data storage devices, which are implemented using any technology, layout, and / or data structure / protocol, including any suitable or desired computer-readable instructions, data structures, program modules, or other types of data.

[0058] Computer-readable media may include, but is not limited to, phase-change memory, static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other non-transitory medium that can be used to store information for access by a computing device. As used in certain contexts herein, computer-readable media generally may not include communication media, such as modulated data signals and carrier waves. Therefore, computer-readable media should generally be understood as non-transitory media.

[0059] Control circuitry (including control circuitry 208, control circuitry 224, and / or other control circuitry) may include circuitry embodied in a robot system, control system / tower, machinery, or any other component / device. Control circuitry may include any collection of processors, processing circuitry, processing modules / units, chips, dies (e.g., semiconductor dies including one or more active devices and / or passive devices and / or connection circuitry), microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines (e.g., hardware state machines), logic circuitry, analog circuitry, digital circuitry, and / or any device that manipulates signals (analog and / or digital) based on hard-coded circuitry and / or operating instructions. Control circuitry as referred to herein may also include one or more circuit substrates (e.g., printed circuit boards), conductive traces and vias and / or mounting pads, connectors, and / or components. Control circuitry may also include one or more storage devices, which may be embodied in the embedded circuitry of a single device, multiple devices, and / or devices. Such data storage devices may include read-only memory, random access memory, volatile memory, non-volatile memory, static memory, dynamic memory, flash memory, cache memory, data storage registers, and / or any device for storing digital information. In examples where the control circuitry includes hardware and / or software state machines, analog circuits, digital circuits, and / or logic circuits, the data storage device / register storing any associated operational instructions may be embedded within or outside the circuitry including the state machine, analog circuitry, digital circuitry, and / or logic circuitry.

[0060] The functions described herein can be implemented by the control circuit 208 of the control system 118 and / or the control circuit 224 of the robot system 102, such as by the control circuits 208 and 224 executing executable instructions to enable the control circuits 208 and 224 to perform functions.

[0061] Calibration, navigation and positioning modes / systems Figure 3 This is a block diagram illustrating a system 300 comprising various localization and / or imaging systems / modalities 302-312 (sometimes referred to as "subsystems 302-312") according to one or more examples, which can be implemented to facilitate anatomical mapping, navigation, localization, and / or visualization for procedures. For example, the various systems 302-312 can be configured to provide data for generating anatomical mapping maps, determining the location of instruments, determining the location of targets, and / or performing other techniques.

[0062] Each system in systems 302-312 can be associated with a corresponding coordinate system (also referred to as a “position coordinate system”) and / or can provide data / information related to the position of instruments and / or anatomical structures, wherein registering various coordinate systems with each other allows for the integration of various systems to provide mapping, navigation, and / or instrument visualization. For example, registration of various modalities with each other can allow a determined position in one modality to be tracked and / or superimposed on / within a reference system associated with another modality, thereby providing a layer of positional information that can be combined to provide a robust positioning system.

[0063] In the example, system 300 is configured to implement one or more positioning techniques (also referred to as "positioning system 300"). Positioning can refer to the process of determining the location and orientation / posture of an instrument or other element / component within a given space or environment.

[0064] In various examples, the anatomical space in which a medical device can be located (i.e., the place where the location and / or shape of the device is determined / estimated) is a 2D or 3D portion of the patient's tracheal, bronchial, vascular, urinary, gastrointestinal, or any organ or space accessed via an internal lumen. Various modalities can be implemented to provide images / representations / models of the anatomical space. For example, imaging modalities can be implemented, which may include, for example, X-ray, fluoroscopy, CT, positron emission tomography (PET), PET-CT, CT angiography, cone-beam computed tomography (CBCT), 3DRA, single-photon emission computed tomography (SPECT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound. One or both of preoperative and intraoperative images relevant to the procedure can be acquired.

[0065] Systems 302-312 can provide information for generating 2D or 3D anatomical models / maps 314 (e.g., airway models). In the example, the anatomical map 314 and / or other positioning information can be displayed to the user (such as a physician) during the procedure to assist the user in performing the procedure. For example, a visualization of the tracked medical device can be overlaid on the anatomical map 314 based on location / sensor data associated with the tracked medical device.

[0066] As shown in the figure, system 300 may include an operating table or other patient platform or positioning / support structure 302 (for convenience, referred to as "support structure 302"). The position of support structure 302 can be determined based on data maintained relating to its location within the surgical / procedural environment. Alternatively or in addition, one or more markers and / or appropriate imaging / positioning modalities may be used to sense or otherwise determine the position of support structure 302.

[0067] System 300 may also include robot system 304, such as robot system 102 (e.g., a robot cart or other device or system including one or more robot end effectors). Data relating to the position and / or state of the robot arm, actuators, and / or other components of robot system 304 may be known or derived from robot command data or other robot data relative to the coordinate system of robot system 304. In some examples, reference system registration 316 occurs between support structure 302 and robot system 304, which may be a relatively coarse registration based on robot system / cart setup procedures (which may have any suitable or desired scheme) (in some cases).

[0068] System 300 may also include an electromagnetic (EM) sensor system 306, which may include an EM field generator (e.g., EM field generator 120) and one or more EM sensors. The EM sensors may be associated with a portion of the tracked / controlled apparatus, such as along the length of the apparatus and / or another elongated member disposed in the working channel of the apparatus. In some embodiments, the EM field generator may be mechanically coupled to a support structure 302 or a robot system 304, in which case registration / association 318 between such systems can be known and / or determined. In some embodiments, registration 318 between the EM sensor system 306 and the robot system 304 may be determined by forward kinematics and / or field generator mounting transformation information. For example, the field generator may be mounted to an end effector / manipulator of the robot system 304, such that the position of the field generator relative to the robot system positioning frame can be known based on a known relationship between the position of the robot end effector and the position of the robot system 304. The EM sensor system 306 may provide apparatus posture and / or path information based on sensor readings associated with the apparatus.

[0069] System 300 may also include an optical camera system 308 comprising one or more cameras or other imaging devices configured to generate images of patient anatomy within its field of view, such as real-time image data during surgery. In the example, registration 320 between the optical camera system 308 and the EM sensor system 306 can be achieved by identifying features (such as those via the medical device tip) in the images generated by the optical camera system 308 that have associated EM sensor data. Registration 320 may also be based, at least in part, on hand-eye interaction when a physician views real-time camera images as an endoscope equipped with EM sensors navigates through the patient's anatomy.

[0070] System 300 may also include a computed tomography (CT) imaging system 310 configured to generate CT images of the patient's anatomy, which may be performed preoperatively and / or intraoperatively. In examples, image processing may be implemented for registration 322 between CT image data and camera image data generated by the optical camera system 308. For example, common features identified in both the camera image data and the CT image data may be recognized to spatially correlate CT image frames with camera image frames. In some examples, the CT imaging system 310 may be used to generate preoperative imaging data for producing anatomical mapping 314 and / or for path navigation planning.

[0071] System 300 may also include a fluorescence fluoroscopy imaging system 312 configured to generate X-ray images (e.g., 2D real-time images) of surgical sites. In some examples, a C-arm fluoroscopy is used to acquire preoperative and / or intraoperative images. The fluorescence fluoroscopy imaging system 312 may be used with a contrast agent introduced into the anatomical structures to generate image data representing the patient's anatomy and / or instruments. In examples, the fluorescence fluoroscopy imaging system 312 may be registered 324 to the CT imaging system 310 using any image processing technique suitable for such registration.

[0072] In the examples, the CT imaging system 310 and / or the fluoroscopy imaging system 312 can be registered 326 to the EM sensor system 306 using various techniques, such as tool registration, transformation functions, etc. In one example, the mechanical structure of the C-arm instruments of systems 310, 312 may have a known physical transformation / relationship relative to the mounting location of the EM field generator of the EM sensor system 306. Such known relationships can be used to register the CT / fluoroscopy image space to the EM sensor image space. Furthermore, in another example, registration 326 includes performing other operations discussed herein, such as those discussed with reference to Figure 4. Connectors 328 and 330 represent the registration / relationship between the CT imaging system 310 and the fluoroscopy imaging system 312 and the anatomical mapping map 314, respectively.

[0073] Although the CT imaging system 310 and the fluorescence fluoroscopy imaging system 312 are illustrated as separate systems, in this example, the same system performs the functions of both the CT imaging system 310 and the fluorescence fluoroscopy imaging system 312.

[0074] The position, shape, and / or orientation of an instrument (such as an endoscope) can be determined using any one or more of systems 302-312, which facilitates the generation of graphical interface data representing the estimated position and / or shape of the instrument relative to anatomical mapping 314. The position, shape, and / or orientation of the instrument and / or anatomical mapping 314 can be displayed on a display device, such as via control system 118 and / or robotic system 102 or another device. In some examples, anatomical mapping 314 also indicates the location of a target, such as a location within an anatomical structure designated for further treatment.

[0075] Although systems 302-312 are discussed in a specific order, they can be implemented in different orders. Furthermore, they can be used in different ways. Additionally, registration can occur between different systems.

[0076] In the example, any number of coordinate systems / spaces of systems 302-312 can be registered / aligned to represent / associate data from multiple coordinate systems within the same coordinate system. In some cases, the coordinate systems used may be specific, while in others, they may be less related. In the example, preoperative and intraoperative data can be represented within the same coordinate system / space. For illustration, preoperative data associated with CT imaging system 310, preoperative data associated with fluoroscopy system 312, intraoperative data associated with EM sensor system 306, intraoperative data associated with CT imaging system 310, and / or intraoperative data associated with fluoroscopy system 312 can be associated / represented within the same coordinate system (such as the intraoperative coordinate system).

[0077] In some examples, one or more of systems 302-312 may be implemented to generate an anatomical mapping 314 preoperatively (or intraoperatively) and / or determine the position of one or more targets within the anatomical mapping 314. Intraoperatively, one or more of systems 302-312 may also be implemented to determine the position of a medical device and / or the position of a target relative to the anatomical mapping 314. As discussed herein, one or more of systems 302-312 may also be implemented to update the anatomical mapping 314, the position of the medical device, the position of the target, etc.

[0078] Example Flowchart Figure 4 Figure 5 and Figure 6Example flowcharts for processes 400, 500, and 600, facilitating various operations / actions according to one or more examples discussed herein, are illustrated respectively. One or more actions / operations associated with processes 400, 500, and / or 600 can be performed by control circuitry or combinations thereof implemented in any of the devices / systems discussed herein (such as control circuitry 208 of control system 118, control circuitry 224 of robot system 102, etc.). Although various blocks / operations are illustrated in a specific order, these blocks / operations can be implemented in other orders (e.g., arranged differently). Furthermore, any block / operation can be eliminated.

[0079] Figure 4A and Figure 4B Example flowcharts are shown for updating navigation guidance 400 based on one or more examples.

[0080] exist Figure 4A In block 402, process 400 includes acquiring or generating preoperative data (also referred to as “preoperative data”), which may include preoperative mapping of anatomical structures (also referred to as “anatomical mapping”). For example, control circuitry may be configured based on any of the techniques discussed herein or other techniques (such as reference anatomical mapping). Figure 3 (Any techniques discussed) can be used to generate preoperative mappings of anatomical structures. For illustration, during the preoperative phase (e.g., prior to surgical procedures), one or more imaging techniques (e.g., CT imaging, MRI imaging, etc.) can be used to capture images of the patient's anatomical structures. The image data can be processed to generate anatomical mappings that may include / indicate various pathways / lumens and / or other features within the anatomical structure. Furthermore, preoperative data may include information about targets within the anatomical structure, such as the location designated as a target within the anatomical structure, the size of the target, etc. In the example, targets may be designated based on user input indicating / specifying the location, analysis of image data used to identify objects / targets including certain characteristics, etc. Additionally, preoperative data may indicate the location of a certain / pre-defined anatomical feature. Therefore, preoperative data can indicate the preoperative location of a target / anatomical feature within the anatomical structure, such as the position of the target / anatomical feature relative to the mapping, the position of the target / anatomical feature relative to a coordinate system (sometimes referred to as the "preoperative coordinate system"), etc. Although box 402 is discussed in the context of its execution during the preoperative phase, box 402 can be executed during the intraoperative phase and / or at other times to generate anatomical mapping, identify the location of targets, etc.

[0081] At box 404, process 400 includes receiving preoperative data about anatomical structures associated with the procedure. For example, during the intraoperative phase, control circuitry may be configured to receive preoperative data generated during the preoperative phase, such as the preoperative data generated in box 402. As described above, preoperative data may include information about anatomical mapping, targets within anatomical structures, certain anatomical features, etc. In the example, the preoperative data is generated days, weeks, etc., prior to the intraoperative phase. However, preoperative data may be generated at other times.

[0082] At box 406, process 400 includes registering / aligning preoperative data with intraoperative data. For example, control circuitry may be configured to register / align a preoperative mapping, a coordinate system / space associated with the preoperative data, the locations of anatomical features specified by the preoperative mapping and / or targets within the preoperative mapping, to the intraoperative mapping, a coordinate system / space associated with the intraoperative environment (e.g., a coordinate system used for intraoperative instrument navigation), anatomical features identified intraoperatively, etc. For illustration, the intraoperative mapping may be associated with an EM space / coordinate system used for intraoperative instrument tracking.

[0083] In some examples, the registration process involves navigating instruments intraoperatively to certain / known anatomical features identified within preoperative data within the anatomical structure. As the instruments navigate to the anatomical features, their position can be determined / tracked. Here, the preoperative position of the anatomical features can be mapped to their corresponding intraoperative positions, thereby registering the preoperative mapping to the current / intraoperative coordinate system / space. Furthermore, in some examples, other techniques are used to register preoperative data to intraoperative data. This may include using various types of intraoperative data, such as instrument position data, image data from the instruments, image data from external imaging systems, features / positions identified based on image analysis, reference markers placed on the patient near certain anatomical features, reference markers on the table / support or another object, etc.

[0084] In the example, by registering / aligning preoperative data to intraoperative data, the preoperative and intraoperative data can be associated with the same coordinate system, allowing the current position of the instrument to be represented relative to the preoperative data. For illustration, preoperative mapping and / or the target can be registered / aligned to the intraoperative coordinate system (or vice versa), where navigation information for the instrument is provided relative to the preoperative mapping. This allows the user to view the current position of the instrument and / or the preoperative position of the target relative to the preoperative mapping. However, as mentioned above, differences in the characteristics of anatomical structures between the preoperative and intraoperative stages, distortions introduced into the surgical environment, and / or other factors can lead to instrument errors (e.g., inaccuracies in providing the current / intraoperative position of the instrument because the instrument position is displayed relative to the preoperative mapping). In the example, procedure 400 is performed to update / correct inaccuracies in the navigation information. However, procedure 400 can be performed in other contexts.

[0085] At box 408, process 400 includes navigating an instrument to a target. For example, control circuitry may be configured to control a robotic system based on user input to navigate an instrument coupled to the robotic system to a target. The user may provide input to navigate the instrument based on viewing navigation information (e.g., graphical interface data) that presents the instrument's real-time position relative to anatomical mapping and / or a target (such as relative to a preoperative mapping). In the example, the instrument may be navigated to within a predetermined distance / proximity to the target and / or parked (e.g., stopped). In any case, the instrument may be guided to a position considered to correspond to the target.

[0086] At block 410, process 400 includes initiating an update to the navigation guidance. In one example, the control circuitry may be configured to receive user input requesting an update to the navigation guidance (e.g., accessing a menu and selecting a navigation update button). In response, process 400 may proceed to block 412. In another example, the control circuitry may automatically determine when to initiate an update to the navigation guidance, such as when the instrument is positioned within a predetermined proximity to the target and remains stationary for more than a threshold amount of time, or when another event occurs.

[0087] In some examples, when an update to the navigation guidance is initiated, the navigation / movement of the device is paused / stopped / restricted. For example, after an update to the navigation guidance is initiated at box 410 or at another time, a notification / message may be displayed to the user that: (i) indicates that the navigation / movement of the device is paused / stopped (e.g., the device is fixed), or (ii) requests the user to stop providing input for controlling the device. However, the navigation of the device may be paused / stopped without providing a notification / message. In any case, pausing / stopping the movement of the device may allow the collection of sensor data from the device (at box 412) or the performance of other operations while the device remains in a relatively fixed position. Furthermore, pausing / stopping the movement of the device may allow the imaging system to capture image data while preserving / enhancing image quality (in Figure 4B (at box 418 in the diagram), because image quality may be degraded due to instrument movement. In some cases, instrument movement / navigation is stopped while performing one or more boxes in boxes 412-430. However, one or more boxes in process 400 can be performed without pausing / stopping / restricting instrument movement.

[0088] At block 412, process 400 includes obtaining sensor data from the instrument. For example, control circuitry may be configured to receive sensor data from the instrument, wherein the sensor data indicates or is used to determine the position / orientation of the instrument, such as the position / orientation of the distal end of the instrument or another part of the instrument where the sensor is located.

[0089] At block 414, process 400 includes determining a first spatial relationship between the instrument and the target based on sensor data. For example, control circuitry may be configured to determine the position / orientation of the instrument relative to the target based on sensor data obtained at block 412. Here, the position / orientation of the instrument and the position of the target may be represented relative to a preoperative mapping (where the instrument position and the preoperative mapping are associated with the same coordinate system). Therefore, the first spatial relationship between the instrument and the target can be determined relative to the preoperative mapping.

[0090] exist Figure 4BIn box 416, process 400 includes placing / positioning the imaging system near the anatomical structure and / or connecting the imaging system. For example, a user may manipulate / move the imaging system to position at least a portion of the imaging system toward the patient, such as with the C-arm of the imaging system toward the area of ​​the anatomical structure being examined / treated. In some cases, a notification / message may be displayed to request the user to position the imaging system in a specific manner, such as at a specific angle relative to the patient, patient table, etc. The user may connectably / communically couple the imaging system to a control system, robotic system, etc. The imaging system may be located outside the internal anatomical site and configured to capture / generate image data depicting the internal anatomical structure. For example, the imaging system may be configured to capture / generate X-ray, CT, or other image data representing the internal anatomical structure.

[0091] Although box 416 is shown as part of process 400, in some cases the imaging system has been positioned / placed in the appropriate location and / or connected to the appropriate device / system, so that box 416 is not performed. In one example, the imaging system is part of a table / support structure on which the patient is positioned and / or has been connected to a control / robotics system.

[0092] At box 418, process 400 includes generating image data and / or receiving image data from an imaging system. For example, the imaging system may generate / capture image data depicting internal anatomical structures (e.g., scanning a patient) and / or transmit the image data to another component / device. For example, control circuitry of a control / robotics system or another system may receive image data from the imaging system. In this example, image data includes X-ray image data, fluorescence fluoroscopy image data, CT image data, positron emission tomography (PET) image data, PET-CT image data, CT angiography image data, cone-beam CT image data, 3DRA image data, single-photon emission computed tomography (SPECT) image data, magnetic resonance imaging (MRI) image data, optical coherence tomography (OCT) image data, ultrasound image data, etc.

[0093] In some examples, when image data is generated at box 418, the imaging system rotates / moves from its initial position / arrangement / orientation to its final position / arrangement / orientation. For example, upon completion of the operation at box 418, a portion of the imaging system may be placed in a lateral position. This position may be less than ideal for continuing procedure 400 and / or performing other operations. For example, this position may cause interference / distortion when acquiring sensor data from the instrument, make it more difficult to perform other parts of the procedure, or make subsequent use of the imaging system less than ideal. In some cases, a message / notification may be provided to request the user to return the imaging system to a specific position / arrangement / orientation, such as the initial or upright position or another unobstructed position. In some cases, user input indicating that the imaging system has been repositioned is received, and procedure 400 can continue. Furthermore, in some cases, the user may move the imaging system to another location away from the patient, such as to a different location in the environment.

[0094] At box 420, process 400 includes determining a second spatial relationship between an instrument and a target location based on image data. In the example, control circuitry may be configured to determine the second spatial relationship based on user input that selects / identifies the instrument and / or target in the image data. Furthermore, in the example, control circuitry may determine the second spatial relationship by performing one or more image processing techniques (e.g., artificial intelligence (AI) techniques, machine-trained models, conventional image analysis / processing, etc.). In the example, one or more image processing techniques are performed automatically, such as without using user input. In other examples, user input is used with or without image processing. In any case, the second spatial relationship may include / represent the distance and / or vector between the distal end of the instrument and the target in the image data. The second spatial relationship may represent the intraoperative or current relationship between the instrument and the target. The following... Figure 5 The example techniques for determining second spatial relationships are discussed within the context of process 500. However, other processes / operations may be performed.

[0095] At box 422, process 400 includes comparing a first spatial relationship with a second spatial relationship. In one example, the first and second spatial relationships are each represented by vectors / lines in a coordinate system / space, where each vector / line represents / indicates the distance and direction from the target to the distal end of the instrument. For example, the first vector (of the first spatial relationship) could represent the distance and direction from the instrument to the target in the coordinate system / space. The second vector (of the second spatial relationship) could represent the distance and direction from the instrument to the target in the same coordinate system / space. With the first spatial relationship represented by a first vector and the second spatial relationship represented by a second vector, the first vector can be compared with the second vector to determine differences in one or more parameters / values / properties between the first and second vectors. In doing so, various vector comparison techniques / properties can be used / compared, such as Euclidean distance, dot product, cross product, Manhattan distance, angle between vectors, vector norm, projection, magnitude, direction, angle, etc. Therefore, in the example at box 422, the difference between the first and second spatial relationships can be determined.

[0096] At box 424, process 400 includes determining whether to update the graphical interface data. For example, in some cases, at box 424, the control circuitry may determine whether the difference between the first spatial relationship and the second spatial relationship meets one or more criteria, such as the difference being non-zero (e.g., the difference between one or more parameters / values / characteristics of the first spatial relationship and one or more parameters / values / characteristics of the second spatial relationship is non-zero, exceeding a threshold). However, one or more other criteria may be used.

[0097] If it is determined that the graphical interface data will not be updated, process 400 can proceed to box 426. Alternatively, if it is determined that the graphical interface data will be updated, process 400 can proceed to box 428.

[0098] At box 426, process 400 includes maintaining graphical interface data. For example, graphical interface data representing anatomical mapping, instrument position (based on sensor data obtained from the instruments at box 412), and / or target position may be displayed. Example techniques for providing graphical interface data are referenced below. Figure 6 Let's discuss this further. At box 426, the control circuitry is able to maintain the graphical interface data in an invariant format, such as by continuing to provide / present the graphical interface data without changing the position of the instrument and / or the target.

[0099] At box 428, process 400 includes updating graphical interface data. For example, control circuitry may be configured to update graphical interface data representing anatomical mapping, instrument position, and / or target position, such that the updated graphical interface data indicates a second spatial relationship between the instrument and the target. In some cases, the update is based on the second spatial relationship and / or the difference between the first and second spatial relationships. The update may include moving the instrument's position (e.g., moving an instrument indicator), moving the target's position (e.g., moving a target indicator), shifting / updating / editing / replacing the anatomical mapping, and / or otherwise updating the graphical interface data. In the example, a message / notification may be displayed to the user to confirm that the user expects the graphical interface data to be updated. The graphical interface data may be updated at box 428 in response to receiving user input including confirmation of updated graphical interface data. Therefore, by updating the graphical interface data, the navigation information of the instrument can be updated.

[0100] At box 430, process 400 includes causing updated graphical interface data to be displayed. For example, control circuitry may cause the updated graphical interface data to be displayed via a display associated with a control system, robotic system, or other system. In one example, causing the updated graphical interface data to be displayed includes displaying the graphical interface data via a display / shown screen. In other examples, causing the updated graphical interface data to be displayed includes sending / providing the updated graphical interface data to a display / shown device for displaying the updated graphical interface data (e.g., content). In some cases, control circuitry may allow the user to provide input to cancel updates to the graphical interface data (e.g., revert to previous / initial graphical interface data) and / or to re-update the graphical interface data (e.g., provide new user input to specify the position of the instrument / target within the image data and re-determine the second spatial relationship).

[0101] Although process 400 exemplifies boxes 424 and 426, in this example, process 400 may continue updating the graphical interface data (at box 428) without performing the operations of box 424 (and / or box 426). In one example, process 400 may continue from box 422 without implementing boxes 424 and / or 426. For example, process 400 may include updating the graphical interface data (at box 428) when determining the difference between a first spatial relationship and a second spatial relationship (at box 422). Here, at box 428, process 400 may include updating the graphical interface data based on the second spatial relationship and / or the difference between the first and second spatial relationships. In some cases, the operations at box 428 may be performed even if the graphical interface data has not actually changed when box 428 is completed (e.g., the difference between the first and second spatial relationships is zero). Therefore, in some cases, box 424 may not be performed.

[0102] Although various examples are discussed in the context of displayable graphical interface data, the techniques discussed herein can alternatively or additionally generate / update navigation data that is not displayable and / or used for other purposes (e.g., navigation data for tracking the location of an instrument, which may or may not be displayable).

[0103] In this example, error management operations / techniques (also known as "error handling techniques") are implemented to detect and / or resolve errors / failures / problems occurring during process 400 (or other processes discussed herein). In some cases, error management techniques include providing one or more messages / notifications (also known as "error messages" or "warning messages") that include information about the error / warning (e.g., the type of error / warning or other details about the error / warning, corrective actions to resolve the error / warning, etc.), requests to the user to perform specific actions to resolve the error / warning, etc. Error / warning messages may be displayed to the user, provided via audio output, via haptic output, and / or other types of output. The user can view the error / warning message and provide user input to confirm that the error / warning message has been viewed, confirm that a corrective action has been performed, initiate a corrective action, etc. In some cases, handling / resolving errors / warnings includes returning to a box in process 400 to start again from a specific point in process 400, or re-performing the action associated with a specific box before continuing process 400. Error management techniques can typically be implemented by control circuitry.

[0104] While example error management techniques typically provide error / warning messages to the user, in some cases, error management techniques handle errors / warnings automatically without providing error / warning messages and / or without receiving user input regarding errors / warnings. Furthermore, although this document discusses the triggering of various errors at certain times during process 400, errors / warnings can be triggered at other times. Moreover, although examples are discussed in the context of providing error / warning messages and continuing when the error / warning is resolved (e.g., the user provides input indicating that the error / warning has been resolved, or it is detected that the error / warning has been resolved, etc.), in some cases, process 400 continues as usual even when an error / warning message is provided.

[0105] In the example, a device attachment error (also known as a “device unloading error”) is triggered when the device is decoupled / detached from the robot system. For example, control circuitry may be configured to detect when a device (and / or other components associated with the device, such as an entry sheath) is removed / detached from the robot system's robot arm / component during any block of process 400. In response, control circuitry can provide notification to reconnect / reattach the device to the robot system and a message that certain operations will be performed again once the device is reconnected. To illustrate, if a device is removed from the robot system during one or more blocks 410-430, process 400 may exit a navigation-guided update (e.g., initiated at block 410), return to block 410, and restart process 400 from that point, etc. In some cases, process 400 may not restart until the device is detected as reconnected to the robot system.

[0106] In the example, a device sensor error is triggered when there is a problem with the sensor data or the device's sensor. For example, the EM sensor or another sensor on the device may experience distortion, loss / reduction in the sensed EM field, malfunction, or other problems. This can degrade the quality of the sensor data. In some cases, moving the imaging system closer to the patient before completing box 412 may cause distortion or other problems. In one example, the control circuitry may be configured to detect (at / after box 412) sensor data associated with distortion exceeding a threshold amount, quality degradation exceeding a threshold amount, etc. Additionally or alternatively, the control circuitry may be configured to detect (at / after box 412) loss / reduction in received sensor data (e.g., receiving fewer than a certain number of EM samples / readings within a predetermined time period). In any case, in response to such detection, the control circuitry may provide a notification to remove the source of distortion and / or re-initiate an update to the navigation guidance. Here, once the source of distortion has been removed and / or user input has been provided to restart, process 400 can restart at box 412 to recapture sensor data. In another example, the control circuitry may be configured to perform a similar detection (at / after box 420), provide notification of an instrument sensor error, and restart at box 412. In other examples, the instrument sensor error is triggered at other times during process 400. Although various examples are discussed in the context of restarting at box 412 to recapture sensor data, in some cases, a warning notification indicating that relatively low-quality image data has been obtained and will be used is provided. Here, process 400 can continue as usual without returning to the previous box. Furthermore, in some cases, instead of restarting at box 412 to recapture sensor data, process 400 may exit navigation guidance updates (e.g., initiated at box 410), thus requiring the user / system to re-initiate the navigation guidance update if necessary.

[0107] In the example, an instrument sensor warning / error is triggered when there is a problem with the sensor data. For example, if the imaging system remains in the end position (e.g., lateral position) when the operation at box 418 is completed, the imaging system may interfere with the acquisition of sensor data from the instrument, as described above. In some cases, sensor data is received from the instrument during the update of graphical interface data at box 428. Therefore, when sensor data is detected to be associated with distortion exceeding a threshold amount, sensor data does not meet one or more quality criteria, or sensor data is not received for a period of time (e.g., fewer than a certain number of EM samples / readings are received within a predetermined time period), a warning / error message indicating poor / low quality of the sensor data and / or requesting that the imaging system be positioned / moved to a specific location / arrangement / orientation (e.g., initial or another unobstructed location / arrangement / orientation) can be provided. Although the example is discussed in the context of the imaging system potentially causing interference, other objects / items may cause undesirable effects, where the warning / error message may request the removal of any objects from the space that may potentially affect the sensor data.

[0108] In the example, an imaging system error (sometimes referred to as an "image data error") is triggered when there is a problem with the image data or the imaging system configured to capture the image data. In one example, the control circuitry may be configured to detect, for example, that the imaging system is not connected or not properly connected to the control / robot system (at / before / after or at any other time) by detecting that no signal is received from the imaging system. In response, the control circuitry may provide a notification indicating that the imaging system is not detected / connected. This notification may provide a request to check the network connectivity, reconnect the imaging system, and / or recapture the image data using the imaging system (execute box 416 / 418 again). In response to receiving user input confirming that the imaging system is reconnected, the control circuitry may return to box 416 / 418. Although the example of an image system error has been discussed in the context of restarting at box 416 / 418, in some cases, a warning notification indicating that relatively low-quality image data has been obtained and that this image data will be used may be provided. In this case, process 400 may continue as normal without returning to the previous box.

[0109] In another example of an imaging system error (sometimes referred to as an "image data error"), the control circuitry may be configured to detect (at / before / after box 418) that image data from the imaging system does not meet one or more criteria. In response, the control circuitry may provide a notification indicating that the image data does not meet certain requirements and / or to recapture the image data using the imaging system (by re-executing box 418). In some cases, the image data does not meet one or more criteria when the orientation tilt / angle of the imaging system relative to the patient's body (e.g., cranial / caudal angle) is not within a predetermined range. Therefore, the notification may indicate that the orientation tilt / angle of the imaging system is incorrect and / or to position the imaging system with a specific orientation tilt / angle. In response to receiving user input confirming that the imaging system is correctly positioned, the control circuitry may return to box 418.

[0110] In another example of an imaging system error (sometimes referred to as an "image data error"), the control circuitry may be configured to detect (at / before / after box 418) missing image data (such as one or more image slices) from the imaging system, and / or a transmission error has occurred. In response, the control circuitry may provide a notification instructing the image data to be recaptured using the imaging system (by re-executing box 418). Upon receiving user input to restart image generation, the control circuitry may return to box 418.

[0111] Figure 5 An example flowchart illustrates a process 500 for determining the spatial relationships between elements depicted within image data, based on one or more examples.

[0112] At box 502, process 500 includes causing image data depicting internal anatomical structures to be displayed. For example, control circuitry may cause the display of image data received from or generated by an imaging system. In one example, causing the image data to be displayed includes displaying the image data via a monitor / display screen. In other examples, causing the image data to be displayed includes sending / providing the image data to a monitor / display device for display. In some cases, the image data may be displayed via an interface, such as those discussed in further detail below. Figure 7 and / or Figure 8 The interface illustrated in .

[0113] At block 504, process 500 includes receiving user input to identify a target in image data. For example, control circuitry may be configured to receive user input (also referred to as “user input data”) via I / O components of a control / robotics system. User input may include a selected / marked location on the image data representing / identifying a target (e.g., the center or another portion of the target). In some cases where the image data comprises multiple image / image data slices (also referred to as “image slices”), the user may navigate within the image data slices to identify slices depicting targets with the largest volume / area or other desired characteristics. The user may provide input to specify the center of the target in such image data. However, the user may specify the target in any image slice and / or any portion of the target. Each image slice may be a cross-sectional image. References below... Figure 7 Discuss example user interfaces and techniques used to specify the target.

[0114] At block 506, process 500 includes receiving user input to identify an instrument in the image data. For example, control circuitry may be configured to receive user input via I / O components of a control / robotics system. User input may include a selected / marked location on the image data representing / identifying an instrument (e.g., the distal end / terminus or another portion of an endoscope, the distal end / terminus or another portion of an instrument / tool ​​deployed through the endoscope, the distal end / terminus or portion of another instrument depicted in the image data, etc.). In some cases where the image data comprises multiple images / image data slices, the user may navigate within the image data slices to identify the distal end of the instrument. The user may provide input to specify the distal end of the instrument in the image data. The user may specify any portion of the instrument in any image slice. References below... Figure 8 This section discusses example user interfaces and techniques used to specify instruments.

[0115] At box 508, process 500 includes analyzing image data to identify the location and / or orientation of an instrument (and / or target). For example, user input received at box 506 regarding the image data may indicate a location / point of interest for the instrument. Here, one or more image processing techniques may be used to analyze the image data to detect / identify one or more image features within a predetermined proximity to the location / point of interest indicated in the user input. One or more image features may be analyzed to identify certain image features related to the instrument, such as image features including known / predetermined characteristics. Image features may include edges, corners, textures, intensity, etc. Once the instrument is identified in the image data, the orientation of the instrument can be determined, such as by identifying / mapping a longitudinal axis passing through the center of the distal end / part of the instrument, where the orientation of the instrument may correspond to the orientation of the longitudinal axis. In some cases, similar analysis may be performed to determine the location / orientation of a target, such as based on user input received at box 504.

[0116] Furthermore, in the example, image data can be analyzed without user input to estimate locations / points of interest within the image data, such as based on the characteristics of the instrument. For example, one or more image processing techniques can analyze image data to detect / identify points / regions of interest associated with known / predetermined shapes, sizes, and / or other characteristics classified as associated with an instrument. In some cases, similar analysis can be performed to determine or estimate the location / orientation of a target. In the example, machine learning or other artificial intelligence (AI) techniques can be used to train algorithms / models to detect / identify instruments within image data. Such techniques can be trained based on manually or automatically labeled training data, such as image data labeled to specify the instruments within it. The algorithm / model can be used as part of one or more image processing techniques.

[0117] Although box 508 is shown as part of process 500, in some cases box 508 is omitted or performed on a specific element(s) within the image data. In one example, process 500 may determine the position of the instrument / target based on coordinates associated with user input (e.g., user-selected coordinates regarding the image data). Here, image processing may or may not be performed.

[0118] At block 510, process 500 includes causing an instrument indicator to be displayed on the image data. For example, control circuitry may be configured to display an instrument indicator representing the distal end of an instrument, wherein the orientation of the instrument indicator is aligned with a determined orientation of the instrument. The instrument indicator may be displayed overlaid on the image data to indicate the position and / or orientation of the distal end of the instrument. Although many examples discuss the instrument indicator as representing the distal end of an instrument, the instrument indicator can represent any part of the instrument. In some cases, target indicators representing a target are also displayed in a similar manner, such as being displayed at a defined location in the image data. As similarly noted above, causing the display of an instrument / target indicator may include: displaying the instrument / target indicator, sending / providing interface data to / from a display / display device for display, etc.

[0119] At block 512, process 500 includes receiving user input, which includes adjustment and / or confirmation of the instrument indicator. For example, the instrument indicator may be configured to be manipulated based on user input, such as repositioning, reorientation, etc. In the example, control circuitry may receive user input as needed to adjust the position / orientation of the instrument indicator on the image data such that the instrument indicator covers the distal end of the instrument in the image data and / or is oriented to match the orientation of the distal end of the instrument in the image data. Once the instrument indicator is aligned to the user's satisfaction, the user can provide user input, which includes confirmation that the instrument indicator is accurately positioned on the image data. In some cases, the user may similarly provide user input regarding the displayed target indicator.

[0120] At block 514, process 500 includes determining a spatial relationship between an instrument and a target based on image data and / or user input. In the example, control circuitry may be configured to determine the spatial relationship based on user input selecting / recognizing the instrument and / or target in the image data, image processing based on recognizing the instrument and / or target in the image data, and / or based on the position of the instrument / target indicator. As described above, in some cases, the spatial relationship may include / represent a distance and / or vector between the distal end of the instrument and the target in the image data. For example, the spatial relationship may be represented by a vector between a point in coordinate space associated with the instrument indicator and a point in coordinate space associated with the target. Here, the vector may include a first set of coordinates for the starting point and a second set of coordinates for the ending point.

[0121] Figure 6 An example flowchart illustrating a process 600 for providing navigation guidance for a medical device, according to one or more examples, is shown. Process 600 may be performed at different times. In one non-limiting example, process 600 is performed as part of an intraoperative procedure, wherein the device is navigated within the anatomical structure to reach a target. However, process 600 may be performed at other times.

[0122] At box 602, process 600 includes navigating an instrument within an anatomical structure. For example, control circuitry may be configured such that the instrument is controlled by a manipulator of a robotic system to navigate within the anatomical structure. In this example, a user may provide user input via a control system, causing the control system to generate robot command data to control the robotic system to manipulate the instrument, such as propulsion / retraction, joint movements, or other movements. The instrument may be navigated to a target that may be associated with an object (also referred to as an "anatomical site," "procedural site," or "desired anatomical site"). Although discussed in the context of a robotic system manipulating an instrument, in some cases, a user may manipulate the instrument partially or entirely by hand / manual means.

[0123] At block 604, process 600 includes receiving sensor data from sensors associated with the instrument. For example, control circuitry may be configured to receive sensor data from sensors associated with the instrument, such as when the instrument is navigating within an anatomical structure. The sensors may be located at the distal end of the instrument, along the instrument, and / or at other locations.

[0124] At box 606, process 600 includes determining the position and / or orientation of an instrument. For example, control circuitry may be configured to use sensor data to determine the position / orientation of the instrument. The position / orientation of the instrument can be the position of the instrument relative to an anatomical mapping, coordinate system / space, etc. The position / orientation of the instrument can be represented by X, Y, Z coordinates, orientation information (e.g., roll / rotation, pitch, yaw), etc. In some cases, the rotation / roll of the instrument (e.g., the clock direction of the distal end of the instrument) is useful when the instrument includes an off-axis imaging device, working channel, or other elements. In the example, the posture of the instrument includes position and orientation, which can be the posture of the distal end of the instrument.

[0125] In the example, at box 606, the direction in which the instrument points / is pointing is also determined. For example, the orientation of the distal end of the instrument can be determined, as well as a vector aligned with the longitudinal axis of the instrument's elongated shaft and extending from its distal end, such as to indicate the direction in which the distal end of the instrument points.

[0126] At block 608, process 600 includes generating graphical interface data related to instrument navigation. For example, control circuitry may be configured to generate graphical interface data (sometimes referred to as "navigation data") indicating the position of the instrument and / or the target. The graphical interface data may include / represent anatomical mapping, where the position of the instrument / target can be depicted / represented relative to the anatomical mapping. Thus, the graphical interface data can indicate the spatial relationship between the instrument and the target.

[0127] At box 610, process 600 includes causing graphical interface data to be displayed within an interface. For example, control circuitry may cause the graphical interface data to be displayed via a display associated with a control system, robotic system, or other system. In an example, causing the graphical interface data to be displayed includes displaying the graphical interface data via a display / shown screen. In other examples, causing the graphical interface data to be displayed includes sending / providing the graphical interface data to a display / shown device for displaying the graphical interface data (e.g., content). The graphical interface data may include one or more interface elements (e.g., indicators, icons, visualizations, etc.) that indicate the position / orientation of an instrument within an anatomical structure, the position / orientation of a target, anatomical features, etc. In an example, the graphical interface data includes anatomical mapping, such as virtual anatomical mapping or other mapping of anatomical structures. In an example, causing the graphical interface data or other types of interface data to be displayed includes using the graphical interface data to generate displayable data.

[0128] In the example, when navigating an instrument within an anatomical structure, one or more boxes in boxes 604-610 are executed. For example, the position of the instrument can be tracked and displayed in real time to assist the user in navigating the instrument within the anatomical structure. Here, the user can view the real-time position of the instrument relative to the anatomical mapping and / or target, which can help the user reach the target with the instrument.

[0129] While various examples have been discussed in the context of generating / providing navigation information in the form of displayable data (e.g., displayable graphical interface data), these techniques can be similarly performed to generate / provide other types of output, such as haptic feedback, audio output, olfactory output, augmented reality output, light output (e.g., flashing lights), projection output, etc. Furthermore, navigation data that may not be configured for display / output can be generated. For example, navigation data may include any information in graphical interface data formats that are either displayable or not.

[0130] Example interface Figure 7 , Figure 8 and Figure 9 Example interfaces 700, 800 and 900 (sometimes referred to as “graphical user interface (GUI)”) are illustrated respectively, based on one or more examples, for facilitating various functions associated with updating navigation guidance.

[0131] Figure 7 and Figure 8 Example interfaces 700 and 800 are shown to provide intraoperative information about instruments / targets and / or receive input about the location of the target / instrument within the anatomical structure. Figure 7The interface 700 (also known as the "target selection interface 700") allows the user to select the location of the target within the image data, while Figure 8 The interface 800 (also known as the "instrument selection interface 800") allows the user to select the position of the instrument within the image data.

[0132] As shown in the figures, interfaces 700 and 800 include an instrument view / segment 702 that provides image data captured from the distal end of an instrument (e.g., an endoscope) within an anatomical structure. For example, instrument view 702 may display real-time image data from a camera / imaging device located at the distal end of the instrument. Here, the real-time image data depicts internal anatomical structures, such as the anatomical lumen in which the instrument is located.

[0133] Furthermore, interfaces 700 and 800 include a mapping view / segment 704 that provides a model / representation (e.g., a virtual representation) of the anatomical mapping 706 and instrument / target location information. Mapping view 704 depicts instruments within an anatomical structure having instrument representations / indicators 708, which in this example extend through the anatomical mapping 706. Mapping view 704 also depicts the location of targets with target representations / indicators 709. In this example, mapping view 704 depicts 3D information, such as a 3D anatomical mapping 706, and planes 710 associated with image views 712(A)-712(C). A user can select one of the image views 712 and can select / mark a corresponding plane in the planes 710, and vice versa, thereby assisting the user in referencing the image views 712. As shown, planes 710 can be orthogonal to each other.

[0134] Each image view 712 includes an image captured by an imaging system (such as a CT imaging system, X-ray imaging system, etc.). Each image view 712 may include image data representing slices / layers (such as 2D slices / layers) of internal anatomical structures. In this example, the image data from the imaging system includes certain characteristics that differ from other forms of image data. For example, image data from the imaging system may use different grayscale values ​​to represent / represent different tissue types. The image data may also include other features common to the type of imaging system used. Figure 7 and Figure 8 In the example, the image data is CT image data.

[0135] like Figure 7 and Figure 8As shown, image view 712 depicts / illustrates various anatomical features that can be recognized by an individual (such as an individual trained to examine CT image data). For example, image view 712 shows instrument feature 714 representing an instrument within an anatomical structure and target feature 716 representing a target within an anatomical structure. Here, instrument feature 714 (also referred to as "image feature 714") represents the distal end of an instrument (e.g., an endoscope), while target feature 716 (also referred to as "image feature 716") represents a pulmonary nodule within a lung. However, other features may be depicted / represented.

[0136] Interfaces 700 and 800 also include a control / interface element 720 configured to adjust one of the image views 712. For example, a user can use the control element 720 to move (e.g., scroll) the selected image view 712(C) up and down. This allows the user to position the desired portion of the image within the center of the image view 712(C) or otherwise view features within the image view 712(C). Alternatively or additionally, the control / interface element 720 can allow the user to navigate among multiple image slices associated with anatomical structures.

[0137] As described above, interfaces 700 and 800 allow users to select the location of instruments and / or targets within image view 712. For example, as Figure 7 As shown, the target selection interface 700 can request a user to position a marker / pointer on a target within image data of one or more image views 712. The target may include certain features that can be recognized by a trained individual. Here, the user moves an indicator 718 (also referred to as a "marker" or "marker") within the target selection interface 700 to place the indicator 718 on a target feature 716 representing the target. The user can seek or be requested to place the indicator 718 on the center / center portion of the target feature 716, such that the indicator 718 in each image view 712 is placed at the center of the target feature 716 in the corresponding image view 712. However, the indicator 718 can be correctly placed on the target feature 716 when it is above at least a portion of the target feature 716. As shown, each image view 712 may include a corresponding indicator 718, where moving one of the indicators 718 can move one or more other indicators 718. Therefore, the indicators 718 can move collaboratively. In the example, the user may be asked to place the indicator 718 in each of the three views 712, where the user may need to provide input to adjust the position of the indicator in the corresponding image view 712 to ensure that the three indicators 718 are correctly positioned (e.g., at the 3D center of the target).

[0138] After the indicator 718 is placed as needed, the user can provide user input indicating that the location has been confirmed / identified / marked. User input can be provided via the target selection interface 700 (e.g., selecting a confirmation button) or otherwise. This allows the target's position / orientation to be determined, such as relative to a coordinate system associated with the imaging system, a coordinate system associated with the image data captured by the imaging system, and / or another coordinate system. In the example, the target's position / orientation in the image coordinate system can be transformed to another coordinate system, such as the sensor / EM coordinate system used for tracking instruments (e.g., another intraoperative coordinate system). In the example, the target's position is represented by X, Y, Z coordinates.

[0139] In addition, such as Figure 8 As shown, the instrument selection interface 800 can request the user to position a marker / pointer on the distal end (or another portion) of an instrument within the image data of one or more image views 712. The distal end of the instrument may include certain features that can be recognized by a trained individual. Here, the user moves an indicator 718 within the instrument selection interface 800 to place the indicator 718 on the instrument feature 714 representing the instrument within an anatomical structure. As described above, each image view 712 may include a corresponding indicator 718, where moving one of the indicators 718 can move one or more other indicators 718. Thus, in a manner similar to that discussed above for the target, the user can place the indicator 718 on any image view 712 on the instrument feature 716. In the example, the user may be requested to place the indicator 718 precisely in each of the three views 712, where the user may need to provide input to adjust the position of the indicator in the corresponding image view 712 to ensure that the three indicators 718 are correctly positioned (e.g., at the distal end of the instrument). In some examples, in addition to Figure 8 In addition to the indicator 718 used to specify the instrument, Figure 7 The indicator 718 used to specify the location of the target is also included. Figure 8 Within the 800-degree interface. This allows users to view the target's location when attempting to position the distal end of the instrument.

[0140] After the indicator 718 is placed as needed, the user can provide user input confirming / identifying / marking the indicated location. User input can be provided via the device selection interface 800 (e.g., selecting a confirmation button) or otherwise. This allows the device's position / orientation to be determined, such as relative to a coordinate system associated with the imaging system, a coordinate system associated with the image data captured by the imaging system, and / or another coordinate system. In the example, the device's position / orientation in the image coordinate system can be transformed to another coordinate system, such as the sensor / EM coordinate system used for tracking the device. In the example, the device's position / orientation includes the position / orientation of the distal end of the device, expressed in X, Y, Z coordinates and / or roll / rotation, pitch, yaw.

[0141] In some examples of determining the orientation / position of an instrument, a message / notification may be presented via an instrument selection interface 800 or another interface, requesting the user to rotate / move image data until the instrument faces the user (e.g., where the longitudinal axis of the instrument extends into or out of the display screen, such as perpendicular to the display screen). For example, the user may provide input to update one of the image views 712 such that instrument feature 714 is positioned to indicate the instrument extending out of the display screen (e.g., the distal end of the instrument points towards the user). The user may provide input to instruct the instrument to be aligned as requested. In some cases, such alignment can aid in analyzing image data to determine the orientation / position of the instrument. For example, the longitudinal axis of the instrument may be determined to extend perpendicular to the display screen. However, in many examples, the instrument may not need to be aligned in this manner to determine its orientation / position.

[0142] exist Figure 7 and Figure 8 In the illustrated example, indicator 718 takes the form of a crosshair / marker; however, any type of indicator / user interface element can be used to mark, annotate, select, or otherwise specify the position of an element. For illustration purposes, indicator 718 can take the form of a cursor, pointer, circle, or any other shape or form.

[0143] Figure 9 An example interface 900 is illustrated for providing input regarding the position and / or orientation of the distal end of an instrument (e.g., to adjust / confirm the position of the instrument). In this example, interface 900 is provided when a user identifies / marks the position of a target and / or the position of the instrument via target selection interface 700 and / or instrument selection interface 800. For example, based on user input specifying the position of the target and / or the instrument within interfaces 700 / 800, image data can be analyzed to determine or estimate the orientation / position of the instrument, as discussed herein. Information regarding the orientation / position of the instrument can then be displayed to the user via interface 900 to confirm and / or adjust the orientation / position.

[0144] As shown in the figure, interface 900 includes views / sections 902, 904, and 906, illustrating an instrument indicator 908 (also referred to as "end indicator 908") positioned on an instrument 910 (also referred to as "instrument feature 910" or "image feature 910") depicted in the image data. Here, view 902 illustrates a 3D view of the image data captured by the imaging system (e.g., 3D CT image / model data), while views 904 and 906 illustrate 2D views of the image data (e.g., 2D CT image data from different viewpoints / planes). The instrument indicator 908 can initially be placed on the image at a position and / or orientation determined based on image processing and the initially selected user position (as referenced above). Figure 8 (As discussed). The instrument indicator 908 can be displayed over the image data in an overlay manner (e.g., on top of the image data).

[0145] In the example, a user can interact with / manipulate instrument indicator 908 to align instrument indicator 908 with instrument 910. For example, instrument indicator 908 can be a selectable / movable interface element, whereby the user can select instrument indicator 908 and move it to a desired position within any of the views 902-906. Furthermore, in the example, interface 900 may include interface controls / elements 912, 914 to reposition / align instrument indicator 908 on instrument 910. Interface control 912 can move instrument indicator 908 (or, in some cases, image data) up, down, left, and right relative to image data. Simultaneously, interface control 914 can rotate instrument indicator 908, thereby adjusting the orientation of instrument indicator 908. Since view 904 is selected, interface controls 912, 914 are illustrated for view 904. However, when other views are selected, interface controls 912, 914 can be similarly provided for other views in views 902, 906. This allows the user to reposition the instrument indicator 908 using interface controls 912, 914 in any of the views 902-906.

[0146] In any case, the user can provide input as needed to change the position and / or orientation of the instrument indicator 908 to ensure that the instrument indicator 908 is accurately positioned above the instrument 910 depicted in the image data. In the example, moving the instrument indicator 908 in one of the views 902-906 can move the instrument indicator 908 in another view of 902-906. That is, the movement of the instrument indicator 908 can be relative to different views 902-906.

[0147] As shown in the figure, interface 900 can provide an image / indication 916 of an instrument indicator accurately positioned on the distal end of the depicted instrument and / or an image / indication 918 of an inaccurately positioned instrument indicator. This can assist the user in aligning the instrument indicator 908 on the instrument 910 within views 902-906. Furthermore, in situations such as... Figure 9 In the example shown, interface 900 includes an instrument view / segment 920 that provides image data captured from the distal end of an instrument within an anatomical structure. However, in some examples, the instrument view 920 may not be displayed.

[0148] When aligning the instrument indicator 908 as needed, the user can provide user input to confirm the position / orientation of the instrument indicator 908. User input can be provided via interface 900 (e.g., selecting a confirmation button) or otherwise. This allows the position / orientation of the instrument to be determined, such as relative to a coordinate system associated with the imaging system, a coordinate system associated with the image data captured by the imaging system, and / or another coordinate system. Therefore, the position of the instrument indicator 908 can be translated / calibrated to a position within the coordinate system. In the example, the position / orientation of the instrument is represented using X, Y, Z coordinates and / or roll / rotation, pitch, and yaw.

[0149] In the example, when confirming the position / or orientation of instrument indicator 908 and / or determining the position / or orientation of the instrument, the spatial relationship between the target and the instrument can be determined. This spatial relationship can be represented by a vector from the user-provided instrument position to the user-provided target position. This spatial relationship can be used to update navigation information.

[0150] exist Figure 9 In some examples of aligning the instrument indicator 908 within a specific context, the interface 900 may restrict the degrees of freedom for placing / moving the instrument indicator 908. For example, the interface 900 may provide two orthogonal views, such as view 904 corresponding to a first plane and view 906 corresponding to a second orthogonal plane, wherein in each view, the user can rotate / translate / move the instrument indicator 908 within the plane of that view. This can assist the user in accurately aligning the instrument indicator 908, such as by allowing the user to position the instrument indicator 908 in a specific view / plane at a time, without having to consider the 3D position of the instrument indicator 908 simultaneously.

[0151] Despite Figures 7 to 9 Elements are displayed within a specific interface, but any element can be eliminated and / or combined in different formats. For example, a view shown in one interface can be eliminated and / or provided in another interface.

[0152] Exemplary navigation update Figure 10-1 and Figure 10-2 Examples are shown of updating navigation information / data related to the location of the instrument based on one or more examples. Figure 10-1 An example is shown in representation 1002 of the initial navigation information for the assumed / estimated position of the indicating device, while Figure 10-2 Example 1004 illustrates an updated navigation information representation of the actual position of the indicating device. The navigation information can be displayable or non-displayable data. Figure 10-1 and Figure 10-2 In the navigation information, various representations are provided to illustrate different aspects of the navigation information. However, this information may or may not be displayed to the user.

[0153] like Figure 10-1 As shown, the navigation information may include an anatomical mapping map 1006 representing other features within the anatomical lumen / access and / or anatomical structures. In this example, the anatomical mapping map 1006 represents a portion of the bronchial tree / bronchial network of the patient's lungs. In this example, the anatomical mapping map 1006 is a preoperative mapping map registered / aligned to the intraoperative coordinate system / space. The navigation information may also include / indicate the position of the instrument 1008 within the anatomical mapping map 1006. Furthermore, the navigation information may initially include / indicate the position of the preoperative target 1010 within the anatomical mapping map 1006. Here, the distal end of the instrument 1008 is positioned... p1 The preoperative target 1010 is axially aligned with the instrument 1008 and positioned relatively close to the instrument 1008.

[0154] Figure 10-2 This illustrates the navigation information after it has been updated based on the techniques discussed herein. In other words, the updated navigation information indicates the actual, more accurate intraoperative position of instrument 1008. For example... Figure 10-1 and Figure 10-2 As shown, vector 1012 can represent the spatial relationship between the distal end of instrument 1008 and intraoperative target 1014, wherein vector 1012 can be determined based on the techniques discussed herein, such as the user specifying the position of the distal end of the instrument and the position of the target based on intraoperative image data. Although the position of instrument 1008 is updated in this example, anatomical mapping 1006 and / or other information can be updated as discussed herein.

[0155] Figure 10-1 This illustrates that instrument 1008 is not actually precisely positioned relative to the target. Therefore, the navigation information is updated to reposition instrument 1008 to its accurate location relative to the target. This can be achieved by shifting / moving the position of the distal end of instrument 1008 to a point within the anatomical mapping 1006. p2 And this is achieved by aligning the intraoperative target 1014 with the preoperative target 1010, such as... Figure 10-2 As shown. Therefore, the target is now relative to Figure 10-2 The view is accurately positioned above and to the right, not as previously... Figure 10-1 It is positioned directly in front of the device 1008 as shown in the diagram.

[0156] Example user interface update Figure 11-1 and Figure 11-2 An example interface 1100 (also referred to as "user interface 1100") based on one or more examples is illustrated, which provides / presents graphical interface data of navigation information about the instrument before and after updating navigation information. Figure 11-1 The initial navigation information is illustrated, while Figure 11-2 The updated navigation information is shown as an example. Interface 1100 can be displayed via a monitor / display device.

[0157] In the example, interface 1100 is provided during navigation updates. For example, a navigation interface (not shown) may be provided during normal navigation of the instrument, offering various views to assist the user in navigating the instrument within anatomical structures. The navigation interface may include one or more views of interface 1100 or other views. In any case, in some examples, it may initially be displayed upon initializing a navigation update. Figure 11-1 The interface is 1100. Then, after updating the navigation information, it can be displayed. Figure 11-2 The interface is 1100. However, Figure 11-1 and / or Figure 11-2 The interface 1100 may be available at other times.

[0158] like Figure 11-1 and Figure 11-2 As shown, interface 1100 includes a mapping view / segment 1102 that provides a model / representation of anatomical mapping 1104 (e.g., a virtual mapping). Anatomical mapping 1104 can be a 2D or 3D representation. Mapping view 1102 shows instrument indicators 1106 representing instruments within the anatomical structure. Mapping view 1102 also shows target indicators 1108 representing targets within the anatomical structure. In this example, anatomical mapping 1104 and / or targets are based on preoperative data and registered / aligned to an intraoperative coordinate system / space.

[0159] In this example, interface 1100 also includes image view 1110, which includes image data captured during the procedure (or, in some cases, preoperatively). For example, image view 1110 may display CT image data captured by a CT imaging system; however, other forms of image data may be provided. In this example, the image data depicts instrument 1112 and target 1114, which are outlined for illustrative purposes. Image view 1110 may also provide orientation indicator 1116, which indicates the direction in which instrument 1112 is pointing. Orientation indicator 1116 may be oriented based on the orientation / position of the instrument determined by any of the techniques discussed herein.

[0160] In the example, interface 1100 also includes an instrument view / segment 1118 that provides image data captured from the distal end of an instrument within an anatomical structure. Although for illustrative purposes, interface 1100 includes three non-limiting views 1102, 1110, and 1118, any of the elements of views 1102, 1110, and 1118 may be eliminated and / or otherwise formatted / combined.

[0161] like Figure 11-1 As shown, the mapping view 1102 of interface 1100 initially includes an instrument indicator 1106 positioned within the anatomical mapping 1104 (e.g., within a first anatomical cavity) in a first manner / location. Here, instrument indicator 1106 indicates that the instrument is positioned in a line with target indicator 1108. Figure 11-2 The interface 1100 is shown after navigation updates, such as those achieved through the use of one or more techniques discussed herein. Figure 11-2 In the diagram, mapping view 1102 shows instrument indicators 1106 positioned in different ways / locations within anatomical mapping 1104 (such as a second anatomical cavity). Figure 11-2 The updated interface 1100 now indicates the more accurate position of the instrument relative to the target and / or anatomical mapping 1104.

[0162] In this example, interface 1100 is updated by maintaining most of the elements of mapping view 1102 in relatively fixed positions. Specifically, anatomical mapping 1104 and target indicator 1108 are maintained / displayed at fixed positions within interface 1100 / mapping view 1102. Simultaneously, instrument indicator 1106 is moved / displaced / updated to reflect the updated navigation information. In this example, when instrument indicator 1106 is displaced, the axis / elongated member of instrument indicator 1106 can maintain the same form. That is, the axis of instrument indicator 1106 can maintain the same shape / curve (e.g., including...) during the update. Figure 11-1 and Figure 11-2(The same shape / form as in the previous example). In other examples, the axis of the instrument indicator 1106 is updated, such as to show a slightly different curve / shape.

[0163] although Figure 11-1 and Figure 11-2 In one example, the instrument indicator 1106 is updated while maintaining other elements in the calibration view 1102; however, in other examples, the instrument indicator 1106 is maintained in a fixed position and other elements are updated. For example, in interface 1100 from... Figures 11-1 to 11-2 During the transition, the instrument indicator 1106 can remain in a fixed position within the mapping view 1102, while the anatomical mapping map 1104 and / or the target indicator 1108 can be displaced / moved.

[0164] In some examples, instrument view 1118 includes an indicator 1120 that indicates the orientation of the target relative to instrument view 1118, such as... Figure 11-2 As shown. Indicator 1120 can indicate the position of the target relative to the distal end of the instrument, which can be based on the coordinate system of the distal end of the instrument. Here, indicator 1120 indicates that the target is not aligned with the instrument axis, as initially based on Figure 11-1 This is understood from the target view 1102. Specifically, indicator 1120 shows the target relative to... Figure 11-2 View 1118 is upward.

[0165] Figure 12 Another block diagram of an example controller 1200 for a medical system according to some specific implementations is shown. In some specific implementations, controller 1200 may be... Figure 2 An example of control circuit 208 and / or control circuit 224. More specifically, controller 1200 is configured to guide the navigation of instruments within anatomical structures.

[0166] The controller 1200 includes a communication interface 1210, a processing system 1220, and a memory 1230. The communication interface 1210 is configured to communicate with one or more components of the medical system. More specifically, the communication interface 1210 includes features for communicating with one or more image sources (such as...) Figure 3 The image source interface (I / F) 1212 communicates with the CT imaging system 310 and / or the fluoroscopic imaging system 312. In some embodiments, the image source I / F 1212 can receive image data depicting an anatomical structure in which instruments are disposed, wherein the image data is captured by an imaging system positioned outside the anatomical structure.

[0167] The memory 1230 may include a non-transitory computer-readable medium (including one or more non-volatile memory elements, such as EPROM, EEPROM, flash memory, or hard disk drive, and other examples), which may store the following software (SW) modules: an interface generation SW module 1232 for generating a graphical interface depicting a first spatial relationship between an instrument and a target within an anatomical structure; a spatial relationship (SR) determination SW module 1234 for determining a second spatial relationship between the instrument and the target within an anatomical structure, at least in part based on image data; and an interface update SW module 1236 for updating the graphical interface to depict the second spatial relationship between the instrument and the target.

[0168] Processing system 1220 may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in controller 1200 (such as in memory 1230). For example, processing system 1220 may execute interface generation (SW) module 1232 to generate a graphical interface depicting a first spatial relationship between the instrument and a target within an anatomical structure. Processing system 1220 may also execute spatial relationship (SR) determination (SW) module 1234 to determine a second spatial relationship between the instrument and the target within an anatomical structure, at least in part, based on image data. Processing system 1220 may further execute interface update (SW) module 1236 to update the graphical interface to depict the second spatial relationship between the instrument and the target.

[0169] Figure 13 An exemplary flowchart depicting example operation 1300 for providing navigation guidance of instruments within anatomical structures is shown according to some specific embodiments. In some specific embodiments, example operation 1300 may be controlled by a controller for a medical system (such as... Figure 12 The controller 1200) executes.

[0170] The controller generates a graphical interface (1302) depicting a first spatial relationship between the instrument and the target within the anatomical structure. The controller also receives first image data depicting the anatomical structure in which the instrument is disposed, wherein the first image data is captured by a first imaging system positioned outside the anatomical structure (1304). The controller determines a second spatial relationship between the instrument and the target within the anatomical structure based at least in part on the first image data (1306). The controller further updates the graphical interface to depict the second spatial relationship between the instrument and the target (1308).

[0171] In some embodiments, determining the second spatial relationship may include: displaying one or more images of the anatomical structure based on the first image data; and receiving user input indicating at least one of the positions of a target or an instrument in the one or more images. In some other embodiments, determining the second spatial relationship may include: estimating at least one of the positions of a target or an instrument based on the first image data.

[0172] In some aspects, the generation of the graphical interface may include: receiving second image data depicting the location of a target within an anatomical structure; receiving sensor data from a sensor associated with an instrument, wherein the sensor data indicates the position of the instrument within the anatomical structure; and determining a first spatial relationship based on the second image data and the sensor data. In some embodiments, the second image data may be captured by a second imaging system when the instrument is not positioned within the anatomical structure.

[0173] In some aspects, determining the second spatial relationship may include: displaying an instrument indicator superimposed on one or more images based on first image data, the one or more images depicting an anatomical structure in which the instrument is disposed; and receiving user input for aligning the instrument indicator with the distal end of the instrument in the one or more images. In some embodiments, the controller may further estimate the posture of the instrument based on the first image data, wherein the instrument indicator is superimposed on the one or more images based on the estimated posture of the instrument. In some embodiments, the user input may include at least one of the following: adjustment of the position or orientation of the instrument indicator relative to the one or more images, or instruction for aligning the instrument indicator with the distal end of the instrument.

[0174] Additional detailed implementation descriptions and terminology Depending on the example, certain actions, events, or functions in any of the processes or algorithms described herein may be performed in a different order, or may be added, combined, or ignored entirely. Therefore, in some examples, not all described actions or events are necessary for the practice of the process.

[0175] Unless otherwise specifically stated or otherwise understood in the context in which they are used, the conditional language used herein (such as “may,” “can,” “possibly,” “can,” “for example,” etc.) means in its ordinary sense and is generally intended to convey that certain examples include certain features, elements, and / or steps that are not included in other examples. Therefore, such conditional language is not generally intended to imply that a feature, element, and / or step is necessary in any way for one or more examples, or to imply that one or more examples, with or without author input or prompting, must include logic for determining whether such features, elements, and / or steps are included in any particular example or whether they will be performed in any particular example. The terms “including,” “contains,” “have,” etc., are synonymous and used in their ordinary sense, and are used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in its inclusive sense (rather than in its exclusive sense) such that when used, for example, to connect a series of elements, the term “or” refers to one, some, or all of the elements in that series. Unless otherwise specified, conjunctions such as “at least one of X, Y, and Z” are understood in the context of general use to convey that an item, term, element, etc., can be X, Y, or Z. Therefore, such conjunctions are generally not intended to imply that certain examples require the existence of at least one X, at least one Y, and at least one Z.

[0176] In the foregoing description, to simplify this disclosure and aid in understanding one or more of the various aspects, features are sometimes grouped together in a single example, drawing, or description thereof. However, this approach of the disclosure should not be construed as reflecting an intention that any claim requires more features than those expressly recited in that claim. Furthermore, any component, feature, or step shown and / or described in the specific examples herein may be applied to or used in conjunction with any other example. Moreover, for each example, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of the subject matter disclosed herein and claimed below is not limited to the specific examples described above, but should be determined solely by a fair reading of the following claims.

[0177] In the examples, certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference, and these ordinal terms do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms used to modify elements such as structures, components, operations, etc. (e.g., "first," "second," "third," etc.) do not necessarily indicate the priority or order of that element relative to any other element, but rather serve to generally distinguish that element from another element with a similar or identical name (but used in ordinal terms). Additionally, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." Furthermore, an operation performed "based on" a certain condition or event may also be performed based on one or more other conditions or events not explicitly stated.

[0178] Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which the exemplary examples pertain. It will be further understood that terms, such as those defined in common dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant field and not as idealized or overly formal, unless explicitly defined herein. As used herein, the term "patient" can generally refer to a human being, an anatomical model, a simulator, a cadaver, and other living or non-living objects.

[0179] For ease of description, the spatial relative terms “external,” “internal,” “upper,” “lower,” “below,” “above,” “vertical,” “horizontal,” and similar terms are used herein to describe the relationship between one element or component shown in the accompanying drawings and another. It should be understood that the spatial relative terms are intended to cover different orientations of the device in use or operation other than those depicted in the accompanying drawings. For example, if the device shown in the accompanying drawings is flipped, a device located “below” or “under” another device may be positioned “above” another device. Therefore, the illustrative term “below” can include both a lower position and an upper position. The device may also be oriented in another direction, and therefore the spatial relative terms may be interpreted differently depending on the orientation.

[0180] Unless otherwise explicitly stated, comparative and / or quantitative terms, such as “less,” “more,” “greater,” etc., are intended to encompass the concept of equality. For example, “less” may mean not only “less” in the strictest mathematical sense, but may also mean “less than or equal to.”

Claims

1. A medical system comprising: First imaging system; and Control circuit, the control circuit being configured to: Generate a graphical interface that depicts the primary spatial relationship between the instrument and the target within the anatomical structure; Receive first image data depicting the anatomical structure in which the instrument is disposed, the first image data being captured by the first imaging system when positioned outside the anatomical structure; The second spatial relationship between the instrument and the target within the anatomical structure is determined, at least in part, based on the first image data; as well as The graphical interface is updated to depict the second spatial relationship between the instrument and the target.

2. The medical system of claim 1, wherein, The first imaging system is a cone-beam computed tomography (CBCT) imaging system.

3. The medical system of claim 1, wherein, The determination of the second spatial relationship includes: Displaying one or more images of the anatomical structure based on the first image data; and Receive user input indicating at least one of the positions of the target or the instrument in the one or more images.

4. The medical system according to claim 1, wherein, The determination of the second spatial relationship includes: The location of the target or the location of the instrument is estimated based on the first image data.

5. The medical system according to claim 1, wherein, The generation of the graphical interface includes: Receive second image data depicting the location of the target within the anatomical structure; Receive sensor data from sensors associated with the instrument, the sensor data indicating the position of the instrument within the anatomical structure; and The first spatial relationship is determined based on the second image data and the sensor data.

6. The medical system according to claim 5, wherein, The second image data is captured by the second imaging system when the instrument is not placed within the anatomical structure.

7. The medical system according to claim 1, wherein, The determination of the second spatial relationship includes: Based on the first image data, an instrument indicator superimposed on one or more images is displayed, the one or more images depicting the anatomical structure in which the instrument is disposed; and Receive user input for aligning the instrument indicator with the distal end of the instrument in one or more images.

8. The medical system according to claim 7, wherein, The control circuit is further configured as follows: The posture of the device is estimated based on the first image data, and the device indicator is superimposed on the one or more images based on the estimated posture of the device.

9. The medical system according to claim 7, wherein, The user input includes adjusting the position or orientation of the instrument indicator relative to the one or more images.

10. The medical system according to claim 7, wherein, The user input includes instructions to align the instrument indicator with the distal end of the instrument.

11. The medical system according to claim 1, wherein, The first spatial relationship is represented by a depiction of the instrument in a first posture relative to the target, and the second spatial relationship is represented by a depiction of the instrument in a second posture relative to the target.

12. A method executed by a controller for a medical system, comprising: Generate a graphical interface that depicts the primary spatial relationship between the instrument and the target within the anatomical structure; Receive first image data depicting the anatomical structure in which the instrument is disposed, the first image data being captured by a first imaging system positioned outside the anatomical structure; The second spatial relationship between the instrument and the target within the anatomical structure is determined, at least in part, based on the first image data; as well as The graphical interface is updated to depict the second spatial relationship between the instrument and the target.

13. The method of claim 12, wherein, The determination of the second spatial relationship includes: Displaying one or more images of the anatomical structure based on the first image data; and Receive user input indicating at least one of the positions of the target or the instrument in the one or more images.

14. The method according to claim 12, wherein, The determination of the second spatial relationship includes: The location of the target or the location of the instrument is estimated based on the first image data.

15. The method according to claim 12, wherein, The generation of the graphical interface includes: Receive second image data depicting the location of the target within the anatomical structure; Receive sensor data from sensors associated with the instrument, the sensor data indicating the position of the instrument within the anatomical structure; and The first spatial relationship is determined based on the second image data and the sensor data.

16. The method according to claim 15, wherein, The second image data is captured by the second imaging system when the instrument is not placed within the anatomical structure.

17. The method of claim 12, wherein, The determination of the second spatial relationship includes: Based on the first image data, an instrument indicator superimposed on one or more images is displayed, the one or more images depicting the anatomical structure in which the instrument is disposed; and Receive user input for aligning the instrument indicator with the distal end of the instrument in one or more images.

18. The method of claim 17, further comprising: The posture of the device is estimated based on the first image data, and the device indicator is superimposed on the one or more images based on the estimated posture of the device.

19. The method of claim 17, wherein, The user input includes at least one of the following: adjustment of the position or orientation of the instrument indicator relative to the one or more images, or indication of alignment of the instrument indicator with the distal end of the instrument.

20. A controller for a medical system, comprising: Processing system; and The memory stores instructions that, when executed by the processing system, cause the controller to: Generate a graphical interface that depicts the primary spatial relationship between the instrument and the target within the anatomical structure; Receive image data depicting the anatomical structure in which the instrument is disposed, the image data being captured by an imaging system positioned outside the anatomical structure; The second spatial relationship between the instrument and the target within the anatomical structure is determined, at least in part, based on the image data; as well as The graphical interface is updated to depict the second spatial relationship between the instrument and the target.

Images