Navigating the tubular network

Abstract

To provide methods and apparatuses that provide improved navigation through tubular networks such as lung airways by providing improved estimation of position and orientation information of a medical instrument (e.g., an endoscope) within the tubular network.SOLUTION: Various input data such as image data, EM data, and robot data are used by different algorithms to estimate the state of the medical instrument, and the state information is used to locate a specific site within a tubular network and / or to determine navigation information for what positions / orientations the medical instrument should travel through to arrive at the specific site. Probability distributions together with confidence values corresponding to the different algorithms are generated and used to determine the estimated state of medical instrument.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 220,770, entitled “NAVIGATION OF TUBULAR NETWORKS,” filed Sep. 18, 2015, which is incorporated by reference in its entirety.

[0002] This description relates generally to surgical robotics, and more particularly to the navigation of medical instruments within a patient's body network. [Background technology]

[0003] Bronchoscopy is a medical procedure that allows a physician to examine the interior of a patient's lung airways, such as the bronchi and bronchioles. The lung airways carry air from the trachea or windpipe to the lungs. During the medical procedure, a thin, flexible, tubular instrument, also known as a bronchoscope, may be inserted into the patient's mouth and passed down the patient's throat to the patient's lung airways, and the patient is generally anesthetized to relax the patient's throat and lungs in preparation for the surgical examination and manipulation during the medical procedure.

[0004] Conventional bronchoscopes usually include a light source and a small camera that allows the physician to inspect the patient's trachea and airways, and rigid tubes may be used in conjunction with bronchoscopes for surgical purposes, for example, when the patient has a large amount of bleeding in the lungs or when a large object is blocking the patient's throat. When rigid tubes are used, the patient is often anesthetized. Concurrent with the advent of other advanced medical devices, the use of robotic bronchoscopes is becoming more and more of a reality. Robotic bronchoscopes offer great advantages in navigation through the tubular network. Robotic bronchoscopes are easy to use and allow convenient treatment and biopsy, even during the bronchoscopy phase.

[0005] Apart from medical devices or platforms, such as the robotic bronchoscope described above, various methods and software models can be used to aid in surgery. For example, a computed tomography (CT) scan of a patient's lungs is often performed during preoperative examination of a surgical procedure. Data from the CT scan can be used to generate a three-dimensional (3D) model of the airways of the patient's lungs, allowing the physician to access a visual reference that may be useful during the surgical procedure of the surgical procedure. Summary of the Invention [Problem to be solved by the invention]

[0006] However, conventional techniques for navigating tubular networks still have challenges, even when utilizing medical devices (e.g., robotic bronchoscopes) and using existing methods (e.g., performing CT scans and generating 3D models). As an example, motion estimation of a medical device (e.g., a bronchoscope instrument) within a patient's body may not be accurate based on changes in device position and orientation, and may not accurately or correctly identify the device's location within the patient's body in real time. Such inaccurate instrument location information may provide misleading information to physicians who use 3D models as visual references during medical surgical procedures.

[0007] Therefore, improved techniques for navigating through tubular networks are needed. [Means for solving the problem]

[0008] The methods and devices disclosed herein provide improved navigation through tubular networks, such as pulmonary airways, by providing improved estimation of position and orientation information of medical instruments, such as flexible or rigid elongated medical instruments (e.g., endoscopes) within the tubular network.

[0009] As an example, the device is a robotic endoscopic instrument that obtains "raw" position and orientation information (collectively, input data) of a desired anatomical site or endoscopic instrument within the tubular network. The endoscopic instrument includes a flexible tip and an instrument device manipulator (IDM) coupled to the endoscopic instrument. Devices such as electromagnetic sensors (EM sensors), imaging devices (e.g., optical sensors), and a robotic control system that controls the medical instrument are coupled to the instrument tip to collect input data as the endoscopic instrument enters and navigates within the tubular network. The IDM is used to control the movement and position of different robotic components (e.g., endoscopic instruments) of the surgical robotic system. A processor is coupled to the endoscopic instrument to receive the input data and determine moment-to-moment movement, position, and orientation information of the medical instrument (e.g., instrument tip) within the tubular network.

[0010] The processor is instructed by the navigation configuration system to use the input data to estimate a state of the medical instrument, which may include information such as position, orientation, relative and absolute depth, branch selection, etc. The processor may be further instructed to use the estimated state to locate a particular site within the tubular network and / or determine navigation information about the position / orientation the medical instrument should take to reach a particular site, which may be referred to as output data or navigation data.

[0011] The navigation configuration system further includes a plurality of algorithm modules that utilize various navigation algorithms to provide estimated state and navigation data. Examples of algorithms used include EM-based algorithms, image-based algorithms, and robot-based algorithms. The estimated state and navigation data generated after utilizing the various algorithms utilize any one or more of EM-based input data, image-based input data, and robot-based input data.

[0012] In some embodiments, a probability distribution with a confidence value is generated by the algorithm module, which is used to identify the estimated state of the medical instrument. The "probability" in "probability distribution" as used herein refers to the probability that the estimation or identification of the position and / or orientation of the medical instrument is correct. For example, different probabilities may be calculated that indicate the relative probability that the medical instrument is in one of several different possible airways in the lung. In contrast, a "confidence value" as used herein reflects a measure of confidence in the state estimation provided by one of the algorithms. For example, relatively close to the airway opening, a particular algorithm may have a high degree of confidence in estimating the position and orientation of the medical instrument, but as the medical instrument progresses further into the bottom of the lung, its confidence value may decrease. In general, the confidence value is based on one or more "external" factors related to the process in which the outcome is identified, while the probability is a relative measure that arises when attempting to identify possible outcomes from one algorithm. The algorithms, probabilities, and confidence values ​​may be variously combined to arrive at the estimated state and navigation data.

[0013] In one embodiment, before performing an actual surgical procedure on a patient, a series of pre-operative steps utilizing improved navigation of a surgical instrument (e.g., an endoscope) within the patient's tubular network may be performed. First, a CT scan of the tubular network is acquired to generate a 3D model of the tubular network. A target area (e.g., a lesion to be biopsied) within the tubular network is selected, and a corresponding path for the surgical instrument to travel within the tubular network to reach the target area is automatically planned and displayed to a user (e.g., a surgeon). After the path is determined, a virtual endoscope may be applied to travel through the tubular network and reach the target area. In an actual surgical procedure, the CT scan, the generated 3D model, and other input data (e.g., image data, EM data, robot data collected over the duration of the procedure) are combined and repeatedly analyzed during the procedure via a surgical configuration system to provide an estimate of real-time movement and position / orientation information of the surgical instrument (e.g., an endoscope) within the tubular network together with navigation information, thereby enabling the surgeon to perform the procedure more expediently. [Brief description of the drawings]

[0014] [Figure 1A] 1 illustrates an example surgical robotic system according to one embodiment. [Figure 1B] 1A-1C show various perspective views of a robotic platform coupled to the surgical robotic system shown in FIG. [Figure 1C] 1A-1C show various perspective views of a robotic platform coupled to the surgical robotic system shown in FIG. [Figure 1D] 1A-1C show various perspective views of a robotic platform coupled to the surgical robotic system shown in FIG. [Figure 1E] 1A-1C show various perspective views of a robotic platform coupled to the surgical robotic system shown in FIG. [Figure 1F] 1A-1C show various perspective views of a robotic platform coupled to the surgical robotic system shown in FIG. [Diagram 2] 1 illustrates an example command console for an example surgical robotic system in accordance with one embodiment. [Figure 3A] FIG. 1B illustrates an isometric view of an example independent drive mechanism of the instrument device manipulator (IDM) shown in FIG. 1A, according to one embodiment. [Figure 3B] FIG. 3B shows a conceptual diagram illustrating how forces can be measured by strain gauges of the independent drive mechanism shown in FIG. 3A according to one embodiment. [Figure 4A] 1 illustrates a top view of an example endoscope according to one embodiment. [Figure 4B] 4B illustrates an example endoscope tip of the endoscope shown in FIG. 4A, according to one embodiment. [Diagram 5] 1 illustrates a schematic setup of an example of an EM tracking system included in a surgical robotic system according to one embodiment. [Figure 6A] 1 illustrates an example anatomical lumen and a 3D model of an example of an example anatomical lumen, according to one embodiment. [Figure 6B]1 illustrates an example anatomical lumen and a 3D model of an example of an example anatomical lumen, according to one embodiment. [Figure 7] 1 illustrates a computer-generated 3D model representing an anatomical space, according to one embodiment. [Figure 8A] 13A-13C show example graphs illustrating on-the-fly alignment of an EM system to a 3D model of a path through a tubular network, according to one embodiment. [Figure 8B] 13A-13C show example graphs illustrating on-the-fly alignment of an EM system to a 3D model of a path through a tubular network, according to one embodiment. [Figure 8C] 13A-13C show example graphs illustrating on-the-fly alignment of an EM system to a 3D model of a path through a tubular network, according to one embodiment. [Figure 8D] 13A-13C show example graphs illustrating on-the-fly alignment of an EM system to a 3D model of a path through a tubular network, according to one embodiment. [Figure 8E] 13 illustrates the effect of aligning an example EM system to a 3D model of a branching tubular network, according to one embodiment. [Figure 8F] 13 illustrates the effect of aligning an example EM system to a 3D model of a branching tubular network, according to one embodiment. [Figure 9A] 1 illustrates a high-level overview of an example block diagram of a navigation configuration system according to one embodiment. [Figure 9B] 9B illustrates a block diagram of an example of the navigation system shown in FIG. 9A, according to one embodiment. [Figure 9C] FIG. 2 illustrates a block diagram of an example of an estimated state data storage device included in a state estimator, according to one embodiment. [Figure 10A] FIG. 1 illustrates a block diagram of an example of an EM alignment module in accordance with one embodiment. [Figure 10B] FIG. 2 illustrates a block diagram of an example branch selection module according to one embodiment. [Figure 10C] 4 illustrates a block diagram of an example of a motion estimation module according to one embodiment. [Figure 10D] 1 illustrates a block diagram of an example of an object detection module according to an example. [Figure 10E] FIG. 2 illustrates a block diagram of an example of an object mapping module according to one embodiment. [Figure 10F] FIG. 2 illustrates a block diagram of an example of a topological reasoning module according to one embodiment. [Figure 11A] 1 shows two example identified objects superimposed on an image of a bronchial network, along with a link connecting the centers of the two objects, according to one embodiment. [Figure 11B] 13 illustrates the correspondence of airway lumens in a real image of a real bronchial network with the airway lumens in a 3D model of the network, according to an embodiment. [Figure 12] 4 illustrates an example process for generating a probability distribution over multiple values ​​of a state estimated by a state estimator, according to one embodiment. [Figure 13] 1 shows an example of how the navigation configuration system can correct errors using navigation through a network of bronchial tubes, according to one embodiment. [Figure 14A] 1 illustrates a sequence of pre-operative steps of an example of a method for navigating an instrument tip to a specific site within a supraglottic network, according to one embodiment. [Figure 14B] 1 illustrates an example user interface for navigating a surgical instrument through a tubular network, according to one embodiment. [Figure 14C] 1 illustrates an example user interface for aligning and controlling instruments during an endoscopic procedure, according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Reference will now be made in detail to several embodiments, examples of which are illustrated in the accompanying drawings. It is to be noted that, wherever practicable, like or similar reference numerals may be used in the drawings and may indicate like or similar functionality. The drawings depict embodiments of the described systems (or methods) for purposes of illustration only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be used without departing from the principles of the invention as described herein.

[0016] I. Surgical Robot System FIG. 1A illustrates an example of a surgical robotic system 100 according to one embodiment. The surgical robotic system 100 includes a base 101 coupled to one or more robotic arms, e.g., robotic arm 102. The base 101 is communicatively coupled to a command console, which will be further described with reference to FIG. 2 in “II. Command Console”. The base 101 may be positioned to allow the robotic arm 102 to access a patient to perform a surgical procedure, while a user, e.g., a physician, may comfortably control the surgical robotic system 100 from the command console. In some embodiments, the base 101 may be coupled to a surgical table or bed for supporting a patient. Although not shown in FIG. 1 for clarity of illustration, the base 101 may include subsystems, e.g., control electronics, pneumatic mechanisms, power sources, light sources, etc. The robotic arm 102 includes multiple arm segments 110 coupled at joints 111, thereby providing the robotic arm 102 with multiple degrees of freedom, e.g., seven degrees of freedom corresponding to the seven arm segments. The base 101 may include a power source 112, pneumatics 113, and control and sensor electronics 114 - including components such as a central processing unit, a data bus, control circuits, and memory - and associated actuators such as motors for moving the robotic arm 102. The electronics 114 in the base 101 may also process and transmit control signals communicated from a command console.

[0017] In some embodiments, the base 101 includes wheels 115 for transporting the surgical robotic system 100. The mobility of the surgical robotic system 100 helps accommodate space constraints in an operating room and facilitates proper positioning and movement of surgical equipment. Additionally, this mobility allows the robotic arm 102 to be configured so that it does not interfere with the patient, physician, anesthesiologist, or any other equipment. During a procedure, a user may control the robotic arm 102 using a control device, such as a command console.

[0018] In some embodiments, the robotic arm 102 includes a set-up joint that uses a combination of a brake and a counterweight to maintain the position of the robotic arm 102. The counterweight may include a gas spring or a coil spring. The brake, e.g., a fail-safe brake, may include mechanical and / or electrical components. Additionally, the robotic arm 102 may be a gravity-assisted passive support type robotic arm.

[0019] Each robotic arm 102 can be coupled to an instrument device manipulator (IDM) 117 using a mechanism changer interface (MCI) 116. The IDM 117 can be removed and replaced with a different type of IDM, for example, a first type of IDM operates an endoscope while a second type of IDM operates a laparoscope. The MCI 116 includes connectors to transfer pneumatic pressure, electrical power, electrical signals, and optical signals from the robotic arm 102 to the IDM 117. The MCI 116 can be a set screw or a base plate connector. The IDM 117 operates a surgical instrument such as an endoscope 118 using technologies including direct drive, harmonic drive, gear drive, belt pulley, magnetic drive, and the like. The MCI 116 can be interchangeable based on the type of IDM 117 and customized for certain surgical procedures. The robotic arm 102 can include a wrist and joint level torque sensing at a distal end, such as a KUKA AG® LBR5 robotic arm.

[0046] The endoscope 118 is a tubular, flexible surgical instrument that is inserted into a patient's anatomy to take images of the anatomy (e.g., body tissue). In particular, the endoscope 118 includes one or more imaging devices (e.g., cameras or sensors) that take images. The imaging devices may include one or more optical components, such as optical fibers, fiber arrays, or lenses. The optical components move with the tip of the endoscope 118 such that movement of the tip of the endoscope 118 changes the images taken by the imaging devices. The endoscope 118 is further described in "IV. Endoscope" with reference to Figures 3A-4B.

[0020] The robotic arm 102 of the surgical robotic system 100 manipulates the endoscope 118 using elongated movement members. The elongated movement members may include pull wires, also referred to as pull or push wires, cables, fibers, or flexible shafts. For example, the robotic arm 102 actuates a number of pull wires coupled to the endoscope 118 to deflect the tip of the endoscope 118. The pull wires may include both metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, and the like. The endoscope 118 may exhibit nonlinear behavior in response to forces applied by the elongated movement members. The nonlinear behavior may be based on the stiffness and compressibility of the endoscope 118 and the variability in sag or stiffness between different elongated movement members.

[0021] 1B-1F show various perspective views of the surgical robot system 100 connected to a robotic platform 150 (or a surgical bed) according to various embodiments. Specifically, FIG. 1B is a side view of the surgical robot system 100, showing the robotic arm 102 manipulating the endoscope 118 to insert the endoscope into the patient's body, with the patient lying on the robotic platform 150. FIG. 1C is a top view of the surgical robot system 100 and the robotic platform 150, showing the endoscope 118 manipulated by the robotic arm being inserted into the patient's body. FIG. 1D is a perspective view of the surgical robot system 100 and the robotic platform 150, showing the endoscope 118 being controlled so that the endoscope 118 is parallel to the robotic platform in the horizontal direction. FIG. 1E is another perspective view of the surgical robot system 100 and the robotic platform 150, showing the endoscope 118 being controlled so that the endoscope 118 is approximately perpendicular to the robotic platform. More specifically, in Figure 1E, the angle between the horizontal plane of the robotic platform 150 and the endoscope 118 is 75°. Figure 1F is a perspective view of the surgical robotic system 100 and robotic platform 150 shown in Figure 1E, more specifically, the angle between the endoscope 118 and an imaginary line 160 connecting one end of the endoscope 180 to the robotic arm 102 at a position further away from the robotic platform is 90°.

[0022] II. Command Console 2 illustrates an example of a command console 200 for the surgical robotic system 100, according to one embodiment. The command console 200 includes a console base 201, a display module 202, e.g., a monitor, and a control module, e.g., a keyboard 203 and a joystick 204. In some embodiments, one or more of the functions of the command console 200 may be incorporated into the base 101 of the surgical robotic system 100 or into another system communicatively coupled to the surgical robotic system 100. A user 205, e.g., a physician, uses the command console 200 to remotely control the surgical robotic system 100 from an ergonomic position.

[0023] The console base 201 may include a central processing unit, memory devices, data buses, and associated data communication ports that decode and process signals, such as camera image data and tracking sensor data from the endoscope 118 shown in FIG. 1. In some embodiments, both the console base 201 and the base 101 perform signal processing for undistributed operation. The console base 201 may also process commands and instructions provided by a user 205 through the control modules 203 and 204. In addition to the keyboard 203 and joystick 204 shown in FIG. 2, the control modules may include other devices, such as a computer mouse, a trackpad, a trackball, a control pad, a video game controller, and sensors (e.g., motion sensors or cameras) that capture hand and finger gestures.

[0024] A user 205 may control a surgical instrument, such as an endoscope 118, using the command console 200 in a velocity mode or a position control mode. In a velocity mode, the user 205 directly controls the pitch and yaw motion of the distal end of the endoscope 118 based on direct manual control using a control module. For example, the movement of the joystick 204 may be mapped to yaw and pitch motion at the distal end of the endoscope 118. The joystick 204 may provide haptic feedback to the user 205. For example, the joystick 204 may vibrate to indicate that the endoscope 118 cannot translate or rotate further in a certain direction. The command console 200 may also provide visual feedback (e.g., a pop-up message) and / or audio feedback (e.g., a beep) to indicate that the endoscope 118 has reached maximum translation or rotation.

[0025] In the position control mode, the command console 200 uses a three-dimensional (3D) map of the patient and a predetermined computer model of the patient to control a surgical instrument, such as an endoscope 118. The command console 200 provides control signals to the robotic arm 102 of the surgical robotic system 100 to steer the endoscope 118 to a target location. Due to the reliance on the 3D map, the position control mode requires precise mapping of the patient's anatomy.

[0026] In some embodiments, a user 205 may manually operate the robotic arm 102 of the surgical robotic system 100 without using the command console 200. When set up in a surgical room, the user 205 may move the robotic arm 102, endoscope 118, and other surgical instruments to access a patient. The surgical robotic system 100 may determine the appropriate configuration of the robotic arm 102 and instruments in response to force feedback and inertial control from the user 205.

[0027] The display module 202 may include an electronic monitor, a virtual reality viewing device, such as goggles or glasses, and / or other means of display device. In some embodiments, the display module 202 is integrated with the control module, such as a tablet-type device with a touch screen. Additionally, a user 205 may use the integrated display module 202 and control module to both view data and enter commands into the surgical robotic system 100.

[0028] The display module 202 may display 3D images using a stereoscopic device, such as a visor or goggles. The 3D images provide an "endo view" (i.e., an endoscopic view), which is a computerized 3D model of the patient's anatomy. The "endo view" provides a virtual environment inside the patient's body and a predicted position of the endoscope 118 within the patient's body. The user 205 compares the "endo view" model with an actual image taken by a camera to help mentally orient and confirm that the endoscope 118 is in a correct-or reasonably correct-position within the patient's body. The "endo view" provides information about the shape of the anatomy, such as the patient's intestine or colon around the distal end of the endoscope 118. The display module 202 may simultaneously display a 3D model and a computed tomography (CT) scan of the anatomy around the distal end of the endoscope 118. Additionally, the display module 202 may overlay a predetermined optimal navigation path of the endoscope 118 onto the 3D model and the CT scan.

[0029] In some embodiments, a model of the endoscope 118 is displayed along with the 3D model to help indicate the status of the surgical procedure. For example, a CT scan identifies lesions in the anatomy that may require a biopsy. During surgery, the display module 202 may show reference images taken by the endoscope 118 that correspond to the current location of the endoscope 118. The display module 202 may automatically display different views of the model of the endoscope 118 depending on user settings and the particular surgical procedure. For example, the display module 202 shows an overhead fluoroscopic view of the endoscope 118 during navigation steps as the endoscope 118 approaches the surgical area of ​​the patient.

[0030] III. Instrumentation Manipulator 3A illustrates an isometric view of one example of an independent drive mechanism of the IDM 117 shown in FIG. 1, according to one embodiment. The independent drive mechanism can tighten or loosen (e.g., independently of one another) the pull wires 321, 322, 323, and 324 of the endoscope, respectively, by rotating the output shafts 305, 306, and 308 of the IDM 117. Just as the output shafts 305, 306, and 308 transmit forces to the pull wires 321, 322, 323, and 324, respectively, through angular motion, the pull wires 321, 322, 323, and 324 also transmit forces to the output shafts. The IDM 117 and / or surgical robotic system 100 can measure the transmitted forces using sensors, such as strain gauges, described further below.

[0031] 3B shows a conceptual diagram illustrating how force may be measured by strain gauges 334 of the independent drive mechanism shown in FIG. 3A, according to one embodiment. Force 331 may be directed away from output shaft 305 coupled to motor mount 333 of motor 337. Thus, force 331 causes horizontal displacement of motor mount 333. Additionally, strain gauge 334 coupled horizontally to motor mount 333 experiences strain in the direction of force 331. Strain may be measured as a ratio of horizontal displacement of tip 335 of strain gauge 334 to overall horizontal width 336 of strain gauge 334.

[0032] In some embodiments, the IDM 117 includes additional sensors, such as an inclinometer or accelerometer, to determine the orientation of the IDM 117. Based on measurements from the additional sensors and / or the strain gauges 334, the surgical robotic system 100 can calibrate readings from the strain gauges 334 to account for gravitational load effects. For example, if the IDM 117 is oriented on its horizontal side, the weight of certain components of the IDM 117 may cause strain on the motor mount 333. Thus, without accounting for gravitational load effects, the strain gauges 334 may measure strain that was not due to strain on the output shaft.

[0033] IV. Endoscopy 4A illustrates a top view of an example endoscope 118 according to one embodiment. Endoscope 118 includes a leader 415 tubular component nested or partially nested within and longitudinally aligned with a sheath 441 tubular component. Sheath 411 includes a proximal sheath portion 412 and a distal sheath portion 413. Leader 415 has a smaller outer diameter than sheath 411 and includes a proximal leader portion 416 and a distal leader portion 417. Sheath base 414 and reader base 418 actuate distal sheath portion 413 and distal leader portion 417 based on control signals, for example, from a user of surgical robotic system 100. Sheath base 414 and reader base 418 are, for example, part of IDM 117 shown in FIG. 1.

[0034] Both the sheath base 414 and the reader base 418 include drive mechanisms (e.g., independent drive mechanisms further described in Section III, Instrument Device Manipulator, with reference to FIGS. 3A and 3B) to control pull wires coupled to the sheath 411 and the reader 415. For example, the sheath base 414 generates a tensile load on the pull wires coupled to the sheath 411 to deflect the distal sheath portion 413. Similarly, the reader base 418 generates a tensile load on the pull wires of the reader 415 to deflect the distal leader portion 417. Both the sheath base 414 and the reader base 418 may also include couplers for routing air pressure, power, electrical signals, or optical signals from the IDM to the sheath 411 and the reader 414, respectively. The pull wire may include a coiled steel tube along the length of the pull wire within the sheath 411 or leader 415, which transfers the axial compression to the origin of the load, such as the sheath base 414 or leader base 418, respectively.

[0035] The endoscope 118 can easily navigate the patient's anatomy due to the multiple degrees of freedom provided by the pull wires coupled to the sheath 411 and the leader 415. For example, four or more pull wires may be used in either the sheath 411 and / or the leader 415, thereby providing eight or more degrees of freedom. In other embodiments, up to three pull wires may be used, providing up to six degrees of freedom. The sheath 411 and the leader 415 may rotate up to 360 degrees along the longitudinal axis 406, thereby providing more degrees of movement. The combination of rotational angles and multiple degrees of freedom provides a user of the surgical robotic system 100 with user-friendly and intuitive control of the endoscope 118.

[0036] FIG. 4B illustrates an example endoscope tip 430 of the endoscope 118 shown in FIG. 4A, according to one embodiment. In FIG. 4B, the endoscope tip 430 includes an imaging device 431 (e.g., a camera), an illumination source 432, and an end of an EM coil 434. The illumination source 432 provides light to illuminate the interior of an anatomical space. The provided light allows the imaging device 431 to record an image of the space, which can then be transmitted to a computer system, such as the command console 200, for processing as described herein. The electromagnetic coil 434 disposed at the tip 430 may be used in conjunction with an electromagnetic tracking system to detect the position and orientation of the endoscope tip 430 while it is disposed within the anatomical system. In some embodiments, the coils may be tilted to provide sensitivity to electromagnetic fields along different axes, providing the ability to measure a total of six degrees of freedom: three position and three angle. In other embodiments, only one coil may be placed in the endoscope tip 430 with its axis along the endoscope shaft of the endoscope 118, and due to the rotational symmetry of such a system, there is no sensitivity to rotation about the axis, and thus, in such a case, only five degrees of freedom are detectable. The endoscope tip 430 further includes a working channel 436 through which a surgical instrument, such as a biopsy needle, may be inserted along the endoscope shaft to access an area near the endoscope tip.

[0037] V. Alignment transformation of EM system to 3D model Schematic setup of the VAEM tracking system FIG. 5 shows a schematic setup of an example of an EM tracking system 505 included in a surgical robotic system 500 according to one embodiment. In FIG. 5, multiple robotic components (e.g., a window field generator, reference sensor as described below) are included in the EM tracking system 505. The surgical robotic system 500 includes a surgical bed 511 that restrains the patient's body. Under the bed 511 is a window field generator (WFG) 512 configured to sequentially activate a set of EM coils (e.g., EM coil 434 shown in FIG. 4B). The WFG 512 generates an alternating current (AC) magnetic field over a large volume, for example, in some cases, it may generate an AC field in a volume of about 0.5 m×0.5 m×0.5 m.

[0038] Additional magnetic fields may be applied by additional magnetic field generators to aid in tracking of the instrument within the body. For example, a planar magnetic field generator (PFG) may be attached to the system arm adjacent to the patient to provide a tilted EM field. A reference sensor 513 may be placed on the patient's body to provide a local EM field to further improve tracking accuracy. Each reference sensor 513 may be attached to a command module 515 by a cable 514. The cable 514 is connected to the command module 515 through an interface unit 516, which handles communication with each device and provides power. The interface unit 516 is coupled to a system control unit (SCU) 517, which acts as an overall interface controller for the various entities mentioned above. The SCU 517 also drives the magnetic field generators (e.g., WFG 512) and collects sensor data from the interface unit 516, from which it calculates the position and orientation of the sensor within the body. The SCU 517 may be coupled to a personal computer (PC) 518 to allow user access and control.

[0039] The command module 515 is also connected to various IDMs 519 that are coupled to the surgical robotic system 500 as described herein. The IDMs 519 are typically coupled to a surgical robotic system (e.g., surgical robotic system 500) and are used to control and receive data from each connected robotic component, e.g., a robotic endoscopic instrument or robotic arm. As mentioned above, by way of example, the IDM 519 is coupled to an endoscopic instrument (not shown here) of the surgical robotic system 500.

[0040] The command module 515 receives data passed from the endoscopic instruments. The type of received data depends on the corresponding type of the attached instrument. For example, the received data may include sensor data (e.g., image data, EM data), robotic data (e.g., endoscope movement data and IDM physical movement data), control data, and / or video data. To better process the video data, the field programmable gate array (FPGA) 520 may be configured to handle image processing. By comparing data obtained from various sensors, devices, and magnetic field generators, the SCU 517 may precisely track the movements of different components of the surgical robotic system 500, as well as, for example, the position and orientation of these components.

[0041] To track the sensor through the patient's anatomy, the EM tracking system 505 may require a process known as "registration," in which the system finds a geometric transformation that aligns a single object between different coordinate systems. For example, a particular anatomical part of a patient has two different representations in 3D model coordinates and EM sensor coordinates. To be able to establish consistency and a common language between these two different coordinate systems, the EM tracking system 505 needs to find a transformation, i.e., registration, that links these two representations. For example, the location of the EM tracker relative to the location of the EM field generators may be mapped into a 3D coordinate system to isolate the corresponding location in the 3D model.

[0042] VB3D model representation 6A and 6B show an example anatomical lumen 600 and an example 3D model 620 of the anatomical lumen, according to one embodiment. More specifically, FIGS. 6A and 6B show the centerline coordinates, diameter measurements, and anatomical space relationships between the actual anatomical lumen 600 and its 3D model 620. In FIG. 6A, the anatomical lumen 600 is roughly tracked in the longitudinal direction by centerline coordinates 601, 602, 603, 604, 605, and 606, where each centerline coordinate roughly approximates the center of a cross-sectional slice of the lumen. The centerline coordinates are connected and visualized by centerline 607. The lumen volume can be further visualized by measuring the lumen diameter at each centerline coordinate, for example, coordinates 608, 609, 610, 611, 612, and 613 represent measurements of lumen 600 corresponding to coordinates 601, 602, 603, 604, 605, and 606.

[0043] FIG 6B illustrates a 3D model 620 of an example of the anatomical lumen 600 shown in FIG 6A, according to one embodiment. In FIG 6B, the anatomical lumen 600 is visualized in 3D space by first locating centerline coordinates 601, 602, 603, 604, 605, and 606 in 3D space based on centerline 607. As an example, at each centerline coordinate, the lumen diameter is visualized as a 2D circular space (e.g., 2D circular space 630) having diameters 608, 609, 610, 611, 612, and 613. By connecting those 2D circular spaces to form a 3D space, the anatomical lumen 600 is approximated and visualized as 3D model 620. By increasing the resolution of the centerline coordinates and measurements, i.e., by increasing the density of the centerline coordinates and measurements for a given lumen or subsection, a more accurate approximation may be identified. The centerline coordinates may also include markers indicating points of interest to the physician, including lesions.

[0044] In some embodiments, a preoperative software package is also used to analyze the generated 3D model of the anatomical space and derive a navigation path based on the 3D model. For example, the software package may derive the shortest navigation path to a lesion (marked by centerline coordinates) or several lesions. This navigation path may be presented intraoperatively to the operator in either two or three dimensions depending on the operator's preference.

[0045] 7 illustrates a computer-generated 3D model 700 representing an anatomical space, according to one embodiment. As described above in FIGS. 6A and 6B, the 3D model 700 may be generated using a centerline 701 obtained by reviewing a preoperatively generated CT scan. In some embodiments, computer software may be able to map a navigation path 702 within the tubular network to access a surgical site 703 within the 3D model 700. In some embodiments, the surgical site 703 may be linked to individual centerline coordinates 704, allowing a computer algorithm to topologically search the centerline coordinates of the 3D model 700 for an optimal path 702 within the tubular network.

[0046] In some embodiments, the tip of the endoscopic instrument within the patient's anatomy is tracked and the tracked location of the endoscopic instrument within the patient's anatomy is mapped and located within a computer model, thereby enhancing navigation of the tubular network.Several techniques may be utilized individually or in combination to track the distal working end of the endoscopic instrument, i.e., the position and orientation of the working end.

[0047] In a sensor-based approach to determining location, a sensor, e.g., an electromagnetic (EM) tracker, may be coupled to the distal working end of the endoscopic instrument to provide a real-time indication of the endoscopic instrument's advancement. In EM-based tracking, an EM tracker built into the endoscopic instrument measures variations in an electromagnetic field generated by one or more EM transmitters. The transmitter (or magnetic field generator) may be placed near the patient (e.g., as part of a surgical bed) to generate a low-intensity magnetic field. This induces a small current in a sensor coil in the EM tracker that correlates to the distance and angle between the sensor and the generator. The electrical signal may then be digitized by an interface unit (on-chip or PCB) and transmitted via cable / wiring to the system cart and then to the command module. The data may then be processed to interpret the current data and calculate the precise position and orientation of the sensor relative to the transmitter. Multiple sensors may be used at various locations on the endoscopic instrument, e.g., the reader and sheath, to calculate the individual positions of those components. Thus, based on readings from the artificially generated EM field, the EM tracker may detect changes in field strength as it moves through the patient's anatomy.

[0048] VC on-the-fly electromagnetic alignment 8A-8D show example graphs 810-840 illustrating on-the-fly (in operation) alignment of an EM system's path through a tubular network to a 3D model, according to one embodiment. The navigation configuration system described herein allows for on-the-fly alignment of EM coordinates to 3D model coordinates without the need for independent alignment prior to an endoscopic procedure. More specifically, FIG. 8A shows that the coordinate systems of the EM tracking system and the 3D model are initially unaligned with each other, and graph 810 in FIG. 8A shows the aligned (or expected) position of an endoscope tip 801 moving along a planned navigation path 802 through a branching tubular network (not shown here), where the aligned position of the instrument tip 801 and the planned path 802 are derived from the 3D model. The actual position of the tip is repeatedly measured by the EM tracking system 505, and multiple measured position data points 803 are generated based on the EM data. As shown in Figure 8A, data points 803 derived from EM tracking are initially located away from the alignment position of the endoscope tip 801 expected from the 3D model, reflecting a lack of alignment between EM coordinates and the 3D model coordinates. There may be several reasons for this, for example, even when the endoscope tip is moving relatively smoothly through the tubular network, there may still be some visible scattering in the EM measurements due to respiratory motion of the patient's lungs.

[0049] Points on the 3D model can also be identified and adjusted based on correlation between the 3D model itself, image data received from an optical sensor (e.g., a camera), and robot data from the robot commands. A 3D transformation between these points and the collected EM data points determines the initial alignment of the EM coordinate system to the 3D model coordinate system.

[0050] FIG. 8B illustrates graph 820 at a later time step compared to graph 810, according to one embodiment. More specifically, graph 820 illustrates that the expected position of the endoscope tip 801, as expected from the 3D model, has moved farther along the planned navigation path 802, as indicated by the shift from the original expected position of the instrument tip 801 shown in FIG. 8A along the path to the position shown in FIG. 8B. During the EM tracking between the generation of graph 810 and the generation of graph 820, additional data points 803 were recorded by the EM tracking system, but the alignment based on the newly collected EM data has not yet been updated. As a result, the data points 803 in FIG. 8B are clustered along the visible path 814, but the position and orientation of that path differs from the planned navigation path 802 along which the endoscope tip is directed by the operator to advance. Eventually, once enough data (e.g., EM data) has been accumulated, a relatively more accurate estimate can be derived from the transformations required to align the EM coordinates to those of the 3D model, as compared to using only the 3D model or only the EM data. The determination of sufficient data may be based on threshold criteria such as total accumulated data or number of direction changes, etc. For example, in a branching tubular network such as a bronchial network, sufficient data may be determined to have been accumulated after two branching points are reached.

[0051] FIG. 8C illustrates a graph 830 just after the navigation configuration system has accumulated a sufficient amount of data to estimate the registration transformation from EM coordinates to 3D coordinates, according to one embodiment. The data points 803 in FIG. 8C have now shifted from their previous positions as shown in FIG. 8B as a result of the registration transformation. As shown in FIG. 8C, the data points 803 derived from the EM data are now along the planned navigation path 802 derived from the 3D model, and each data point in the data points 803 now reflects a measurement of the expected position of the endoscope tip 801 in the coordinate system of the 3D model. In some embodiments, the registration transformation can be updated to improve accuracy as more data is collected. In some cases, the data used to determine the registration transformation can be a subset of the data selected in a moving window, which can change the registration over time, thereby providing the ability to account for changes in the relative coordinates of the EM and the 3D model, for example due to patient motion.

[0052] 8D shows an example graph 840 where the expected position of the endoscope tip 801 has reached the end of the planned navigation path 802 and reached the target location within the tubular network, according to one embodiment. As shown in FIG. 8D, the recorded EM data points 803 are now generally a track along the planned navigation path 802 that represents the tracking of the endoscope tip throughout the procedure. Each data point reflects a transformed position due to registration updates from the EM tracking system to the 3D model.

[0053] 8A-8D may be displayed sequentially on a display visible to a user as the endoscope tip advances through the tubular network. In some embodiments, the processor may be configured with instructions from the navigation configuration system such that the model shown on the display remains substantially constant as the measurement path shown on the display shifts to align the measurement data points on the display, thereby allowing the user to maintain a fixed frame of reference and remain visually constant in the model and planned path shown on the display.

[0054] 8E and 8F show the effect of registering an example EM system to a 3D model of a branching tubular network, according to one embodiment. In FIG. 8E and 8F, 3D graphs showing electromagnetic tracking data 852 and a model of a patient's bronchial system 854 are shown without (shown in FIG. 8E) and with (shown in FIG. 8F) a registration transformation. In FIG. 8E, without registration, tracking data 860 has a shape that corresponds to a path through bronchial system 854, but with an arbitrary offset and rotation. In FIG. 8F, by applying registration, tracking data 852 has been shifted and rotated to correspond to a path through bronchial system 854.

[0055] Mathematical analysis of the VD alignment transformation Regarding detailed analysis (e.g., mathematical analysis) and methods of registration, in some embodiments, a registration matrix can be used to perform the registration between the EM tracking system and the 3D model, and as an example, the matrix may represent translation and rotation in six dimensions. In alternative embodiments, a rotation matrix can be used to perform the registration using translation vectors.

[0056]

number

[0057]

number

[0058]

number

[0059] From a mathematical reasoning perspective, as an example, applying an alignment transformation involves a shift from one coordinate system (x,y,z) to a new coordinate system (x',y',z') that typically has the axes rotated to a different 3D orientation and where the origin is shifted an arbitrary amount in each dimension. For example, a radian rotation θ to an azimuth angle may be represented by matrix M1, a rotation φ to a tilt angle may be represented by matrix M2, etc., and further rotation matrices may be written as a product of rotation matrices. Similarly, a translation vector of (ΔxΔyΔz) may be chosen to represent the translation of the origin by Δx, Δy, and Δz in the x-axis, y-axis, and z-axis, respectively.

[0060] The registration transformation may be identified by methods such as singular value decomposition into a cross-correlation matrix between the measured EM positions and the estimated positions in the 3D model. The transformation matrix components may then be extracted from the decomposition, for example, by identifying appropriate principal components. An error signal may also be generated from the residuals of the identified transformation, and the size of the error signal may be used to identify a confidence level in the position. As more data is measured and the registration transformation is identified more accurately, this error signal may decrease, thus indicating an increased confidence in the estimated position.

[0061] VI. Navigation Configuration System VI.A. High-Level Overview of the Navigation Configuration System 9A-9C show block diagrams of an example of a navigation configuration system 900 according to one embodiment. More specifically, FIG. 9A shows a high-level overview of a block diagram of an example of a navigation configuration system 900 according to one embodiment. In FIG. 9A, the navigation configuration system 900 includes a plurality of input data stores, a navigation module 905 that receives various types of input data from the plurality of input data stores, and an output navigation data store 990 that receives output navigation data from the navigation module. The block diagram of the navigation configuration system 900 shown in FIG. 9A is merely an example, and in alternative embodiments not shown, the navigation configuration system 900 may include different and / or additional entities. Similarly, the functions performed by the various entities of the system 900 may vary according to different embodiments.

[0062] Input data, as used herein, refers to raw data collected and / or processed from input devices (e.g., command module, optical sensor, EM sensor, IDM) to generate estimated state information and output navigation data for the endoscope. The multiple input data stores 910-940 include image data store 910, EM data store 920, robot data store 930, and 3D model data store 940. Each type of input data store stores the type of data indicated by the name for access and use by the navigation module 905. Image data may include information such as one or more image frames taken by an imaging device at the instrument tip and a frame rate or timestamp that allows identification of the elapsed time between frame pairs. Robot data includes data related to the physical movement of the medical instrument or part of the medical instrument (e.g., the instrument tip or sheath) within the tubular network. Examples of robotic data include command data instructing the instrument tip to reach a particular anatomical site and / or reorient within the tubular network (e.g., with a particular pitch, roll, yaw, insertion, and retraction of one or both of the leader and sheath), insertion data representing the insertion motion of a portion of the medical instrument (e.g., the instrument tip or sheath), IDM data, and mechanical data representing the mechanical motion of an elongated member of the medical instrument, e.g., the motion of one or more pull wires, tendons, or shafts of an endoscope that drive the actual movement of the medical instrument within the tubular network. EM data is collected by EM sensors and / or EM tracking systems, as described above. 3D model data is derived from 2D CT scans, as described above.

[0063] The output navigation data storage 990 receives and stores the output navigation data provided by the navigation module 905. The output navigation data indicates information that assists in directing the medical instrument through the tubular network to reach a particular destination within the tubular network and is based on estimated state information of the medical instrument at each instant in time, the estimated state information including the position and orientation of the medical instrument within the tubular network. In one embodiment, the output navigation data is provided in real-time indicating the movement of the medical instrument and updates of the position / orientation information as the medical instrument moves within the tubular network to better assist in navigation through the tubular network.

[0064] To identify the output navigation data, the navigation module 905 finds (or identifies) the estimated state of the medical instrument within the tubular network. As shown in FIG. 9A, the navigation module 905 further includes various algorithm modules, such as an EM-based algorithm module 950, an image-based algorithm module 960, and a robot-based algorithm module 970, each of which may primarily use a particular type of input data and contribute different types of data to the state estimator 980. As shown in FIG. 9A, the different types of data output by these modules, labeled EM-based data, image-based data, and robot-based data, may generally be referred to as "intermediate data" for purposes of explanation. The detailed configuration of each algorithm module and the state estimator 980 will be described more fully below in FIG. 9B.

[0065] VI.B. Navigation Module 9B illustrates a block diagram of an example of the navigation module 905 shown in FIG. 9A, according to one embodiment. As mentioned above, the navigation module 905 further includes a state estimator 980 and a number of algorithm modules that utilize different algorithms to navigate through the tubular network. For clarity of explanation, the state estimator 980 will be described first, followed by a description of the various modules that exchange data with the state estimator 980.

[0066] VI.B.1 State Estimation Unit A state estimator 980 included in the navigation module 905 receives various intermediate data and provides an estimated state of the instrument tip as a function of time, where the estimated state indicates an estimated position and orientation information of the instrument tip within the tubular network. The estimated state data is stored in an estimated data storage device 985 included in the state estimator 980.

[0067] 9C illustrates a block diagram of an example of an estimated state data store 985 included in state estimator 980, according to one embodiment. Estimated state data store 985 may include branch data store 1086, position data store 1087, depth data store 1088, and orientation data store 1089, although this particular division of data stores is merely an example and in alternative embodiments not shown, different and / or additional data stores may be included in estimated state data store 985.

[0068] The various storage devices described above represent the estimated state data in a wide variety of ways. In particular, the bifurcation data refers to the position of the medical instrument with respect to a set of bifurcations (e.g., bifurcation, trifurcation, or division into four or more bifurcations) in the tubular network. For example, the bifurcation data can be a set of bifurcation options selected by the instrument as it moves through the tubular network based on a larger set of available bifurcations, e.g., as provided by a 3D model mapping the entire tubular network. The bifurcation data can further include information ahead of the instrument tip's location, such as bifurcations (bifurcations) that the instrument tip is close to but has not yet crossed, but that may have been detected based on, e.g., current position information of the tip relative to the 3D model or based on captured images of the upcoming bifurcations.

[0069] The position data indicates a three-dimensional position of a portion of a medical device within a tubular network or a portion of a tubular network embodiment thereof. The position data can be in the form of an absolute position or a relative position, for example, to a 3D model of the tubular network. As an example, the position data can include the position of a particular branch.

[0070] The depth data indicates depth information of the instrument tip within the tubular network. Example depth data include the total insertion (absolute) depth of the medical instrument into the patient and the (relative) depth within an identified bifurcation. The depth data may be determined based on position data for both the tubular network and the medical instrument.

[0071] The orientation data indicates orientation information of the instrument tip and may include overall roll, pitch, and yaw relative to the 3D model as well as pitch, roll, and roo within the identified branch.

[0072] Referring again to FIG. 9B, the state estimator 980 again provides the estimated state data to the algorithm module to generate more accurate intermediate data, which the state estimator uses to generate improved and / or updated estimated states, etc., forming a feedback loop. For example, as shown in FIG. 9B, the EM-based algorithm module 950 receives previous EM-based estimated state data (not shown in FIG. 9B), also referred to as data associated with timestamp "t-1". The state estimator 980 uses this data to generate "estimated state data (past)" associated with timestamp "t-1". The state estimator 980 then provides the data again to the EM-based algorithm module. The "estimated state data (past)" may be based on a combination of different types of intermediate data (e.g., robot data, image data) associated with timestamp "t-1" as generated and received from different algorithm modules. The EM-based algorithm module 950 then runs the algorithm using the estimated state data (past) to output improved, updated EM-based estimated state data to the state estimator 980, which is denoted here as "EM-based estimated state data (present)" and has a timestamp t associated with it. This process continues and is repeated for future timestamps as well.

[0073] Because the state estimator 980 may use several different kinds of intermediate data to arrive at an estimate of the state of the medical instrument within the tubular network, the state estimator 980 is configured to account for a variety of different kinds of errors and uncertainties, both in measurement and analysis, that each type of underlying data (robotic, EM, image) and each type of algorithmic module may produce or introduce into the intermediate data used for consideration in determining the estimated state. To address these, two concepts are considered: probability distributions and confidence values.

[0074] The "probability" in "probability distribution" as used herein refers to the probability that an estimate of a possible position and / or orientation of the medical instrument is accurate. For example, one of the algorithmic modules may calculate different probabilities indicating the relative probability that the medical instrument is in one of several different possible branches in the tubular network. In one embodiment, the type of probability distribution (e.g., discrete distribution and continuous distribution) is chosen to match the characteristics of the estimated state (e.g., the type of estimated state, e.g., continuous location information vs. discrete branch options). As an example, in a three-branch, the estimated state for identifying which segment the medical instrument is in may be represented by a discrete probability distribution and may include three discrete values ​​of 20%, 30%, and 50% representing the chance of being in a position within each of the three branches identified by one of the algorithmic modules. As another example, the estimated state may include that the roll angle of the medical instrument is 40±5 degrees and the segment depth of the instrument tip in the branch may be 4±1 mm, each represented by a Gaussian distribution, which is a type of continuous probability distribution. Different methods for generating the probabilities may be used, and the methods vary depending on the algorithm module, as will be described more fully below with reference to subsequent figures.

[0075] In contrast, a "confidence value," as used herein, reflects a measure of confidence in the state estimate provided by one of the algorithms based on one or more factors. For EM-based algorithms, factors such as distortions to the EM field, imprecision in the EM registration, patient shifts or movements, and patient breathing may affect the confidence in the state estimate. In particular, the confidence value in the state estimate provided by an EM-based algorithm may depend on the patient's particular breathing cycle, the movement of the patient or the EM field generator, and the location in the anatomical structure where the instrument tip is located. For image-based algorithms, examples of factors that may affect the confidence value in the state estimate include the lighting conditions at the location in the anatomical structure where the image is taken, the presence of fluid, tissue, or other obstructions behind or in front of the optical sensor taking the image, the patient's breathing, the general fluid inside the tubular network and the state of the tubular network (e.g., lungs), such as occlusion of the tubular network, and the particular motion technique used, for example, in navigation or image capture.

[0076] For example, one factor may be that a particular algorithm has different levels of accuracy at different depths in the patient's lungs, such that a particular algorithm may have a high degree of confidence in estimating the position and orientation of the medical instrument if it is relatively close to the airway opening, but may have a decreased degree of confidence as the medical instrument advances further into the base of the lung. In general, the confidence value is based on one or more systemic factors related to the process by which the outcome is identified, while the probability is a relative measure that arises when attempting to identify an accurate outcome from multiple probabilities using a single algorithm based on underlying data.

[0077] As an example, a formula for calculating the outcome of an estimated state represented by a discrete probability distribution (eg, branch / segment identification in a 3-branch involving an estimated state of three values) can be: S1=C EM *P 1,EM +C Image *P 1,Image +C Robot *P 1,Robot ; S2=C EM *P 2,EM+C Image *P 2,Image +C Robot *P 2,Robot ; S3=C EM *P 3,EM +C Image *P 3,Image +C Robot *P 3,Robot

[0078] In the above formula example, S i (i=1,2,3) represent examples of possible values ​​of the estimated state when three possible segments are identified or present in the 3D model, and C EM , C Image , and C Robot represents the confidence values ​​corresponding to the EM-based algorithm, the image-based algorithm, and the robot-based algorithm, and P i,EM , P i,Image , P i,Robot denotes the probability of segment i.

[0079] To better illustrate the concept of probability distribution and confidence value associated with the estimated state, a detailed example is provided here. In this example, the user is trying to identify the segment where the instrument tip is located within a specific three-branch in the central airway (prediction region) of the tubular network, and three algorithm modules are used, including an EM-based algorithm, an image-based algorithm, and a robot-based algorithm. In this example, the probability distribution corresponding to the EM-based algorithm may be 20% for the first branch, 30% for the second branch, and 50% for the third (last) branch, and the confidence value applied to this EM-based algorithm and the central airway is 80%. In the same example, the probability distribution corresponding to the image-based algorithm is 40%, 20%, and 40% for the first, second, and third branches, respectively, and the confidence value applied to this image-based algorithm is 30%, while the probability distribution corresponding to the robot-based algorithm is 10%, 60%, and 30% for the first, second, and third branches, respectively, and the confidence value applied to this image-based algorithm is 20%. The difference in confidence values ​​applied to the EM-based and image-based algorithms indicates that for segment identification in the central airway, the EM-based algorithm may be a better option compared to the image-based algorithm. Finally, an example of the mathematical calculation of the estimated state is: In the first branch, 20%*80%+40%*830%+10%*820%=30%; in the second branch, 30%*880%+20%*830%+60%*820%=42%; and in the third branch, 50%*880%+40%*830%+30%*820%=58% It can be.

[0080] In this example, the output estimated state of the instrument tip can be the result values ​​(e.g., the results 30%, 42%, and 58%) or a derivative of these result values, such as a determination that the instrument tip is in the third branch.

[0081] As mentioned above, the estimated state may be represented in a number of different ways. For example, the estimated state may further include a set of data representing the absolute depth from the airway to the position of the tip of the instrument and a set of branches traversed by the instrument in the tubular network, the set being a subset of the total set of branches provided, for example, by a 3D model of the patient's lungs. Application of probability distributions and confidence values ​​to the estimated state may improve the accuracy of the estimation of the position and / or orientation of the tip of the instrument in the tubular network.

[0082] VI.B.2 Algorithm Module As shown in Figure 9B, the algorithm modules include an EM-based algorithm module 950, an image-based algorithm module 960, and a robot-based algorithm module 970. The algorithm modules shown in Figure 9B are merely examples, and in alternative embodiments, different and / or additional algorithm modules including different and / or additional navigation algorithms may also be included in the navigation module 905.

[0083] VI.B.2.I. EM-Based Algorithm Module The EM-based algorithm module 950 further includes an EM alignment module 952 and a branch selection module 954. The EM alignment module 952 performs alignment of EM coordinates to 3D model coordinates. Figure 10A illustrates a block diagram of an example of the EM alignment module 952 according to one embodiment. The EM alignment module 952 receives as inputs estimated state data (past) (e.g., branch data) from the estimated state data store 1086, EM data from the EM data store 920, and 3D model data from the 3D model data store 940.

[0084] Based on the received data, the EM registration module 952 performs an on-the-fly registration of the EM tracking data to the 3D model, as described above with respect to Section V. After an initial registration is determined, the EM registration module 952 continuously updates an estimate of the registration transformation based on the received data to improve the accuracy of the transformation and to compensate for changes to the navigation configuration system 900, such as changes due to patient motion. The EM registration module 952 outputs the registration transformation data to the registration transformation data storage 1053. In one embodiment, the registration transformation data reflects the best-matched registration transformation and may also be sent to the state estimator 980 and the branch selection module 954.

[0085] 10B illustrates a block diagram of an example of the branch selection module 954 according to one embodiment. The branch selection module 954 receives as inputs estimated state data (past) (e.g., branch data) from estimated state data store 985, EM data from EM data store 920, registration transformation data from registration transformation data store 1053, and 3D model data from 3D model data store 940. Based on the received data, the branch selection module 954 determines an estimate of the position and orientation of the endoscope tip relative to the 3D model of the tubular network, and provides EM-based estimated state data (current) to the state estimator 980. By way of example, the EM-based estimated state data may be represented as a probability distribution (e.g., discrete distributions of 20%, 30%, and 50% for three segments of three branches, as described above). Furthermore, when at a branch indicated by the received branch data, the branch selection module 954 may compare the pitch and yaw of the tip to the angle of each branch in the 3D model to estimate which branch was selected by the user to follow. The branch selection module 954 outputs the EM-based estimated state data (current) to the estimated data storage device 985 .

[0086] VI.B.2.II. Image-Based Algorithm Module Referring again to Figure 9B, the image-based algorithm module 960 uses the image data to identify an estimated state of the instrument within the tubular network. The image-based algorithm module 960 further includes one or more different types of image-based algorithm modules that utilize different image-based algorithms. As shown in Figure 9B, an example is shown that includes an object-based algorithm module 962. In alternative embodiments not shown, other types of image-based algorithms may be utilized and corresponding algorithm modules may be included in the image-based algorithm module 960.

[0087] The object-based algorithm module 962 detects and analyzes objects present in the field of view of the image data, such as branching apertures or particles, to identify an estimated state. In one embodiment, the object-based algorithm module 962 includes an object detection module 963, an object mapping module 964, a topological reasoning module 965, and a motion estimation module 966. In some embodiments, the object-based algorithm module 962 may or may not need to apply the different modules 963, 964, 965, and 966 in a certain order, and when actually performing the process of the object-based algorithm described by the object-based algorithm module 962, the order in which each module in the module 962 is utilized is different from that shown in FIG. 9B.

[0088] With reference to FIG. 10C, the motion estimation module 963 receives as input image data from the image data store 910, estimated state data (past) (particularly branch data) from the estimated state data store 985, and 3D model data from the 3D model data store 940. Based on the received image data, the motion estimation module 963 measures the motion of the medical instrument between multiple image frames based on the received image data. Example techniques used include optical flow and image registration techniques, among others. This measurement identifies differential motion, such as forward and backward motion or rotational motion of the instrument tip in its own local reference frame. This motion can be combined with the previous estimated state input to calculate a new estimated state. In particular, forward (or backward) motion can be translated into an increase (or decrease) in depth from the previous estimated state. Similarly, differential roll is translated into a change in roll angle from the previous estimated state. These measurements allow the movement through the tubular network to be estimated. As mentioned above, these estimates can be expressed as a probability distribution (e.g., a roll angle of the medical instrument of 40±5 degrees represented by a Gaussian distribution). The output estimated states are stored in the estimated states data store 985 .

[0089] In one embodiment, if the estimated state and bifurcation data at a particular instant indicate that the instrument tip is at or near a bifurcation, this motion estimation may include identifying a new estimated bifurcation that the instrument tip is entering or is estimated to have entered. For example, if the bifurcation data indicates that the endoscope tip is at a bifurcation, pitch and yaw motions may be measured to identify a change in pointing angle, and the new estimated angle may be compared to expected angles of different bifurcations in the 3D model of the tubular network. It may then be determined which bifurcation the endoscope will face as it enters the new bifurcation. Estimated state data reflecting each of these estimates of the new position, orientation, and / or bifurcation entry is output to state estimator 980.

[0090] 10D illustrates a block diagram of an example of an object detection module 964 according to an example. The object detection module 964 receives as input image data (e.g., image frames) and outputs object data to an object data store 1063 and estimated state data to an estimated state data store 985. The object data indicates information about what objects have been identified as well as the object's position, orientation, and size expressed as a probability.

[0091] In particular, the object detection module 964 detects one or more objects and one or more features of the objects in the image that may indicate branching points in the tubular network, and then identifies their location, size, and orientation. The objects may be calculated or represented in the object detection module 964 as two-dimensional shapes such as circles / ellipses / ovals for the detected branching points. This corresponds to the image data used to capture the objects being images from a camera on the instrument tip that is typically oriented along an axis substantially parallel to the direction of the segment in which the instrument is. As a result, objects such as branches in the tubular network appear as simple shapes such as ellipses in the image. In one embodiment, in a given image of a tubular network, each branch typically appears as a dark roughly elliptical region, and these regions may be automatically detected as objects by the processor using a region detection algorithm such as Maximum Stable Extremum Region (MSER). These regions may then be fitted with appropriate free parameters such as the ellipse center, major and minor axes, and angle in the image to define the object (e.g., an ellipse). The rotation measurements and identified matches between the model lumens and lumens in the image are also output to the state estimator 980 and the topological reasoning module 966. An example of identified objects, along with links connecting their centers, superimposed on an image of the bronchial network is described with reference to Figures 11A and 11B.

[0092] In one embodiment, the "airway" may also be identified as an object present in the image data. The object detection module 964 may use light reflection intensity in combination with other techniques to identify the airway.

[0093] The object detection module 964 may further track the detected object across a set of sequential image frames to detect which of a set of possible branches in the tubular network it has entered. Tracking of the relative position of the object in the image frames may be used to determine a local absolute measurement of the roll angle within the branching network.

[0094] 10E illustrates a block diagram of an example of an object mapping module 965 according to one embodiment. The object mapping module 965 receives as inputs 3D model data from 3D model data store 940, object data (e.g., detected objects such as shapes representing possible branches in a tubular network) from object data store 1063, and estimated state data (past) from estimated state data store 985.

[0095] Based on the received input data, the object mapping module 965 outputs object mapping data to the object mapping data storage 1065 and outputs image-based estimated state data (current) to the estimated state data storage 985. As an example, the object mapping data indicates mapping information between the physical branches (lumens) shown in the image data (based on the detected objects) and the virtual branch information generated by the 3D model. The estimated state data (current) generated by the module 965 includes an identification of each branch of the tubular network visible in the image and an estimation of the rotation of the endoscope tip relative to the 3D model. As described above, the estimated state data (current) can be represented as a probability distribution. The identification of visible lumens can include, for example, the coordinates in x and y of each identified lumen center in the image based on the association of the object size and each identified lumen location with a particular branch of the tubular network correlated with the 3D model virtual image data.

[0096] In some embodiments, the 3D model is generated prior to the endoscopic procedure, so that a virtual image of the tubular network can be pre-computed to accelerate the process. In an alternative embodiment not shown, the tubular network can be represented by a structure such as a tree diagram of luminal midlines, each such midline describing a 3D path, thereby allowing the expected location of local bifurcation centers from any viewpoint to be compared with the actual location of the bifurcation centers identified based on EM and / or robotic data.

[0097] 11A and 11B show an example object-lumen mapping performed by object mapping module 965, according to one embodiment. More specifically, FIG. 11A shows two example identified objects 1101 and 1102 superimposed on an image of a bronchial network 1105, along with a link 1103 connecting the centers of the two objects, according to one embodiment. In the example shown, the identified objects 1101 and 1102 are oval shaped.

[0098] 11B illustrates the matching of airway lumens in an actual image 1110 of an actual bronchial network with a corresponding virtual image 1120 from a 3D model of the same network, according to an embodiment. In the actual image 1110, ellipses are identified that correspond to two different branches identified, in one embodiment, with identified centers 1111 and 1112, which indicate the centerline coordinates of the branches as described above in FIGs. 6A and 6B. The 3D model virtual image 1120 is a simulated representation of the actual bronchial network shown in the actual image 1110, with estimated centers 1121 and 1122 of the endoscope tip identified by the state estimator 980 shown corresponding to the locations of the identified centers 1111 and 1112.

[0099] When both images 1110 and 1120 are presented to a user via a user interface, the 3D model image 1120 can be rotated or translated to improve the match between the actual image 1110 and the virtual image 1120, and the amount of rotation required for the rotation or translation can be output as a correction to the current estimated state (e.g., rotation of the instrument tip).

[0100] In one embodiment, the probabilities applied to the possible estimated states generated by the object mapping module 965 are based on the degree of correspondence between the identified centers 1111 and 1112 detected in the actual image 1110 and the estimated centers 1121 and 1121 in the 3D model image 1120; as an example, the probability of being within a lumen with the identified center 1112 decreases as the distance between the estimated center 1122 and the identified center 1112 increases.

[0101] 10F illustrates a block diagram of an example of a topological reasoning module 966 according to one embodiment. The topological reasoning module 966 receives as inputs image data from 3D model data store 940, object mapping data from object mapping data store 1065, and estimated state data (past) from estimated state data store 985.

[0102] Based on the received data, the topological reasoning module 966 determines which branch the endoscope tip is facing, and thereby generates a prediction of which branch it will enter if it moves forward. As described above, this determination may be expressed as a probability distribution. In one embodiment, when the instrument tip moves forward, the topological reasoning module 966 determines that a new branch of the tubular network has been entered, and identifies which branch the tip has entered into. The determination of which branch is facing and which segment it will enter may be made, for example, by comparing the relative size and position of different identified objects (e.g., ovals). As an example, as a particular lumen branch is entered, the corresponding object detected will grow larger in successive image frames and also become more centrally located in those frames. This means that if a behavior is identified for one of the objects, the topological reasoning module 966 assigns increasingly larger probabilities to the corresponding estimated state as the endoscope tip moves toward the lumen associated with that object. Correspondingly, other branches are assigned lower probabilities until finally the object shape entirely disappears from the image. In one embodiment, the probability that the medical instrument is within those branches depends only on the probability that the branch was misidentified by the object mapping module 964. The output of the topological reasoning module 966 is image-based estimated state data representing the estimated probability of being within each of a set of possible branches in the branch network.

[0103] VI.B.2.III. Robot-Based Algorithm Module The robot based algorithm module 970 uses the robot data to provide robot based estimated state data to the state estimator 980. FIG. 9B shows the robot based algorithm module 970 receiving estimated state data (past) and providing estimated state data (present) to the state estimator 980. The robot data includes data related to the physical movement of the medical instrument or a portion of the medical instrument (e.g., the instrument tip) within the tubular network. Example robot data include command data instructing the instrument tip to reach a particular anatomical site and / or change orientation within the tubular network (e.g., with a particular pitch, roll, yaw, insertion, and retraction of one or both of the leader and sheath), insertion data representing the insertion movement of a portion of the medical instrument (e.g., the instrument tip or sheath), IDM data, and mechanical data representing the mechanical movement of an elongated member of the medical instrument, e.g., the movement of one or more pull wires, tendons, or shafts of an endoscope that drive the actual movement of the medical instrument within the tubular network.

[0104] In an ideal system, the input pitch, roll, yaw, insertion, and retraction commands given to the IDM specifically to control the instrument tip would cause the instrument tip to change motion exactly as input, but in practice this is generally not the case. Friction, nonlinearities in the instrument motion, obstructions, and other effects in the system may cause the input motion to differ from the output motion. Thus, the estimated state data provided by the raw robot input data is only an estimate of the actual motion. As with the above algorithm, the estimated state data determined from the robot data may be probabilistic to account for this uncertainty in the actual position information.

[0105] VI.C. Generating Probability Distributions 12 illustrates an example process for generating a probability distribution over a number of values ​​that specify an estimated state, as may be performed by any individual algorithm module described above, according to one embodiment. For purposes of example, this process is described with respect to estimated state module 980, but in practice may be used by any algorithm module.

[0106] In Fig. 12, the process of the Bayesian estimation method is shown, and in alternative embodiments not shown, other methods of generating probability distributions can be utilized. As shown in Fig. 12, the state estimator 980 first identifies (1201) a prior probability distribution of the estimated state. At this stage, this probability distribution can be, for example, a value distribution initialized based on user input identifying a starting point close to an entry point into the branching network. Subsequently, a distribution is identified from a prior output state estimate that is adjusted based on robot, image, or EM input data.

[0107] For example, if the surgical robotic system 100 commands the endoscope tip to move forward a particular distance, e.g., 1 mm, it may estimate a new or updated probability distribution as having a center 1 mm forward compared to the previous probability distribution. More generally, if a robotic command is expected to change a variable x by a particular amount, represented by an expected distribution of value changes Q(Ax), the new estimated distribution P'(x) may be expressed as the convolution of the previous distribution P(x) with Q(Ax). This new estimated state is treated as the previous probability estimate for the variable in subsequent steps.

[0108] In the next step 1202, the state estimator 980 receives an estimate distribution of the estimated state based on one or more algorithmic modules. This value distribution may be expressed in various ways, such as expected values ​​and error ranges or explicit distribution functions over the values. In either case, the value distribution includes some estimate of the expected values ​​and error ranges of the variables, either explicitly or implicitly. In some cases, only expected values ​​are transmitted and error ranges are identified by the state estimator 980 using records of past performance or pre-calibrated confidence values. In one embodiment, when using expected values ​​and error ranges, the state estimator 980 may treat the estimate distribution as a normal distribution with the expected values ​​having a mean and a standard deviation determined by the error range. In the case of discrete variables or discrete states, such as branch identities, the estimate distribution typically includes one or more probability values ​​assigned to each of one or more branches, such that the total probability adds up to 1 (or 100%). In some cases, some branches may not be assigned explicit probabilities, and the total probability may be less than 1, in which case the remaining probabilities can be treated as being uniformly distributed across all other branches.

[0109] In step 1203, the state estimator 980 generates a posterior probability distribution based on the estimate distribution received from step 1202 and the prior probability distribution identified from step 1201. As an example, this posterior probability distribution is calculated as an application of Bayes' theorem, which states that for an observation A, the posterior probability P(x|A) of a variable having value x is given by the formula

[0110]

number

[0111] The state estimator 980 determines each of the quantities P(x), P(A), and P(A|x) from the inputs generated during steps 1201 and 1202 as well as from a base model of a set of possible observations. For example, P(A), which represents the prior probability of x, may simply be read from the value determined in step 1201. Similarly, P(A|x) may be read from the function determined in step 1202. P(A) may be set as a constant representing the number of possible values ​​of A. For example, if estimating a discrete probability distribution of branches where an endoscope tip faces a division into a certain number (e.g., 10) of branches, the probability P(A) of observing the tip falling into any given branch may be set as 1 / 10. Alternatively, the model may be implemented to variably weight the possible observations A, e.g., if the branch sizes vary, P(A) may be set proportional to the size of branch A divided by the sum of all branch sizes, which represents the notion that a randomly placed tip is more likely to be found in a large branch than a small branch. For continuous distributions such as roll angle, depth, or orientation, P(A) is the probability density function over all possible observations A. For example, if roll angles range from 0 radians to 2π radians, P(A) may be set to 1 / (2π) for all values ​​of A, representing the equally likely probability density of any roll angle being observed. More generally, given P(x) as an estimate of the probability of x, P(A) is given by the formula P(A)=P(A|x)P(x)+P(A|~x)P(~x), where ~x means "not x".

[0112] In one embodiment, the result of step 1203 is a new probability distribution of values ​​of the variable / state. In one embodiment, if multiple independent measurements each relate to a given variable, the multiple independent measurements may be adjusted in sequence by taking the output of step 1203 as input to step 1201 and using each new measurement as the estimate distribution in step 1202. This creates a loop 1205 over the measurements. Alternatively, step 1202 may incorporate multiple independent estimate distributions, which may be combined into a single updated estimate in step 1203 using Bayesian conditional updating on the multiple measurements.

[0113] For example, if differential motion is being measured, the motion estimation module 963 may utilize a method 1200 having a process that takes a previous probability distribution in step 1201 over the motion quantities (e.g., expected range of motion based on robot data), receives an estimate distribution in step 1202 over the differential motion measurements, and generates a posterior probability distribution in step 1203 to estimate the actual value of the motion. This output may then be convolved with the pre-estimated values ​​(from the previous estimated state) to generate a posterior probability distribution for multiple values ​​of the state. In some embodiments, once all measurements have been incorporated into the updated probability distribution, this new probability distribution is reported in step 1204. The process 1200 may be applied in parallel to generate new probability distributions for multiple variables (states), such as position, orientation, roll, depth, and / or branching. The output of step 1204 of each such process may be combined together to generate a new estimated state (E) 1202, which represents the output of the motion estimation module 963.

[0114] VI.D.Error correction In some embodiments, the navigation module 905 allows for estimation of variables even when certain input data is not available. For example, before registration of the EM system, the output of the branch selection module 954 is ignored by the state estimator 980. Nevertheless, state representing position and / or orientation (e.g., tip insertion depth) may still be estimated based on available input data, such as robotic input data. Each time the instrument tip is commanded to move deeper into the branch network, the estimated state of tip insertion depth may be updated based on this estimated movement. Thus, before registration, tip insertion depth may be estimated based on the robotic data, and after registration, the depth may be estimated based on the data from the branch selection module 954, but also in part based on the robotic data, using a weighting function.

[0115] Based on the many possible independent measurements mentioned above, including global measurements such as object-based image tracking, optical flow, EM tracking, and local measurements such as measurements from the robot input database, there is a probability of over-determination / over-fitting of the instrument's position estimate. As a result, the accuracy of the estimated position within the branching network may be greater than the accuracy of an estimate made using any one of the modules alone. Furthermore, in some embodiments, the navigation configuration system 900 demonstrates the ability to recover from errors due to using measurements from one module that contradict measurements from another module, allowing the system to "rethink" a previously made determination. An example of how errors can be corrected is described more fully below with reference to FIG. 13.

[0116] FIG. 13 shows an example of how the navigation configuration system 900 can correct errors in navigation through a tubular network, according to one embodiment. In FIG. 13, a simplified model of a branching tree 1300 with estimated states is shown at four different time stages: stages 1310, 1320, 1330, and 1340. At each time stage, the estimated position of the instrument tip used for navigation (1312, 1322, 1332, and 1342, respectively) is shown, as well as the corresponding actual position of the tip (1311, 1321, 1331, and 1341, respectively). In one embodiment, over time during the four stages, the accuracy of the estimated state (based on underlying probabilities and confidences) may change, for example, the estimated state may start out accurate, become inaccurate, and then become accurate again.

[0117] 13, initially, in state 1310, the tip's actual location is 1311 in the top branch of the branch tree 1300, and the navigation configuration system 900 accurately estimates that location as an estimated location 1312. However, when entering one of two branches near the actual location 1311, one or more modules within the system 900 may misidentify the branch based on the probability provided by one or more of the algorithmic modules, causing the system to conclude that the tip is in the left branch 1345 when in fact the tip is in the right branch 1346. For example, the state estimator 980 initially assigns an 80% chance of being in the left branch and a 20% chance of being in the right branch.

[0118] However, as the endoscope tip progresses down the right branch 1346 to state 1320, the estimated location 1322 becomes increasingly spatially distant from the actual location 1321. Thus, in one embodiment, the branch selection module 954 may report with increasing confidence that the tip is probably in the right branch 1346, based on the shift in probabilities provided by the underlying modules. Thus, the aggregate probability estimates of the state estimator 980 may also shift, increasing the probability of being in the right branch 1346 and correspondingly decreasing the probability of being in the left branch 1345.

[0119] The system 900 may proceed to state 1330, where the tip is at actual position 1331 and reaches the split into three branches. However, at this time state, the estimator 980 may still estimate that the most likely state is estimated position 1332.

[0120] In state 1340, based on the 3D model, the state estimator 980 may anticipate a certain distance to the next branch split, in this example a 2-branch split rather than a 3-branch split. In this situation, the branch selection module 954 and the object mapping module 964 may both strongly estimate that the tip is in the branch shown on the right, which further strongly adjusts the probabilities of being in the left and right branches, leading to a near-certain assignment of the tip location to the correct branch, here the right branch 1346.

[0121] As a result, in state 1340, state estimator 980 accurately estimates that the instrument tip is at estimated location 1342, which is very close to actual location 1341. If the endoscope is along the user's desired path, system 900 can now proceed to determine which branch of the three-way branch to enter next. Alternatively, the user can backtrack and continue down the left lumen of the bifurcation tree 1300. In some embodiments, this updated estimate is shown to the user on the display so that the user can be aware that the previous estimate was incorrect and that the error has now been corrected.

[0122] VII. Preoperative route planning for navigation preparation Navigation to a particular point in a tubular network of a patient's body requires that certain steps be taken pre-operatively to create a 3D model of the tubular network and generate the information necessary to determine a navigation path within the 3D model. Figures 14A-14C show example pre-operative steps for preparing a surgical instrument (e.g., an instrument tip) to navigate through an example tubular network, according to various embodiments.

[0123] 14A illustrates an example pre-operative step sequence of a method 1400 for navigating an instrument tip to a particular site within a tubular network, according to one embodiment. Beside each step of the method 1400, a corresponding image is shown showing a representation of the data involved in planning a path and navigating through the tubular network.

[0124] First, a CT scan of the tubular network is obtained in step 1405, where the data from the CT scan provides 3D information about the structure and connections of the tubular network. For example, the image in step 1405 shows a cross-sectional slice of a patient's lung.

[0125] In step 1410, a 3D model is generated based on the acquired CT scan data, and the generated 3D model can be used to assign a unique identity to each branch of the tubular network, allowing for convenient navigation within the network. For example, the image in step 1410 shows a 3D model of the patient's bronchial network.

[0126] In step 1415, a target 1416 is selected, which may be, for example, a lesion to be biopsied or a portion of an organ tissue to be surgically repaired. In one embodiment, a user may select the location of the target by interacting with a computer display, which may show a 3D model, such as by clicking with a mouse or touching a touch screen. The selected target may then be displayed to the user. For example, the target 1416 is marked in the 3D bronchial model generated from step 1410.

[0127] In step 1420, a path 1421 is anatomically planned from an entry point 1422 to the target 1416, where the path 1421 identifies a series of branches in the network to follow to reach the target 1416. In one embodiment, the tubular network may be tree-like and the path 1421 may be uniquely determined by the structure of the tubular network, while in another embodiment, the tubular network may be cyclic and the path may be found by an appropriate algorithm, such as a shortest path algorithm.

[0128] Once the path 1421 is determined, a virtual endoscopy 1425 may be performed to provide the user with a preview of the endoscopic procedure. The 3D model generated from step 1410 is used to generate a series of 2D images as would be seen by the endoscope tip while navigating the net corresponding to the 3D model. The path 1421 may be shown as a curve that may be traced from the entry point 1422 to the target 1416.

[0129] Once the virtual endoscope tip reaches the location of the target 1416, a virtual instrument alignment procedure 1430 may be performed to show the user how to manipulate the endoscopic instrument to perform the surgical procedure at the target location. For example, in the illustration, the user manipulates a biopsy needle 1431 of the virtual endoscope to biopsy a lesion 1432 that is beneath the surface of a bronchial tube. The lesion location is highlighted, allowing the user to align the needle to the lesion and then use the needle to pierce the surface of the bronchial tube to access the lesion beneath. This mimics the steps that would be performed during an actual surgical procedure, allowing the user to practice before performing the procedure.

[0130] 14B shows an example user interface 1434 for navigating a surgical instrument through a tubular network, according to one embodiment. The user interface 1434 allows a user to have various views of the tubular network during an endoscopic procedure. In addition to a real-time view from an imaging device (e.g., an endoscopic camera), the user interface 1434 may show the user various navigational views. For example, the location of the endoscope tip may be overlaid on a slice segment of a CT scan 1435, with a marker (e.g., crosshairs 1436) indicating the location of the instrument. In this case, the CT image is an image of the patient's lungs, and the endoscope used is a bronchoscope.

[0131] 14B, user interface 1434 also shows a view 1440 representing the 3D model with a highlighted path 1441 and icons indicating endoscope position 1442 and target position 1443. Several additional icons 1444 of user interface controls that enable features such as map panning and zooming are also shown in view 1440.

[0132] View 1445 shows a virtual endoscopic view from the estimated tip position which can be compared to the real-time view to confirm that the tip position has been accurately estimated. Path indicators 1446 and / or position / orientation indicators 1447 to aid in navigation are also shown in view 1446. The top left shows state information 1448 determined by state estimator 980, indicating that the current distance to target "t1" is 5.4 cm and the calculated roll angle is -139 degrees.

[0133] View 1450 shows a virtual view of the endoscope shaft 1451 displayed from a "third person" perspective up close to the endoscope tip. The virtual view can be generated by assuming that the endoscope shaft is along a path already traveled by the tip, and a view of the surrounding area is generated based on a 3D model near the selected perspective. Alternatively, the view can be generated from images taken during a previous pass through the area in the procedure. In an alternative embodiment, the virtual perspective can project beyond the endoscope tip position to show the next branch. This view can further highlight the intended path, informing the user in advance which branch to steer into when the endoscope tip reaches the next branch.

[0134] 14C illustrates an example user interface 1480 for alignment and control of an instrument during an endoscopic procedure, according to one embodiment. View 1455 illustrates a cross-sectional view at a later stage in the procedure than view 1435, with the endoscope tip near the target and an endoscopic instrument, in this case a biopsy needle, being deployed from the working channel of the endoscope. Crosshairs 1436 highlight the new location of the tip. Because the bronchoscope in the image is further into the patient's lungs, the cross-sectional slice has changed to show a deeper cross-sectional segment. The icon identifying the location of the endoscope tip has been replaced with an icon showing the needle's location and orientation. View 1460 illustrates a map view of the 3D model zoomed around the target location 1443 and an instrument icon 1461 representing the biopsy needle. At the target 1443 is a biopsy location 1466 where the biopsy needle should be applied. As shown in this view, the endoscope tip has been navigated to approximately the end of the pathway 1441 and has reached a position close enough to the target location 1443 that the needle can reach the biopsy location 1466.

[0135] View 1465 shows a virtual view near target location 1443, highlighting biopsy location 1466 to help the user visualize precisely where the needle should be inserted to collect tissue for biopsy. Additional information such as target size and distance to target may also optionally be overlaid on the image.

[0136] View 1670 shows a zoomed-in view of a 3D icon 1471 representing a biopsy needle superimposed on a cross-sectional image of the patient's lungs. The roll of the instrument is indicated by a multi-colored pattern 1475 near the 3D icon 1471. As can be seen from the state information 1473, this particular orientation represents a 0 degree roll, and the needle is currently 4 mm from the biopsy location 1466. A path 1474 is displayed, indicating how the needle should be aimed to contact the biopsy location 1466.

[0137] VIII. Mechanical Configuration of Navigation System More generally, the navigation and tracking techniques disclosed herein may be implemented using a suitably configured computer system. The processor in the computer system may comprise one or more components that process electronic signals and may comprise one or more of a central processing unit, a video processor, a logic circuit, a gate array logic, a field programmable gate array, an integrated circuit, or an application specific integrated circuit. The computer system includes a central processing unit (CPU, also referred to herein as "processor" and "computer processor"), which may be a single-core or multi-core processor or multiple processors for parallel processing. The CPU may execute a series of machine-readable instructions, which may be embodied in a program or software. The instructions may be stored in a memory location. Examples of operations performed by the CPU may include fetch, decode, execute, and writeback. The CPU may be part of a circuit, such as an integrated circuit. One or more other components of the system may be included in the circuit. In some cases, the circuit includes an application specific integrated circuit (ASIC).

[0138] A computer system may also include one or more of the following: memory or memory locations (e.g., random access memory, read-only memory, flash memory), electronic storage units (e.g., hard disks), communications interfaces (e.g., network adapters) for communicating with one or more other systems, and peripherals such as cache, other memory, data storage devices, and / or electronic display adapters. The memory, storage units, interfaces, and peripherals communicate with the CPU through a communications bus, such as a motherboard.

[0139] The storage unit may be a data storage unit (or data repository) that stores data. The computer system may be operatively coupled to a computer network ("network") using a communication interface. The network may be the Internet, an internet, and / or an extranet, or an intranet and / or an extranet that communicates with the Internet. In some cases, the network may be a telecommunications network and / or a data network and may include one or more computer servers. The storage unit may store files such as drivers, libraries, and saved programs. The storage unit may store user data, e.g., user preferences and user programs. The computer system may include one or more additional data storage units that are external to the computer system, such as located on a remote server that communicates with the computer system through an intranet or the Internet in some cases.

[0140] The methods described herein can be implemented by machine (e.g., computer processor) executable code stored in an electronic storage location of a computer system, such as a memory or electronic storage unit. The machine executable or machine readable code can be provided in the form of software. In use, the code can be executed by the processor. In some cases, the code can be retrieved from a storage unit and stored in the memory so as to be accessible by the processor. In some situations, the electronic storage unit can be omitted and the machine executable instructions are stored in the memory.

[0141] The code may be precompiled and configured for use with a machine having a processor configured to execute the code, or may be compiled during run-time. The code may be provided in a programming language that may be selected to allow the code to be executed in a precompiled or compiled state.

[0142] The methods and systems of the present disclosure may be implemented by one or more methods. The methods may be implemented using a processor as described herein, for example, by software when executed by one or more computer processors.

[0143] IX. Additional Considerations Alternative views and embodiments of surgical robotic system 1100, surgical robotic system 1200, and other surgical robotic systems including the above-described components are further shown and described in U.S. Provisional Patent Application No. 62 / 162,486, filed May 15, 2015, U.S. Provisional Patent Application No. 62 / 162,467, filed May 15, 2015, U.S. Provisional Patent Application No. 62 / 193,604, filed July 17, 2015, U.S. Provisional Patent Application No. 62 / 201,518, filed August 5, 2015, U.S. Provisional Patent Application No. 62 / 203,530, filed August 11, 2015, and U.S. Provisional Patent Application No. 62 / 235,394, filed September 30, 2015.

[0144] Upon reading this disclosure, those skilled in the art will recognize additional and alternative structural and functional designs in accordance with the principles disclosed herein. Thus, while specific embodiments and applications have been shown and described, it should be understood that the disclosed embodiments are not limited to the precise configurations and components disclosed herein. Various modifications, changes and variations, apparent to those skilled in the art, may be made in the arrangement, operation and details of the methods and apparatus disclosed herein without departing from the spirit and scope as defined by the appended claims.

[0145] As used herein, any reference to "one embodiment" or "an embodiment" means that a particular element, feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily all refer to the same embodiment.

[0146] Some embodiments may be described using the phrases "coupled" and "connected," along with their derivatives. For example, some embodiments may be described using the term "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other. The embodiments are not limited in this context unless otherwise stated.

[0147] As used herein, the terms "comprises," "comprising," "includes," "including," "has," "having," or any other variation thereof, are intended to cover non-exclusive inclusions. For example, a process, method, article, or apparatus that includes a list of elements is not necessarily limited to only those elements, and may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless expressly indicated to the contrary, "or" refers to an inclusive "or" rather than an exclusive "or." For example, conditions A or B satisfy any one of the following: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), or both A and B are true (or exist).

[0148] Furthermore, the use of "a" or "an" is used to describe multiple elements and multiple components of the embodiments herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one, and the singular also includes the plural unless it is clear that this is not meant. (Embodiment 1) accessing robotics data related to the physical manipulation of an elongated medical instrument to be inserted into a patient's vascular network; accessing image data captured by an imaging device disposed adjacent a tip of the elongated medical instrument; accessing EM data captured using an EM (Electromagnetic) sensor located near the instrument tip and at least one external EM sensor located outside the patient's body or EM generated outside the patient; determining a robot based estimated state of the instrument tip based on the robot data, the robot based estimated state being associated with a first confidence value; determining an image based estimated state of the instrument tip based on the image data, the image based estimated state being associated with a second confidence value; determining an EM based estimated state of the instrument tip based on the EM data, the EM based estimated state being associated with a third confidence value; determining an estimated state of the instrument tip based on the robot-based, image-based, and EM-based estimated states and confidence values; A method comprising: (Embodiment 2) The method of embodiment 1, wherein the elongated medical instrument is a flexible endoscope comprising an endoscopic structure and a plurality of cables that can be variably retracted or associated with changing the direction of the instrument tip, and the endoscope is coupled to an instrument device manipulator (IDM) to variably insert, retract, and roll the endoscope from or within the tubular network. (Embodiment 3) The robot data includes: pitch, roll, and yaw of the instrument tip based on tension of at least one of a plurality of cables of the elongated medical instrument; and Insertion distance of the elongated medical device into the tubular network 3. The method of embodiment 1 or 2, comprising at least one of: (Embodiment 4) A method according to any one of the preceding claims, wherein the image data is based on a series of images taken sequentially in time, each image in the series being associated with a timestamp. (Embodiment 5) 5. The method of any one of claims 1 to 4, wherein the EM sensor comprises a coil of conductive material. (Embodiment 6) The method of any one of embodiments 1 to 5, wherein the estimated state indicates a position of the instrument tip within the tubular network including at least one of a position in three-dimensional (3D) space, an orientation in 3D space, an absolute depth of the instrument tip within the tubular network, a relative depth of the instrument tip within a branch of the tubular network, and a branch position relative to a corresponding 3D model of the tubular network. (Embodiment 7) The estimated state of the instrument tip is a joint estimated state based on the robot-based estimated state, the EM-based estimated state, and the image-based estimated state, and the joint estimated state is, for each of the robot-based estimated state, the EM-based estimated state, and the image-based estimated state, expressing the corresponding estimated state as a probability distribution containing a set of possible values; weighting the corresponding estimated state represented by the set of possible values ​​using the corresponding confidence value applied to each value of the set of possible values. 7. The method of embodiment 6, wherein the compound is produced by (Embodiment 8) The confidence value of a corresponding one of the estimated states represents a degree of confidence in the identification of the corresponding estimated state, the confidence value being determined based on the following factors: a process used to generate the corresponding estimated state; received data used to generate the corresponding estimated state; and Possible areas for the device tip to be placed within the tubular network The method of embodiment 7, based on at least one of the following: (Embodiment 9) accessing a previous estimated state of the instrument tip based on data captured at a previous time; determining at least one of the robot based estimated state, the image based estimated state, and the EM based estimated state based on the previous estimated state; 9. The method of any one of embodiments 1 to 8, further comprising: (Embodiment 10) Identifying the EM based estimation state includes: performing a registration of an EM system to a 3D model, the EM system including coordinates based on received EM data, the 3D model including coordinates based on a 3D model of the tubular network; generating the EM-based estimated state based on the performed registration; 10. The method of any one of embodiments 1 to 9, further comprising: (Embodiment 11) The method of embodiment 10, wherein the 3D model of the tubular network is generated based on a computed tomography (CT) scan of the tubular network. (Embodiment 12) Identifying the EM based estimation state includes: determining an estimate of the location and orientation of the instrument tip relative to possible branches within the tubular network; 12. The method of any one of embodiments 1 to 11, further comprising: (Embodiment 13) 13. The method of claim 12, wherein the EM-based estimated state is represented as a discrete probability distribution, each value of the probability distribution indicating the probability that the instrument tip is within a corresponding one of the possible branches. (Embodiment 14) Identifying the image based estimation state includes: determining movement of the elongated medical device within the tubular network based on the accessed image data; and generating the image-based estimated state based on the measured movements; and 14. The method of any one of embodiments 1 to 13, further comprising: (Embodiment 15) The image-based estimation state is the absolute depth of the instrument tip within the tubular network; the depth of the instrument tip relative to the corresponding branch; and Rolling of the tool tip against the tubular mesh 15. The method of claim 14, wherein the probability distribution is expressed as at least one continuous probability distribution of (Embodiment 16) Object detection is Detecting at least one object; A method according to any one of embodiments 1 to 15, further comprising generating the image-based estimation based on the at least one detected object. (Embodiment 17) 18. The method of claim 16, wherein the at least one object is represented as a two-dimensional shape. Identifying the image based estimation state includes: - mapping physically detected objects in a real image and virtual objects generated by a 3D model of the tubular network, the virtual objects corresponding to the physically detected objects; generating the image based estimate state based on the mapping information; and 18. The method of any one of embodiments 1 to 17, further comprising: (Embodiment 19) Identifying the image based estimation state includes: determining a topology of a region where the instrument tip is to be placed within the tubular network; generating an image-based estimated state based on the identified topology; and 19. The method of any one of embodiments 1 to 18, further comprising: (Embodiment 20) 20. The method of claim 19, wherein the identified topology is a division into multiple branches facing the instrument tip. (Embodiment 21) accessing a navigation path to a target location in the tubular network; determining navigation instructions to point the instrument tip to the target location based on the estimated state; providing said navigation instructions for presentation to an operator; 21. The method of any one of embodiments 1 to 20, further comprising: (Embodiment 22) A non-transitory computer-readable storage medium containing computer program instructions that, when executed by a processor, cause the processor to: accessing robotics data related to the physical manipulation of an elongated medical instrument to be inserted into a patient's vascular network; accessing image data captured by an imaging device positioned proximate a tip of the elongated medical instrument; accessing electromagnetic (EM) data captured using an EM sensor located near the instrument tip and at least one external EM sensor located outside the patient's body or EM generated outside the patient; determining a robot based estimated state of the instrument tip based on the robot data, the robot based estimated state being associated with a first confidence value; determining an image based estimated state of the instrument tip based on the image data, the image based estimated state being associated with a second confidence value; determining an EM based estimated state of the instrument tip based on the EM data, the EM based estimated state being associated with a third confidence value; determining an estimated state of the instrument tip based on the robot-based, image-based, and EM-based estimated states and confidence values; A non-transitory computer-readable storage medium for causing a program to be executed.

Claims

1. 1. A method of operating a system, the system including a processor; A method of operating the system includes: the processor displaying a model of the patient's vascular network; the processor accessing robotic data, the robotic data including insertion data describing a movement of an elongated medical instrument into the tubular network; the processor accessing sensor data captured using a sensor disposed within at least a distal portion of the elongated medical instrument; determining a robot based estimated state of the elongated medical instrument based on the insertion data of the robot data, the robot based estimated state being associated with a first confidence value; determining a sensor-based estimated state of the elongated medical instrument based on the sensor data, the sensor-based estimated state being associated with a second confidence value; determining, by the processor, a relative position of the elongated medical device relative to the displayed model of the tubular network based on the first confidence value and the second confidence value; A method of operating the system comprising: the processor displaying the determined relative position of the elongated medical instrument with respect to a displayed model of the tubular network.

2. 2. The method of claim 1, wherein the elongated medical instrument is a flexible endoscope comprising an endoscopic structure and a plurality of cables that can be retracted or the direction of the elongated medical instrument is changed, and the flexible endoscope is coupled to an instrument device manipulator (IDM) to at least one of insert, retract, and roll the flexible endoscope from or within the tubular network.

3. The robot data includes: The method of claim 1 , comprising at least one of pitch, roll, and yaw of the elongated medical instrument based on tensioning at least one of a plurality of cables of the elongated medical instrument.

4. The method of operating the system of claim 1, wherein the sensor-based estimated state indicates a position of the elongated medical instrument within the tubular network including at least one of: a position in three-dimensional (3D) space, an orientation in 3D space, an absolute depth of the elongated medical instrument within the tubular network, a relative depth of the elongated medical instrument within a branch of the tubular network, and a branch position relative to a corresponding 3D model of the tubular network.

5. The second confidence value represents a degree of confidence in the sensor-based estimated state identification, and the second confidence value is determined based on the following factors: a process used to generate the sensor-based state estimate; received data used to generate the sensor-based state estimate; or The method of claim 1 , wherein the elongated medical device is based on a possible area within the tubular network.

6. The method of claim 1 , further comprising: the processor accessing a past estimated state of the elongated medical instrument based on data captured at a past time, and the processor determining the robot-based estimated state is based on the past estimated state.

7. The processor determining the sensor-based estimated state includes: the processor performing a registration of a sensor system to a 3D model, the sensor system including coordinates based on the received sensor data, the 3D model including coordinates based on a 3D model of the tubular network; The method of claim 1 , further comprising: the processor generating the sensor-based estimated state based on the performed alignment.

8. The method of claim 7 , wherein the 3D model of the tubular network is generated based on a computed tomography (CT) scan of the tubular network.

9. Determining the sensor-based estimated state includes: The method of claim 1 , further comprising the processor determining an estimate of a position and orientation of the elongated medical instrument relative to possible branches in the tubular network.

10. The method of claim 9 , wherein the sensor-based estimated state is represented as a probability distribution, each value of the probability distribution indicating a probability that the elongated medical instrument is within a corresponding one of the possible branches.

11. The processor may further include determining an image-based estimation state, the determining of the image-based estimation state comprising: the processor accessing image data captured by an imaging device positioned proximate a tip of the elongated medical instrument; determining, by the processor, the movement of the elongated medical instrument within the tubular network based on the accessed image data; generating the image-based estimated state based on the measured motion; The method of claim 1 further comprising:

12. The image-based estimation state is the absolute depth of the elongated medical device within the tubular network; the relative depth of the elongated medical device to a corresponding bifurcation; and A roll of the elongated medical device relative to the tubular mesh.

12. The method of claim 11, wherein the at least one continuous probability distribution of

13. The processor determining the image based estimation state includes: said processor mapping physically detected objects in an actual image to virtual objects generated by a 3D model of the tubular network, said virtual objects corresponding to said physically detected objects; generating the image-based estimated state based on the mapped physically detected objects and the virtual objects; 12. A method of operating the system of claim 11, further comprising:

14. The processor may further include determining an image-based estimation state, the determining of the image-based estimation state comprising: the processor determining a topology of a region in which the elongated medical device is to be disposed within the tubular network; generating an image-based estimated state based on the identified topology; The method of claim 1 further comprising:

15. The method of claim 14 , wherein the identified topology includes a plurality of branches facing the elongated medical instrument.

16. The method of claim 1 , further comprising the processor affecting the first confidence value and the second confidence value based on movement of the patient.

17. A method of operating the system described in claim 1, wherein the first confidence value and the second confidence value are affected by the patient's breathing.

18. The method of claim 7 , further comprising the processor updating the registration due to movement of the patient.

19. commanding the processor to move the elongated medical device in the tubular network from a current location to a location deeper in the tubular network; The method of claim 1 , further comprising: the processor updating an estimated state of the robot base based on the commanded movement.

20. 1. A robotic medical system for bronchoscopy, the robotic medical system comprising: Robots and a robotically controllable bronchoscope removably coupled to the robot, the bronchoscope configured for insertion into a patient's vascular network, a distal end of the bronchoscope being articulatable, and the bronchoscope including a sensor; A display and At least one computer readable medium having instructions stored thereon; the instructions, when executed, cause one or more processors to: causing the display to render a model of the vascular network of the patient; accessing robot data, the robot data including insertion data describing a movement of the bronchoscope into the tubular network by the robot; accessing sensor data obtained from the sensor of the bronchoscope; determining a robot-based estimated state of the bronchoscope based on the insertion data, the robot-based estimated state being associated with a first confidence value; determining a sensor-based estimated state of the bronchoscope based on the sensor data, the sensor-based estimated state being associated with a second confidence value; determining a relative position of the bronchoscope within the tubular network relative to a displayed model of the tubular network based on the first confidence value and the second confidence value; displaying the relative position of the identified bronchoscope with respect to the displayed model of the tubular network; A robotic medical system that performs the following:

21. 21. The robotic medical system of claim 20, wherein the instructions, when executed, further cause the one or more processors to affect the first confidence value and / or the second confidence value based on movement of the patient.

22. 21. The robotic medical system of claim 20, wherein the instructions, when executed, further cause the one or more processors to perform a registration between a coordinate system of the sensor and a 3D model of the tubular network, and the sensor-based estimated state is further based on the registration.

23. The robotic medical system of claim 22 , wherein the instructions, when executed, further cause the one or more processors to update the registration with movement of the patient.

24. The robotic medical system of claim 20, wherein the first confidence value and / or the second confidence value are affected by the patient's breathing.

25. 1. A robotic medical system for bronchoscopy, the robotic medical system comprising: an instrument device manipulator configured to couple with a bronchoscope, the bronchoscope configured to be inserted into a patient's tubular network, a distal end of the bronchoscope being articulatable, the bronchoscope including a sensor; A display and At least one computer readable medium having instructions stored thereon; the instructions, when executed, cause one or more processors to: causing the display to render a model of the vascular network of the patient; accessing robotic data, the robotic data including insertion data describing a movement of the bronchoscope into the tubular network; accessing sensor data obtained from the sensor of the bronchoscope; determining a robot-based estimated state of the bronchoscope based on the insertion data, the robot-based estimated state being associated with a first confidence value; determining a sensor-based estimated state of the bronchoscope based on the sensor data, the sensor-based estimated state being associated with a second confidence value; determining a relative position of the bronchoscope within the tubular network relative to a displayed model of the tubular network based on the first confidence value and the second confidence value; displaying the relative position of the identified bronchoscope with respect to the displayed model of the tubular network; A robotic medical system for bronchoscopy.

Images