Combining strain-based shape sensing with catheter control

Abstract

The present invention provides a robotic system and method for navigating a luminal network that can improve strain-based shape sensing. In one aspect, the system can compare strain-based shape data with shape data determined based on robotic data (e.g., kinematic model data, torque measurements, mechanical model data, command data, etc.), and adjust the strain-based shape data as needed. Any portion of the strain-based shape data can be adjusted, weighted differently, or discarded based on the comparison. For example, data from a trusted source can indicate that the shape of an instrument exhibits or should exhibit one or more characteristics. If the system determines that any portion of the strain-based shape data is inconsistent with such characteristics, the system can adjust the portion of the strain-based shape data so that the adjusted strain-based shape data is consistent with the characteristics of the instrument.

Description

[0001] Cross-references to related patent applications

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 715,668, filed on August 7, 2018, which is hereby incorporated by reference in its entirety. Technical Field

[0003] The systems and methods disclosed herein relate to surgical robotics, and more particularly to navigation of medical devices within a tubular network of a patient's body. Background Art

[0004] Bronchoscopy is a medical procedure that allows a physician to examine the interior of a patient's lung airways, such as the bronchi and bronchioles. The lung airways carry air from the trachea or airways to the lungs. During the medical procedure, a thin, flexible, tubular tool called a bronchoscope may be inserted into the patient's mouth and down through the patient's throat into his / her lung airways, and the patient is typically anesthetized in order to relax their throat and lung cavity for surgical examination and surgery during the medical procedure.

[0005] A bronchoscope may include a light source and a small camera that allows a physician to examine a patient's trachea and airways, and a rigid tube may be used in conjunction with a bronchoscope for surgical purposes, such as when there is heavy bleeding in a patient's lungs or when a large object is blocking a patient's larynx. When a rigid tube is used, the patient is typically anesthetized. Robotic bronchoscopes offer a huge advantage when navigating through tubular networks. They are convenient to use and allow for convenient administration of treatments and biopsies even during bronchoscopy sessions.

[0006] In addition to mechanical devices or platforms (e.g., the above-mentioned robotic bronchoscope), various methods and software models can be used to assist in performing surgical procedures. For example, a computed tomography (CT) scan of a patient's lungs is often performed during the preoperative period of a surgical examination. Data from the CT scan can be used to generate a three-dimensional (3D) model of the airways of the patient's lungs, and the generated 3D model enables a physician to access visual references that may be useful during the operating process of a surgical examination.

[0007] However, even when employing medical devices (e.g., robotic bronchoscopes) and using existing methods (e.g., performing CT scans and generating 3D models), prior art techniques for navigating tubular networks remain challenging. For example, the motion estimation of a medical device (e.g., a bronchoscope tool) within a patient may be inaccurate based on the position and orientation changes of the device, and thus the position of the device may not be accurately or correctly located within the patient in real time. Inaccurate position information of such instruments may provide misleading information to physicians who use 3D models as visual references during medical procedures.

[0008] Therefore, there is a need for improved techniques for navigating within a network of tubular structures. Summary of the invention

[0009] The present invention describes a robotic system and method for navigating a luminal network that can improve strain-based shape sensing. In one aspect, the system can compare strain-based shape data with shape data determined based on robotic data (e.g., command data, force and distance data, mechanical model data, kinematic model data, etc.), and adjust the strain-based shape data as needed. Any portion of the strain-based shape data can be adjusted, weighted differently, or discarded based on the comparison. For example, data from a trusted source can indicate that the shape of an instrument exhibits or should exhibit one or more characteristics. If the system determines that any portion of the strain-based shape data is inconsistent with such characteristics, the system can adjust the portion of the strain-based shape data so that the adjusted strain-based shape data is consistent with the characteristics of the instrument.

[0010] Thus, one aspect relates to a method of navigating an instrument within an internal region of a body. The method may include: accessing robotic data about the instrument; accessing strain data from an optical fiber positioned within the instrument, the strain data indicating strain on a portion of the instrument positioned within the internal region of the body; determining shape data based on the strain data; comparing the robotic data and the shape data; adjusting the shape data based on the comparison of the robotic data and the shape data; determining an estimated state of the instrument based on the adjusted shape data; and outputting the estimated state of the instrument.

[0011] The aspects described in the above paragraphs may also include one or more of the following features in any combination: (a) wherein adjusting the shape data includes modifying at least a portion of the shape data such that determination of the estimated state of the instrument is based on the modified portion of the shape data; (b) wherein adjusting the shape data includes removing at least a portion of the shape data such that determination of the estimated state of the instrument is not based on the removed portion of the shape data; (c) wherein the method further includes accessing electromagnetic (EM) data captured using (i) an electromagnetic sensor located proximal to the distal end of the instrument and (ii) at least one external EM sensor or EM field generator located outside the body, comparing the EM data to the shape data, and determining based on the comparison of the EM data and the shape data. (d) wherein the method further comprises accessing image data captured by an imaging device located proximal to the distal end of the instrument, comparing the image data and the shape data, and further adjusting the shape data based on the comparison of the image data and the shape data; (e) wherein the strain data is generated based on a fiber Bragg grating (FBG) generated on a portion of the optical fiber; (f) wherein the shape data comprises one of a curvature value of the portion of the instrument or time history data of the portion of the instrument; (g) wherein the method further comprises adjusting the shape data based on determining that the curvature value is greater than or equal to a threshold curvature value in the robot data; (h) wherein the method further comprises adjusting the shape data based on determining that the time history data satisfies the robot data wherein the method further comprises adjusting the shape data based on a threshold time history condition in the instrument; (i) wherein the method further comprises adjusting the shape data based on a temperature change; (j) wherein the method further comprises adjusting the shape data based on a determination that the distal end of the instrument is undergoing articulation; (k) wherein the method further comprises adjusting the shape data based on a determination that a non-shape changing strain is being applied to the instrument; (l) wherein the method further comprises assigning a confidence value to the robot data corresponding to the first portion based on a determination that the first portion of the instrument comprises a distal end of the instrument, the confidence value being higher than the confidence value assigned to the shape data corresponding to the first portion; (m) wherein the method further comprises assigning a confidence value to the robot data corresponding to the first portion based on a determination that the first portion of the instrument comprises a proximal end of the instrument giving to the robot data corresponding to the first part a confidence value that is lower than the confidence value assigned to the shape data corresponding to the first part; (n) wherein the method also includes determining an estimated state of a sheath covering the instrument based on the estimated state of the instrument; (o) wherein the method also includes assigning a confidence value to the shape data based on a comparison of the shape data and additional data indicative of the shape of the sheath covering the instrument; (p) wherein the method also includes determining, based on the estimated state of the instrument, that damage to the instrument is about to occur, and controlling the instrument to avoid the damage; and (q) wherein the method also includes determining that a mismatch between the robot data and the shape data has been detected for at least a threshold amount of time, and outputting a warning indicating that the instrument may be damaged.

[0012] Another aspect relates to a method of navigating an instrument within an internal region of a body. The method may include: accessing robotic data about the instrument; accessing strain data from an optical fiber positioned within the instrument, the strain data indicating strain on a portion of the instrument positioned within the internal region of the body; determining shape data based on the strain data; comparing the robotic data and the shape data; adjusting a confidence value associated with the shape data based on the comparison of the robotic data and the shape data; determining an estimated state of the instrument based on the adjusted confidence value; and outputting the estimated state of the instrument.

[0013] The aspects described in the above paragraphs may also include one or more of the following features in any combination: (a) wherein the method further includes accessing EM data captured using (i) an electromagnetic (EM) sensor located proximal to the distal end of the instrument and (ii) at least one external EM sensor or EM field generator located outside the body, comparing the EM data to the shape data, and further adjusting the confidence value associated with the shape data based on the comparison of the EM data to the shape data; (b) wherein the method further includes accessing image data captured by an imaging device located proximal to the distal end of the instrument, comparing the image data to the shape data, and further adjusting the confidence value associated with the shape data based on the image data and shape data to adjust a confidence value associated with the shape data; (c) wherein the strain data is generated based on a fiber Bragg grating (FBG) generated on a portion of the optical fiber; (d) wherein the shape data includes one of a curvature value of the portion of the instrument or time history data of the portion of the instrument; (e) wherein the method further includes adjusting the confidence value based on determining that the curvature value is greater than or equal to a threshold curvature value in the robot data; (f) wherein the method further includes adjusting the confidence value based on determining that the time history data satisfies a threshold time history condition in the robot data; (g) wherein the method further includes adjusting the confidence value based on a temperature change. (h) adjusting the confidence value based on a determination that the distal end of the instrument is undergoing articulation; (i) wherein the method further comprises adjusting the confidence value based on a determination that a non-shape changing strain is being applied to the instrument; (j) wherein the method further comprises assigning a confidence value to the robot data corresponding to the first portion based on a determination that the first portion of the instrument comprises the distal end of the instrument, the confidence value being higher than the confidence value assigned to the shape data corresponding to the first portion; (k) wherein the method further comprises assigning a confidence value to the robot data corresponding to the first portion based on a determination that the first portion of the instrument comprises the proximal end of the instrument, the confidence value being lower than the confidence value assigned to the shape data corresponding to the first portion. (i) wherein the method further comprises determining an estimated state of a sheath covering the instrument based on the estimated state of the instrument; (ii) wherein the method further comprises adjusting the confidence value based on a comparison of the shape data and additional data indicative of the shape of the sheath covering the instrument; (iii) wherein the method further comprises determining, based on the estimated state of the instrument, that damage to the instrument is about to occur, and controlling the instrument to avoid the damage; and (iv) wherein the method further comprises determining that a mismatch between the robot data and the shape data has been detected for at least a threshold amount of time, and outputting a warning indicating that the instrument may be damaged. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.

[0015] Figure 1AAn exemplary surgical robotic system is shown according to one embodiment.

[0016] Figures 1B to 1F shows a coupled to Figure 1A Various perspective views of a robotic platform of a surgical robotic system are shown.

[0017] Figure 2 An exemplary command console for an exemplary surgical robotic system is shown according to one embodiment.

[0018] Figure 3A According to one embodiment, Figure 1A An isometric view of an exemplary independent drive mechanism of an instrument device manipulator (IDM) is shown.

[0019] Figure 3B A conceptual diagram is shown according to one embodiment, which shows how Figure 3A The strain gauges shown are independent of the drive mechanism to measure the force.

[0020] Figure 4A A top view of an exemplary endoscope is shown according to one embodiment.

[0021] Figure 4B According to one embodiment, Figure 4A An exemplary endoscope cross section of an endoscope is shown.

[0022] Figure 4C An exemplary strain-based shape sensing mechanism is shown according to one embodiment.

[0023] FIG. 4D to FIG. 4E The actual shape of an exemplary endoscope and a strain-based prediction of the endoscope are shown, according to one embodiment.

[0024] Figure 5 An exemplary schematic setup of an EM tracking system included in a surgical robotic system according to one embodiment is shown.

[0025] FIG. 6A to FIG. 6B An exemplary anatomical lumen and an exemplary 3D model of the anatomical lumen are shown according to one embodiment.

[0026] Figure 7 A computer-generated 3D model representing an anatomical space is shown according to one embodiment.

[0027] Fig. 8A A high-level overview of an example block diagram of a navigation configuration system is shown according to one embodiment.

[0028] Figure 8BA block diagram illustrating exemplary modules included in a strain-based algorithm module is shown according to one embodiment.

[0029] Figure 8C A block diagram illustrating an example of robot data stored in a robot data repository is shown according to one embodiment.

[0030] Fig. 9 An exemplary block diagram of a shape data determination module according to one embodiment is shown.

[0031] Fig.10 An exemplary block diagram of a shape data comparison module and a shape data adjustment module according to one embodiment is shown.

[0032] Fig.11 An exemplary block diagram of a shape-based state estimation module is shown according to one embodiment.

[0033] Fig.12 A flow chart illustrating an exemplary method operable by a surgical robotic system, or component thereof, for determining and adjusting shape data, according to one embodiment is shown.

[0034] Fig.13 A conceptual diagram illustrating an exemplary method operable by a surgical robotic system, or component thereof, for operating an instrument, according to one embodiment is shown.

[0035] Fig.14 A conceptual diagram illustrating an exemplary method operable by a surgical robotic system, or component thereof, for operating an instrument, according to one embodiment is shown.

[0036] Reference will now be made specifically to several embodiments, examples of which are shown in the accompanying drawings. It should be noted that similar or identical reference numerals may be used in the drawings and may represent similar or identical functions whenever feasible. The accompanying drawings depict embodiments of the systems (or methods) described herein for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods shown herein may be employed without departing from the principles described herein. DETAILED DESCRIPTION

[0037] I. Surgical Robotic System

[0038] Figure 1A An exemplary surgical robotic system 100 is shown according to one embodiment. The surgical robotic system 100 includes a base 101 coupled to one or more robotic arms, such as a robotic arm 102. The base 101 is communicatively coupled to the command console referenced in Section II. Figure 2The base 101 can be positioned so that the robotic arm 102 can enter to perform surgery on a patient, and a user, such as a physician, can control the surgical robotic system 100 from the comfort of the command console. In some embodiments, the base 101 can be coupled to a surgical table or bed for supporting a patient. Although for clarity, the base 101 is not shown in FIG. Figure 1A 1, but the base 101 may include subsystems such as control electronics, pneumatics, power sources, light sources, etc. The robotic arm 102 includes a plurality of arm segments 110 coupled at joints 111, which provide the robotic arm 102 with multiple degrees of freedom, for example, seven degrees of freedom corresponding to seven arm segments. The base 101 may contain a power source 112, pneumatic pressure devices 113, and control and sensor electronics 114—including components such as a central processing unit, data bus, control circuits, and memory—and associated actuators such as motors to move the robotic arm 102. The electronics 114 in the base 101 may also process and transmit control signals communicated from the command console.

[0039] In some embodiments, the base 101 includes wheels 115 to transport the surgical robotic system 100. The mobility of the surgical robotic system 100 helps accommodate space constraints in a surgical operating room and facilitates proper positioning and movement of surgical equipment. In addition, the mobility allows the robotic arm 102 to be configured so that the robotic arm 102 does not interfere with the patient, physician, anesthesiologist, or any other equipment. During a procedure, a user can control the robotic arm 102 using a control device such as a command console.

[0040] In some embodiments, the robot arm 102 includes a mounting joint that uses a combination of a brake and a counter-balancing device to maintain the position of the robot arm 102. The counter-balancing device may include a gas spring or a coil spring. The brake (e.g., a fail-safe brake) may include mechanical components and / or electronic components. In addition, the robot arm 102 may be a gravity-assisted passive support type robot arm.

[0041] Each robotic arm 102 can be coupled to an instrument device manipulator (IDM) 117 using a mechanism converter interface (MCI) 116. The IDM 117 can be removed and replaced with a different type of IDM, for example, a first type of IDM manipulates an endoscope and a second type of IDM manipulates a laparoscope. The MCI 116 includes a connector for transmitting pneumatic pressure, electrical power, electrical signals, and optical signals from the robotic arm 102 to the IDM 117. The MCI 116 can be a set screw or a baseboard connector. The IDM 117 manipulates surgical instruments such as an endoscope 118 using technologies including direct drive, harmonic drive, gear drive, belt and pulley, magnetic drive, etc. The MCI 116 is interchangeable based on the type of IDM 117 and can be customized for a certain type of surgical procedure. The robotic arm 102 may include joint-level torque sensing and a wrist at the distal end, such as a KUKA LBR5 robot arm.

[0042] Endoscope 118 is a tubular, flexible surgical instrument that is inserted into a patient's anatomy to capture images of the anatomy (e.g., body tissue). Specifically, endoscope 118 includes one or more imaging devices (e.g., cameras or other types of optical sensors) that capture images. The imaging devices may include one or more optical components, such as optical fibers, fiber arrays, or lenses. The optical components move with the distal end of endoscope 118 such that movement of the distal end of endoscope 118 results in changes in the image captured by the imaging devices. Endoscope 118 is referenced in Section IV. Endoscopes. FIG. 3A to FIG. 4B Further description.

[0043] The robotic arm 102 of the surgical robotic system 100 manipulates the endoscope 118 using an elongated motion member. The elongated motion member may include a pull wire (also known as a push-pull wire), a cable, a fiber, or a flexible shaft. For example, the robotic arm 102 actuates a plurality of pull wires coupled to the endoscope 118 to deflect the distal end of the endoscope 118. The pull wires may include both metallic and non-metallic materials, such as stainless steel, Kevlar, tungsten, carbon fiber, etc. The endoscope 118 may exhibit nonlinear behavior in response to the force applied by the elongated motion member. The nonlinear behavior may be based on the stiffness and compressibility of the endoscope 118, and the variability of the slack or stiffness between different elongated motion members.

[0044] Figures 1B to 1F Various perspective views of a surgical robotic system 100 coupled to a robotic platform 150 (or surgical bed) are shown according to various embodiments. Specifically, Figure 1B A side view of the surgical robotic system 100 is shown with the robotic arm 102 manipulating the endoscope 118 to insert the endoscope into a patient, with the patient lying on the robotic platform 150 . Figure 1CA top view of the surgical robotic system 100 and the robotic platform 150 is shown with the endoscope 118 manipulated by the robotic arm being inserted into a patient. Figure 1D A perspective view of the surgical robotic system 100 and the robotic platform 150 is shown, with the endoscope 118 being controlled to be horizontally positioned parallel to the robotic platform. Figure 1E Another perspective view of the surgical robotic system 100 and the robotic platform 150 is shown, and the endoscope 118 is controlled to be positioned perpendicularly relative to the robotic platform. Figure 1E In FIG. 1 , the angle between the horizontal surface of the robot platform 150 and the endoscope 118 is 75 degrees. Figure 1F Shows Figure 1E A perspective view of the surgical robotic system 100 and the robotic platform 150 is shown, and in more detail, the angle between the endoscope 118 and the virtual line 160 is 90 degrees, which connects one end 180 of the endoscope 118 and the robotic arm 102 located relatively far away from the robotic platform.

[0045] II. Command Console

[0046] Figure 2 An exemplary command console 200 for an exemplary surgical robotic system 100 is shown according to one embodiment. The command console 200 includes a console base 201, a display module 202, such as a monitor, and control modules, such as a keyboard 203 and a joystick 204. In some embodiments, one or more of the command console 200 functions may be integrated into the base 101 of the surgical robotic system 100 or another system communicatively coupled to the surgical robotic system 100. A user 205, such as a physician, uses the command console 200 to remotely control the surgical robotic system 100 from an ergonomic position.

[0047] The console base 201 may include a central processing unit, a memory unit, a data bus, and associated data communication ports that are responsible for interpreting and processing signals such as camera images and tracking sensor data, such as from the endoscope 118 shown in FIG. 1. In some embodiments, both the console base 201 and the base 101 perform signal processing to achieve load balancing. The console base 201 may also process commands and instructions provided by the user 205 through the control modules 203 and 204. In addition to Figure 2 In addition to the keyboard 203 and joystick 204 shown, the control module may also include other devices such as a computer mouse, a touch pad, a trackball, a control pad, a video game controller, and sensors (e.g., motion sensors or cameras) that capture hand and finger gestures.

[0048] The user 205 may use the command console 200 to control a surgical instrument, such as the endoscope 118, in a speed mode or a position control mode. In the speed mode, the user 205 uses the control module to directly control the pitch and yaw motion of the distal end of the endoscope 118 based on direct manual control. For example, the motion on the joystick 204 may be mapped to the yaw and pitch motion in the distal end of the endoscope 118. The joystick 204 may provide tactile feedback to the user 205. For example, the joystick 204 vibrates to indicate that the endoscope 118 cannot be further translated or rotated in a certain direction. The command console 200 may also provide visual feedback (e.g., a pop-up message) and / or auditory feedback (e.g., a beep) to indicate that the endoscope 118 has reached a maximum translation or rotation.

[0049] In the position control mode, the command console 200 uses a three-dimensional (3D) map of the patient and a predetermined computer model of the patient to control a surgical instrument, such as the endoscope 118. The command console 200 provides control signals to the robotic arm 102 of the surgical robotic system 100 to maneuver the endoscope 118 to a target location. Because of its reliance on a 3D map, the position control mode requires accurate mapping of the patient's anatomy.

[0050] In some embodiments, the user 205 can manually manipulate the robotic arm 102 of the surgical robotic system 100 without using the command console 200. During setup in a surgical operating room, the user 205 can move the robotic arm 102, endoscope 118, and other surgical equipment to enter a patient's body. The surgical robotic system 100 can rely on force feedback and inertial control from the user 205 to determine the appropriate configuration of the robotic arm 102 and equipment.

[0051] The display module 202 may include an electronic monitor, a virtual reality viewing device such as goggles or glasses, and / or other devices of a display device. In some embodiments, the display module 202 is integrated with the control module, for example, as a tablet device with a touch screen. In addition, the user 205 can use the integrated display module 202 and control module to view data and input commands to the surgical robotic system 100.

[0052] The display module 202 can display the 3D image using a stereoscopic device such as a visor or goggles. The 3D image provides an "inside view" (i.e., an endoscopic view), which is a computer 3D model showing the patient's anatomical structure. The "inside view" provides a virtual environment of the patient's interior and the expected position of the endoscope 118 in the patient's body. The user 205 compares the "inside view" model with the actual image captured by the camera to help mentally orient and confirm that the endoscope 118 is in the correct - or approximately correct - position in the patient's body. The "inside view" provides information about the anatomical structure around the distal end of the endoscope 118, such as the shape of the patient's small intestine or colon. The display module 202 can simultaneously display the 3D model and the computed tomography (CT) scan of the anatomical structure around the distal end of the endoscope 118. In addition, the display module 202 can overlay the navigation path of the endoscope 118 that has been determined on the 3D model and the scan / image (e.g., CT scan) generated based on the preoperative model data.

[0053] In some embodiments, a model of the endoscope 118 is displayed along with the 3D model to help indicate the status of the surgical procedure. For example, a CT scan identifies a lesion in an anatomical structure that may require a biopsy. During operation, the display module 202 may show a reference image captured by the endoscope 118 corresponding to the current position of the endoscope 118. The display module 202 may automatically display different views of the model of the endoscope 118 based on user settings and a particular surgical procedure. For example, the display module 202 shows a top fluoroscopic view of the endoscope 118 as the endoscope 118 approaches the operating area of ​​the patient during a navigation step.

[0054] III. Instrument manipulator

[0055] Figure 3A An isometric view of an exemplary independent drive mechanism of the IDM 117 shown in FIG. 1 according to one embodiment is shown. The independent drive mechanism can tighten or loosen the pull wires 321, 322, 323, and 324 of the endoscope (e.g., independently of each other) by rotating the output shafts 305, 306, 307, and 308 of the IDM 117, respectively. Just as the output shafts 305, 306, 307, and 308 transmit force downward to the pull wires 321, 322, 323, and 324, respectively, through angular motion, the pull wires 321, 322, 323, and 324 transmit force back to the output shafts. The IDM 117 and / or the surgical robot system 100 can measure the transmitted force using sensors (e.g., strain gauges described further below).

[0056] Figure 3B A conceptual diagram is shown according to one embodiment, which shows how Figure 3AThe force 331 can be directed away from the output shaft 305 of the motor mount 333, which is coupled to the motor 337. Thus, the force 331 causes a horizontal displacement of the motor mount 333. Additionally, the strain gauge 334, which is horizontally coupled to the motor mount 333, experiences strain in the direction of the force 331. The strain can be measured as the ratio of the horizontal displacement of the tip 335 of the strain gauge 334 to the total horizontal width 336 of the strain gauge 334.

[0057] In some embodiments, the IDM 117 includes additional sensors, such as an inclinometer or an accelerometer, to determine the orientation of the IDM 117. Based on the measurements from the additional sensors and / or the strain gauge 334, the surgical robotic system 100 can calibrate the readings from the strain gauge 334 to account for gravity loading effects. For example, if the IDM 117 is oriented on the horizontal side of the IDM 117, the weight of certain components of the IDM 117 can cause strain on the motor mount 333. Therefore, without accounting for gravity loading effects, the strain gauge 334 can measure strain that is not caused by strain on the output shaft.

[0058] IV. Endoscopy

[0059] IV.A. Endoscope top view

[0060] Figure 4A A top view of an exemplary endoscope 118 according to one embodiment is shown. The endoscope 118 includes a leader 415 tubular member nested or partially nested within and longitudinally aligned with a tubular member of a sheath 411. The sheath 411 includes a proximal sheath segment 412 and a distal sheath segment 413. The leader 415 has a smaller outer diameter than the sheath 411 and includes a proximal leader segment 416 and a distal leader segment 417. The sheath base 414 and the leader base 418 actuate the distal sheath portion 413 and the distal leader segment 417, respectively, based on, for example, a control signal from a user of the surgical robotic system 100. The sheath base 414 and the leader base 418 are, for example, a portion of the IDM 117 shown in FIG. 1.

[0061] Both the sheath base 414 and the guide base 418 include a drive mechanism (eg, as described in Section III. Instrument Device Manipulators). FIG. 3A to FIG. 3B415 ) to control a pull wire coupled to the sheath 411 and the leader 415. For example, the sheath base 414 generates a tensile load on the pull wire coupled to the sheath 411 to deflect the distal sheath segment 413. Similarly, the leader base 418 generates a tensile load on the pull wire coupled to the leader 415 to deflect the distal leader segment 417. Both the sheath base 414 and the leader base 418 may also include a coupling for routing pneumatic pressure, electricity, electrical signals, or optical signals from the IDM to the sheath 411 and the leader 415, respectively. The pull wire may include a steel coil along the length of the pull wire within the sheath 411 or the leader 415 that transmits axial compression back to the load origin, such as the sheath base 414 or the leader base 418, respectively.

[0062] Because the pull wires coupled to the sheath 411 and the guide 415 provide multiple degrees of freedom, the endoscope 118 can easily navigate the patient's anatomy. For example, four or more pull wires can be used in the sheath 411 and / or the guide 415, thereby providing eight or more degrees of freedom. In other embodiments, up to three pull wires can be used, thereby providing up to six degrees of freedom. The sheath 411 and the guide 415 can rotate up to 360 degrees along the longitudinal axis 406, thereby providing more degrees of motion. The combination of rotation angles and multiple degrees of freedom provides the user of the surgical robotic system 100 with user-friendly and autonomous control of the endoscope 118. Although Figure 4A Although not shown, endoscope 118 may include one or more optical fibers for sensing shapes in one or more portions of endoscope 118. For example, Figure 4B As shown, the optical fiber may be included in a leader portion of the endoscope 118. Alternatively or additionally, the optical fiber may be included in a sheath portion of the endoscope 118. As will be explained in more detail below, information from the optical fiber may be used in conjunction with information from other input sources (such as other input sensors, modeling data, known properties and characteristics of the endoscope, etc.) to enhance the performance of the navigation system, catheter control, etc.

[0063] IV.B. Endoscopic Cross-Sectional View

[0064] Figure 4B According to one embodiment, Figure 4A An exemplary endoscope cross-section 430 of the endoscope 118 is shown. Figure 4B, an endoscope cross section 430 includes an illumination source 432, an electromagnetic (EM) coil 434, and a shape sensing fiber 436. The illumination source 432 provides light to illuminate an interior portion of an anatomical space. The light provided may allow an imaging device disposed at the end of the endoscope 118 to record an image of the space, which may then be transmitted to a computer system such as a command console 200 for processing as described herein. The EM coil 434 may be used with an EM tracking system to detect the position and orientation of the end of the endoscope when the end of the endoscope 118 is disposed within an anatomical system. In some embodiments, the coil may be angled to provide sensitivity to the EM field along different axes, thereby enabling the ability to measure all six degrees of freedom: three positions and three angles. In other embodiments, only a single coil may be disposed within the end of the endoscope 118, wherein its axis is oriented along the endoscopic axis of the endoscope 118; due to the rotational symmetry of such a system, it is insensitive to rolling around its axis, so that only five degrees of freedom may be detected in this case. The endoscope cross section 430 also includes a working channel 438 through which a surgical instrument, such as a biopsy needle, can be inserted along the endoscope shaft, thereby allowing access to the area near the end of the endoscope. As used herein, "instrument" may refer to surgical instruments, medical instruments, and any other instruments or devices that can navigate in a luminal network.

[0065] Although the illustrated embodiment is disclosed as including an illumination source 432 and an EM coil 434 and corresponding imaging devices and EM tracking systems, it is contemplated that modified embodiments of the endoscope described herein may lack one or more of such features. Additionally, although the shape sensing fiber 436 is described as being integrated into the endoscope 118, in other embodiments, any of the one or more shape sensing fibers 436 may alternatively be a removable working channel device that can be inserted into the working channel 438 and removed from the working channel 438 after shape sensing is performed. In other embodiments, the shape sensing fiber 436 may be mounted on the exterior of the endoscope 118.

[0066] IV.C. Shape Sensing Fiber

[0067] Figure 4CA system 450 is shown having a shape detector 452 that can be used to generate and detect light for determining the shape of an instrument, an endoscope 454, and an optical fiber 456. The optical fiber 456 can include one or more segments of a fiber Bragg grating (FBG) 458 that reflects certain wavelengths of light while transmitting other wavelengths of light. The grating 458 can include a series of refractive index modulations to produce spatial periodicity in the refractive index. During the manufacture of the grating 458, the modulations can be spaced apart by known distances, thereby causing reflection of a known wavelength band. The shape detector 452 can transmit light through the optical fiber 456 and receive light reflected from the optical fiber 456. The shape detector 452 can also generate reflectance spectral data based on the wavelength of light reflected by the grating 458.

[0068] like Figure 4C As shown, a single optical fiber may include multiple sets of fiber Bragg gratings. Endoscope 454 may include multiple optical fibers, and shape detector 452 may detect and analyze signals from more than one optical fiber. One or more optical fibers may include Figure 4A Leading element 415, Figure 4A The shape detector 452 may be operably coupled to a controller configured to determine the geometry or configuration of the optical fiber 456 and, therefore, the geometry or configuration of at least a portion of the endoscope 454 (or other elongated instrument, such as a catheter, etc.) based on spectral analysis of the detected reflected light signal.

[0069] A controller (e.g., surgical robotic system 500) within or in communication with shape detector 452 may analyze the reflected spectral data to generate position and orientation data of endoscope 454 in two-dimensional or three-dimensional space. Specifically, when endoscope 454 is bent, optical fiber 456 positioned inside is also bent, which induces strain on optical fiber 456. When strain is induced on optical fiber 456, the pitch of the modulation will change according to the amount of strain on optical fiber 456. To measure strain, light is sent down optical fiber 456 and the characteristics of the returning light are measured. For example, grating 458 may produce a reflected wavelength that varies with strain on optical fiber 456 (and other factors such as temperature). Based on the specific wavelength of light reflected by grating 458, the system may determine the amount of strain on optical fiber 456, and further predict the shape of optical fiber 456 based on the amount of strain (e.g., based on how the strain characteristics of a "straight" endoscope may differ from the strain characteristics of a "bent" endoscope). Thus, the system can determine, for example, the extent to which the endoscope 454 has been bent in one or more directions (e.g., in response to commands from the surgical robotic system 500) by identifying differences in the reflectance spectral data.

[0070] In some embodiments, optical fiber 456 includes multiple cores in a single cladding. In such embodiments, each core can be operated as a separate optical fiber with sufficient distance and a cladding separating the cores so that the light in each core does not significantly interact with the light carried in other cores. In other embodiments, the number of cores can vary, or each core can be included in a separate optical fiber. When strain and shape analysis is applied to multi-core optical fibers, the bending of optical fiber 456 can cause strain on the core, which can be measured by monitoring the wavelength shift in each core. By making two or more cores eccentrically arranged in optical fiber 456, the bending of optical fiber can cause different strains on each core. These strains vary with the local curvature of the fiber. For example, if the region of the core containing grating 458 is located at the point where optical fiber 456 is bent, it can be used to determine the bending amount at those points. These data are combined with the known spacing of grating 458 and can be used to reconstruct the shape of optical fiber 456.

[0071] Fiber optics are suitable for data acquisition within a patient because line of sight to the shape sensing fiber is not required. Various systems and methods for monitoring the shape and relative position of an optical fiber in three dimensions are described in U.S. Patent Application Publication No. 2006 / 0013523, filed on July 13, 2005, entitled "Fiber Optic Position and Shape Sensing Device and Method Relating Thereto," and U.S. Patent No. 6,389,187, filed on June 17, 1998, entitled "Optical Fiber Bend Sensor," both of which are incorporated herein by reference in their entirety.

[0072] Although the illustrated embodiments use fibers with Bragg gratings, in modified variations, the optical fiber may include slight imperfections that cause variations in the refractive index along the fiber core. These variations may result in a small amount of backscattering known as Rayleigh scattering. Changes in strain or temperature of the optical fiber cause changes in the effective length of the optical fiber. This change in effective length results in a variation or change in the spatial location of the Rayleigh scattering points. Cross-correlation techniques can measure this variation in Rayleigh scattering and can extract information about strain. These techniques may include using optical frequency domain reflectometry techniques in a manner very similar to that associated with low reflectivity fiber gratings.

[0073] Methods and apparatus for calculating birefringence in an optical fiber based on Rayleigh scattering and apparatus and methods for measuring strain in an optical fiber using a spectral shift of Rayleigh scattering can be found in PCT Publication WO 2006 / 099056 filed on March 9, 2006 and U.S. Patent 6,545,760 filed on March 24, 2000, both of which are incorporated herein by reference in their entirety. Birefringence can be used to measure axial strain and / or temperature in a waveguide.

[0074] IV.D. Improving Strain-Based Shape Data

[0075] Strain-based shape sensing can allow the shape of an endoscope or other instrument to be reconstructed by measuring the strain along an optical fiber running within the instrument. The measurement of strain captures the spatiotemporal variation of the reflection of light on the gratings within the optical fiber. The distance between each grating affects the reflection and can therefore be used to measure the strain at a precise location along the optical fiber (or instrument). However, in some cases, strain-based shape sensing can be negatively affected by noise. In such cases, it may be difficult to distinguish between actual strain changes and false strain changes.

[0076] The improved strain-based shape sensing system may utilize other data available to the system (e.g., robotic data, image data, EM data, etc.) to improve the accuracy (or adjust the confidence) of its strain-based shape sensing or state estimates determined based on such strain-based shape sensing. Alternatively or in addition, the improved strain-based shape sensing system may utilize shape data based on its strain-based shape sensing determination to improve the accuracy (or adjust the confidence) of its other data (e.g., robotic data, image data, EM data, etc.) or state estimates determined based on such data.

[0077] FIG. 4D to FIG. 4EIt is shown how the system can utilize information available to the system to improve, adjust, or weight its strain-based shape sensing. The system can access data available to the system, such as robotics data (e.g., command data, force and distance data, mechanical model data, kinematic model data, etc.), and determine certain characteristics about the shape of an instrument (or a specific portion thereof) navigated within a patient based on such data. Such characteristics can include curvature information (e.g., the maximum curvature that the instrument can assume, or a range of acceptable curvature values ​​given the current force and distance data indicated by the robotics data), movement information (e.g., the maximum speed that the instrument can move, or a range of acceptable speed values ​​given the current force and distance data indicated by the robotics data), sheath information (e.g., the current shape of a sheath covering one or more portions of the instrument), etc. Upon determining that the strain-based shape prediction does not satisfy one or more constraints indicated by these characteristics determined based on the robotics data, the system can adjust the strain-based shape prediction so that the adjusted strain-based shape prediction satisfies these constraints, reduce the confidence or weight associated with a particular strain-based shape prediction, or ignore the strain-based shape prediction.

[0078] Figure 4D The actual shape 472 of the endoscope 118, the shape prediction 473 based on the robotic data, and the strain-based shape prediction 474 of the endoscope 118 are shown. The actual shape 472 exhibits an actual curvature 476, while the shape prediction 473 based on the robotic data exhibits a predicted curvature 477, and the shape prediction 474 based on the strain exhibits a predicted curvature 478. Figure 4D In an example of , the system may determine one or more conditions that the endoscope 118 is expected to satisfy based on the robotic data (e.g., a curvature value at a given point along the endoscope 118 should be within a predetermined range of values, or should be within a range of values ​​determined based on the tension on the pullwire and / or the distance the pullwire has been actuated). Upon determining that a portion of the strain-based shape prediction 474 does not satisfy such conditions (e.g., by indicating a predicted curvature value outside of an expected range of curvature values ​​determined based on the robotic data corresponding to the endoscope 118), the system may adjust the strain-based shape prediction 474 so that the portion of the strain-based shape prediction 474 satisfies these conditions (e.g., so that the shape data no longer indicates a predicted curvature value outside of the expected range of curvature values), reduce the confidence or weight associated with the portion of the strain-based shape prediction 474, or ignore the portion of the strain-based shape prediction 474 (e.g., avoid using the portion of the strain-based shape prediction 474 when estimating the current state of the endoscope 118). For example, as Figure 4DAs shown, the system may determine a robot data-based shape prediction 473 that exhibits a predicted curvature 477 at a given point based on robot data (e.g., tension and distance). The system may compare the predicted curvature 477 to a predicted curvature 478 exhibited by a strain-based shape prediction 474. Upon determining that the predicted curvature 478 is different from the predicted curvature 477, the system may adjust the predicted curvature 487 to be equal to the predicted curvature 477. Alternatively, upon determining that the predicted curvature 478 is not within a given threshold range (e.g., ±10%) from the predicted curvature 477, the system may adjust the predicted curvature 487 to be within the threshold range (e.g., if the predicted curvature 478 is outside the range, it is set to the upper limit of the threshold range, and if the predicted curvature 478 does not reach the range, it is set to the lower limit of the threshold range). Additionally or alternatively, the system may reduce the confidence value associated with the strain-based shape prediction 474 based on determining that the predicted curvature 478 is different from the predicted curvature 477 (or the predicted curvature 478 is not within a given threshold range), and / or increase the confidence value associated with the strain-based shape prediction 474 based on determining that the predicted curvature 478 is equal to the predicted curvature 477 (or the predicted curvature 478 is within a given threshold range).

[0079] Figure 4E 482 and 484 of the shape prediction based on the robot data of the endoscope 118 are shown. The shape prediction based on the robot data 482 shows the predicted movement 486, while the shape prediction based on the strain 484 shows the predicted movement 488. Figure 4D As described, the system may determine one or more conditions that the endoscope 118 is expected to satisfy based on the robotics data. For example, based on the robotics data, the system may determine that the speed at which the endoscope 118 should move is within a certain range of values. Upon determining that a portion of the strain-based shape prediction 484 does not satisfy such conditions, the system may adjust the strain-based shape prediction 484 so that the portion of the strain-based shape prediction 484 satisfies these conditions (e.g., so that the shape data no longer indicates a predicted speed value outside of the expected range of speed values), reduce the confidence or weight associated with the portion of the strain-based shape prediction 484, or ignore the portion of the strain-based shape prediction 484 (e.g., avoid using the portion of the strain-based shape prediction 484 when estimating the current state of the endoscope 118). For example, as Figure 4EAs shown, the system may determine a robot data-based shape prediction 482 that exhibits a predicted movement 486 based on the robot data (e.g., tension and / or distance over time). The system may compare the predicted movement 486 with a predicted movement 488 exhibited by the strain-based shape prediction 484. Upon determining that the predicted movement 488 is different from the predicted movement 486, the system may adjust the predicted movement 488 to be the same as the predicted movement 486. Alternatively, upon determining that the predicted movement 488 is not within a given threshold range (e.g., the movement speed is within ±10% of the movement speed of the predicted movement 486), the system may adjust the predicted movement 488 to be within the threshold range (e.g., if the predicted movement speed of the strain-based shape prediction 484 exceeds the movement speed range, it is set to the upper limit of the threshold movement speed range, and if the predicted movement speed of the strain-based shape prediction 484 does not reach the movement speed range, it is set to the lower limit of the threshold movement speed range). Additionally or alternatively, the system may reduce the confidence value associated with strain-based shape prediction 484 based on determining that predicted movement 488 is different from predicted movement 486 (or predicted movement 488 is not within a given threshold), and / or increase the confidence value associated with strain-based shape prediction 484 based on determining that predicted movement 488 is the same as predicted movement 486 (or predicted movement 488 is within a given threshold).

[0080] The following refers to Figures 8 to Fig.11 The process of acquiring strain data and other data (some or all of which may be used to improve strain-based shape data) and determining state estimates is described in more detail.

[0081] V. Registration transformation from EM system to 3D model

[0082] Schematic setup of the VAEM tracking system

[0083] In certain embodiments, an EM tracking system may be used in conjunction with the systems described herein. Figure 5 An exemplary schematic arrangement of such an EM tracking system 505 that may be included in a surgical robotic system 500 according to one embodiment is shown. Figure 5 In the EM tracking system 505, a plurality of robotic components (e.g., a window field generator, a reference sensor, as described below) are included. The surgical robotic system 500 includes a surgical bed 511 for holding a patient's body. Below the bed 511 is a window field generator (WFG) 512, which is configured to sequentially activate a set of EM coils (e.g., Figure 4B EM coil 434 is shown. WFG 512 generates an alternating current (AC) magnetic field within a large volume; for example, in some cases, it may form an AC field in a volume of approximately 0.5 m x 0.5 m x 0.5 m.

[0084] Additional fields may be applied by additional field generators to help track instruments in the body. For example, a planar field generator (PFG) may be attached to a system arm adjacent to the patient and oriented to provide an EM field at an angle. Reference sensors 513 may be placed on the patient's body to provide a local EM field, thereby further increasing tracking accuracy. Each of the reference sensors 513 may be attached to a command module 515 via a cable 514. Cable 514 is connected to the command module 515 and provides power via an interface unit 516, which handles communication with its corresponding device. Interface unit 516 is coupled to a system control unit (SCU) 517, which acts as an overall interface controller for the various entities described above. SCU 517 also drives field generators (e.g., WFG 512), and collects sensor data from interface unit 516, from which the position and orientation of the sensor in the body is calculated. SCU 517 may be coupled to a personal computer (PC) 518 to allow user access and control.

[0085] The command module 515 is also connected to various IDMs 519 coupled to the surgical robotic system 500, as described herein. The IDMs 519 are typically coupled to a single surgical robotic system (e.g., the surgical robotic system 500) and are used to control and receive data from its corresponding connected robotic components; such as robotic endoscopic tools or robotic arms. As described above, as an example, the IDM 519 is coupled to an endoscopic tool (not shown here) of the surgical robotic system 500.

[0086] The command module 515 receives data transmitted from the endoscopic tool. The type of data received depends on the corresponding type of the attached instrument. For example, exemplary received data includes sensor data (e.g., image data, EM data), robot data (e.g., command data, force and distance data, mechanical model data, kinematic model data, etc.), control data and / or video data. In order to better process the video data, the field programmable gate array (FPGA) 520 can be configured to handle image processing. Comparing the data obtained from various sensors, devices and field generators allows the SCU 517 to accurately track the movement of different components of the surgical robot system 500, as well as, for example, the position and orientation of these components.

[0087] In order to track the sensor through the patient's anatomy, the EM tracking system 505 may require a process known as "registration," in which the system finds a geometric transformation that aligns a single object between different coordinate systems. For example, a particular anatomical site on a patient has two different representations in 3D model coordinates and EM sensor coordinates. In order to be able to establish consistency and a common language between these two different coordinate systems, the EM tracking system 505 needs to find a transformation that relates the two representations (i.e., registration). For example, the position of an EM tracker relative to the position of an EM field generator can be mapped to a 3D coordinate system to isolate the corresponding position in the 3D model.

[0088] VB3D model representation

[0089] FIG. 6A to FIG. 6B An exemplary anatomical lumen 600 and an exemplary 3D model 620 of the anatomical lumen are shown according to one embodiment. More specifically, FIG. 6A to FIG. 6B The centerline coordinates, diameter measurements, and the relationship of the anatomical space between the actual anatomical lumen 600 and its 3D model 620 are shown. Fig. 6A 6, 601, 602, 603, 604, 605, and 606, where each centerline coordinate is approximately proximate to the center of a tomographic scan slice of the lumen. The centerline coordinates are connected and visualized by centerline 607. The volume of the lumen can be further visualized by measuring the diameter of the lumen at each centerline coordinate, for example, diameters 608, 609, 610, 611, 612, and 613 represent measurements of the anatomical lumen 600 corresponding to coordinates 601, 602, 603, 604, 605, and 606.

[0090] Figure 6B According to one embodiment, Fig. 6A An exemplary 3D model 620 of an anatomical lumen 600 is shown. Figure 6B , the anatomical lumen 600 is visualized in 3D space by first locating centerline coordinates 601, 602, 603, 604, 605, and 606 in 3D space based on centerline 607. For example, at each centerline coordinate, the lumen diameter is visualized as a 2D circular space (e.g., 2D circular space 630) having diameters 608, 609, 610, 611, 612, and 613. By connecting these 2D circular spaces to form a 3D space, the anatomical lumen 600 is approximated and visualized as a 3D model 620. A more accurate approximation can be determined by increasing the resolution of the centerline coordinates and measurements, that is, increasing the density of centerline coordinates and measurements for a given lumen or subsegment. The centerline coordinates may also include markers to indicate points of interest to the physician, including ablation lesions.

[0091] In some embodiments, the preoperative software package is also used to analyze and derive a navigation path based on the generated 3D model of the anatomical space. For example, the software package can derive the shortest navigation path leading to a single ablation lesion (marked by centerline coordinates) or to several ablation lesions. The navigation path can be presented to the operator in two or three dimensions during surgery based on the operator's preference. In certain specific implementations, the navigation path (or a portion thereof) can be selected by the operator before surgery. Path selection can include identifying one or more target locations (also referred to as "targets") within the patient's anatomical structure.

[0092] Figure 7 A computer generated 3D model 700 representing an anatomical space is shown according to one embodiment. FIG. 6A to FIG. 6B As discussed in the 3D model 700, a 3D model 700 can be generated using a centerline 701 obtained by viewing a CT scan generated preoperatively. In some embodiments, computer software can map a navigation path 702 within the tubular network to access a surgical site 703 (or other target) within the 3D model 700. In some embodiments, the surgical site 703 can be connected to a separate centerline coordinate 704, which allows a computer algorithm to topologically search the centerline coordinates of the 3D model 700 for an optimal path 702 within the tubular network. In certain embodiments, the topological search of the path 702 can be constrained by certain operator-selected parameters, such as the location of one or more targets, one or more waypoints, etc.

[0093] In some embodiments, the distal end of the endoscopic tool within the patient's anatomy is tracked, and the tracked position of the endoscopic tool within the patient's anatomy is mapped and placed within the computer model, which enhances the navigation capabilities of the tubular network. To track the distal working end of the endoscopic tool, i.e., the position and orientation of the working end, a variety of methods can be used alone or in combination.

[0094] In a sensor-based approach for positioning, a sensor such as an EM tracker may be coupled to the distal working end of an endoscopic tool to provide a real-time indication of the progress of the endoscopic tool. In EM-based tracking, an EM tracker embedded in an endoscopic tool measures changes in the electromagnetic field formed by one or more EM transmitters. The transmitter (or field generator) may be placed near the patient (e.g., as part of a surgical bed) to form a low-intensity magnetic field. This induces a small current in the sensor coil of the EM tracker that is related to the distance and angle between the sensor and the generator. The electrical signal may then be digitized by an interface unit (on-chip or PCB) and sent back to the system vehicle via a cable / wiring and then to the command module. The data may then be processed to interpret the current data and calculate the precise position and orientation of the sensor relative to the transmitter. Multiple sensors may be used at different locations in the endoscopic tool, such as in the lead and sheath, in order to calculate the individual positions of these components. Thus, based on readings from the artificially generated EM field, the EM tracker may detect changes in field strength as it moves through the patient's anatomy.

[0095] VI. Navigation Configuration System

[0096] VI.A. High-Level Overview of the Navigation Configuration System

[0097] Fig. 8A An exemplary block diagram of a navigation configuration system 900 according to one embodiment is shown. Fig. 8A In the embodiment, the navigation configuration system 900 includes a plurality of input data repositories, a navigation module 905 receiving various types of input data from the plurality of input data repositories, and an output navigation data repository 990 receiving output navigation data from the navigation module 905. Fig. 8A The block diagram of the navigation configuration system 900 shown is merely an example, and in alternative embodiments not shown, the navigation configuration system 900 may include different and / or additional elements, or may not include any elements. Fig. 8A Likewise, the functions performed by the various elements of the navigation configuration system 900 may vary according to different implementations. The navigation configuration system 900 may be similar to the navigation system described in U.S. Patent No. 9,727,963, issued on August 8, 2017, the entirety of which is incorporated herein by reference.

[0098] As used herein, input data refers to raw or processed data collected from an input device (e.g., a command module, an optical sensor, an EM sensor, an IDM) used to generate estimated state information of the endoscope 118 (or other instrument) and output navigation data. Multiple input data repositories 901-941 may include a strain data repository 901, an image data repository 910, an EM data repository 920, a robotic data repository 930, a 3D model data repository 940, and other data repositories 941. Each type of input data repository 901-941 stores data of the type indicated by the name for access and use by the navigation module 905. The strain data may include one or more strain measurements along the endoscope 118 (e.g., Figure 4C shape detector 452 generates and / or stores).

[0099] The image data may include one or more image frames captured by an imaging device at the instrument tip, as well as information that allows the time elapsed between pairs of frames to be determined, such as a frame rate or a timestamp.

[0100] The robotic data may include data typically used by the system for functions related to the control of an instrument (e.g., an endoscope 118 and / or its sheath) and / or the physical movement of an instrument or a portion of an instrument (e.g., an instrument tip or sheath) within a tubular network. The robotic data may allow the state of the instrument to be inferred based on data measured while navigating the instrument within the tubular network. Kinematic and dynamic models may be generated based on a priori information collected during the calibration phase. This a priori information may be stored on the device and read and utilized by the robot to improve the drive, control, and navigation of the instrument, and to improve other types of data available to the robot (e.g., EM data, image data, strain-based shape data, etc.). The robotic data may include parameters specific to each instrument.

[0101] Figure 8C shows that can be stored in Fig. 8A Examples of robot data in the robot data repository 930. Figure 8CAs shown, the robot data may include command data 931 instructing the instrument tip to reach a specific anatomical site and / or change its orientation (for example, joint motion data instructing the instrument to exhibit desired joint motion with a specific pitch, roll, and yaw, insertion and retraction data instructing the insertion and retraction of one or both of the guide and the sheath, etc.), force and distance data 932 (for example, the distance the pull wire has been actuated since the device was loaded onto the robot, the amount of force applied to the pull wire measured by the torque sensor in the IDM, the amount of insertion force applied by the robotic arm to insert or retract the instrument, etc.), mechanical model data 933 representing the mechanical motion of the slender member of the instrument (for example, the motion of one or more pull wires, tendons, or shafts of an endoscope that drive the actual movement of the medical device within the tubular network), kinematic model data 934 representing the motion and shape of the instrument (for example, geometric parameters indicating the position of the instrument, and / or any changes in the geometric parameters relative to a predetermined or reference position or coordinate system), etc.

[0102] The EM data may be acquired by one or more EM sensors (eg, located proximal to the instrument tip) and / or an EM tracking system as described above. The 3D model data may be derived, inter alia, from a 2D CT scan as described above.

[0103] The output navigation data repository 990 receives and stores the output navigation data provided by the navigation module 905. The output navigation data indicates information that helps guide the instrument through the patient's anatomical structure and, in one example, through the tubular network to reach a specific destination within the tubular network, and is based on estimated state information of the instrument at each moment. The estimated state information may include the position and orientation of the instrument within the tubular network. In one embodiment, as the instrument moves within the tubular network, updated output navigation data indicating the movement and position / orientation information of the instrument is provided in real time, which better assists its navigation in the tubular network.

[0104] To determine the output navigation data, the navigation module 905 locates (or determines) the estimated state of the instrument within the tubular network. Fig. 8A As shown, the navigation module 905 also includes various algorithm modules, such as a strain-based algorithm module 945, an EM-based algorithm module 950, an image-based algorithm module 960, a robot-based algorithm module 970, and an algorithm module 971 based on other data. Each of these modules may mainly consume certain types of input data and contribute different types of data to the state estimator 980. Fig. 8AAs shown, for illustration, the different kinds of data output by these modules (labeled as strain-based estimated state data, EM-based estimated state data, image-based estimated state data, and robot-based estimated state data, as well as estimated state data based on other data) can generally be referred to as "intermediate data". In some cases, the navigation module 905 determines that damage or failure (e.g., buckling, prolapse, etc.) is about to occur to the instrument based on the estimated state of the instrument. In such cases, the navigation module 905 can control the instrument in a manner to avoid damage or failure. The detailed composition of each algorithm module and the state estimator 980 is described in more detail below.

[0105] VI.B. Navigation Module

[0106] VI.B.1. State Estimator

[0107] As described above, the navigation module 905 also includes a state estimator 980 and multiple algorithm modules that use different algorithm navigation to navigate in the tubular network. For clarity of description, the state estimator 980 is described first, and then the various modules that exchange data with the state estimator 980 are described.

[0108] A state estimator 980 included in the navigation module 905 receives various intermediate data and provides an estimated state of the instrument tip (or other portion of the instrument) as a function of time, wherein the estimated state indicates an estimated position and orientation information of the instrument tip (or other portion of the instrument) within the tubular network. The estimated state data is stored in an estimated state data repository 985 included in the state estimator 980. Although the description herein is described in the context of determining an estimated position and orientation information of the instrument tip (or other portion of the instrument) within the tubular network, in other arrangements, generally speaking, this information can be used to determine an estimated position and orientation information of the instrument tip (or other portion of the instrument) relative to the patient.

[0109] VI.B.2. Estimated State Data Repository

[0110] The estimated state data repository 985 may include a fork data repository, a position data repository, a depth data repository, and an orientation data repository. However, this particular failure of the data repository is only an example, and in an alternative embodiment not shown, different and / or additional data repositories may be included in the estimated state data repository 985.

[0111] The various repositories described above represent estimated state data in various ways. Bifurcation data may refer to the position of a medical device relative to a set of branches (e.g., bifurcations, trifurcations, or divisions into more than three branches) within a tubular network. For example, based on a larger set of available branches provided, for example, by a 3D model that maps the entire tubular network, the bifurcation data may be a set of branch selections selected by the instrument as it traverses the tubular network. Bifurcation data may also include information ahead of the position of the instrument tip, such as branches (forks) that the instrument tip is close to but has not yet traversed, but which may be detected, for example, based on current position information of the tip relative to the 3D model or based on a captured image of an upcoming bifurcation.

[0112] The position data may indicate the three-dimensional position of some portion of the instrument within the tubular network or the three-dimensional position of some portion of the tubular network itself. The position data may be in the form of an absolute position or a relative position relative to, for example, a 3D model of the tubular network. For example, the position data may include an indication of the location of the instrument within a particular branch. The identification of a particular branch may also be stored as a segment identification (ID) that uniquely identifies the particular segment of the model in which the end of the instrument is located.

[0113] The depth data may indicate the depth information of the instrument tip within the tubular network. Exemplary depth data includes the total insertion (absolute) depth of the instrument within the patient and the (relative) depth within an identified branch (e.g., a segment identified by the position data repository 1087). The depth data may be determined based on the position data regarding both the tubular network and the instrument.

[0114] The orientation data may indicate orientation information of the instrument tip, and may include overall roll, pitch, and yaw relative to the 3D model, as well as pitch, roll, and yaw within the identified branches.

[0115] VI.B.3. Data Output to Algorithm Module

[0116] like Fig. 8A As shown, the state estimator 980 provides the estimated state data back to the algorithm module for generating more accurate intermediate data, and the state estimator uses the intermediate data to generate improved and / or updated estimated states, and so on to form a feedback loop. Fig. 8AOne or more algorithm modules shown receive estimated state data. The state estimator 980 uses the data to generate "estimated state data (previous)" associated with timestamp "t-1". The state estimator 980 then provides the data to one or more of the algorithm modules (which may be a combination of algorithm modules that are different from the algorithm modules from which the estimated state data was previously received). The "estimated state data (previous)" may be based on a combination of different types of intermediate data (e.g., image data, mechanical model data, command data, dynamics model data) associated with timestamp "t-1" generated and received from different algorithm modules. For example, estimated state data based on a combination of non-strain-based data 972 may be provided to a strain-based algorithm module 945, and the strain-based algorithm module 945 may determine strain-based estimated state data and output it to the state estimator 980.

[0117] Next, one or more of the algorithm modules uses the received estimated state data (previous) to run its corresponding algorithm to output improved and updated estimated state data to the state estimator 980, which is represented by "estimated state data (current)" shown for the corresponding algorithm module and is associated with a timestamp "t". This process can be repeated for future timestamps to generate estimated state data.

[0118] Since the state estimator 980 may use several different kinds of intermediate data to obtain its estimate of the state of the instrument within the tubular network, the state estimator 980 is configured to account for various different kinds of errors and uncertainties in both measurement and analysis, and each type of underlying data (robotic, EM, image) and each type of algorithmic module may create or carry into the intermediate data for consideration in determining the estimated state. To address these issues, the two concepts of probability distribution and confidence value are discussed.

[0119] As used herein, the term "probability" in the phrase "probability distribution" refers to the probability that the estimate of the possible position and / or orientation of the instrument is correct. For example, different probabilities can be calculated by one of the algorithm modules, which indicates the relative probability that the instrument is in one of several different possible branches in the tubular network. In one embodiment, the type of probability distribution (e.g., discrete distribution or continuous distribution) is selected to match the characteristics of the estimated state (e.g., the type of estimated state, such as continuous position information and discrete branch selection). For example, the estimated state for identifying which segment the instrument enters for the trifurcation can be represented by a discrete probability distribution, and can include three discrete values ​​20%, 30% and 50%, which represent the chance of being in a position in each of the three branches as determined by one of the algorithm modules. For another example, the estimated state may include an instrument roll angle of 40±5 degrees, and the segment depth of the instrument tip in the branch may be 4±1mm, each represented by a Gaussian distribution, which is a class of continuous probability distributions. Different methods or modes can be used to generate probabilities, which will vary with the algorithm modules, as described more fully below with reference to the following figures.

[0120] In contrast, as used herein, a "confidence value" reflects a measure of the confidence of a state estimate provided by one of the algorithms based on one or more factors. For strain-based algorithms using shape sensing fibers, factors such as temperature, proximity to the proximal end of the catheter, and the like may affect the confidence in the estimate of the state. For example, thermal expansion and contraction of the fiber portion may falsely indicate that the instrument is bending. In addition, in some embodiments, the strain measurement of the distal portion of the instrument relies on shape / position data determined based on strain measurements of the proximal portion of the instrument (e.g., closer to the shape detector 452), and any error in the strain measurement of the proximal portion may be amplified in the strain measurement of the distal portion. For EM-based algorithms, factors such as distortion of the EM field, inaccuracy of EM registration, offset or movement of the patient, and breathing of the patient may affect the confidence in the state estimate. Specifically, the confidence value in the state estimate provided by the EM-based algorithm may depend on the patient's breathing cycle, the movement of the patient or the EM field generator, and the position within the anatomical structure where the instrument tip is located. For image-based algorithms, exemplary factors that may affect the confidence value in the state estimate include lighting conditions at the location within the anatomy where the image is captured, the presence of fluid, tissue, or other obstructions against or in front of the optical sensor capturing the image, the patient's breathing, the condition of the patient's own tubular network (e.g., lungs) such as general fluid flow within the tubular network and obstruction of the tubular network, and the specific operating techniques used in, for example, navigation or image capture.

[0121] For example, one factor may be that a particular algorithm has different levels of accuracy at different depths in a patient's lungs, such that relatively close to an airway opening, a particular algorithm may have a high confidence in its estimate of the instrument's position and orientation, but the confidence value may drop the farther the instrument travels in the base of the lung. In general, confidence values ​​are based on one or more system factors associated with the process of determining an outcome, whereas probability is a relative measure produced when attempting to determine the correct outcome from multiple possibilities using a single algorithm based on underlying data.

[0122] For example, a mathematical formula for computing a result of an estimated state represented by a discrete probability distribution (e.g., a branch / segment identification of a trifurcated state with three values ​​of the estimated state involved) may be as follows:

[0123] S1=C EM *P 1，EM +C Image *P 1，Image +C Robot *P 1，Robot ;

[0124] S2=C EM *P 2，EM C Image *P 2，Image +C Robot *P 2，Robot ;

[0125] S3=C EM *P 3，EM +C Image *P 3，Image +C Robot *P 3，Robot .

[0126] In the above exemplary mathematical formula, S i (i=1, 2, 3) represents possible example values ​​of the estimated state when 3 possible segments are identified or exist in the 3D model, C EM , C Image and C Robot represents the confidence values ​​corresponding to the EM-based algorithm, the image-based algorithm, and the robot-based algorithm, and P i，EM , P i，Image and P i，Robot represents the probability of segment i.

[0127] To better illustrate the concepts of probability distribution and confidence values ​​associated with estimated states, a detailed example is provided herein. In this example, a user attempts to identify a segment in a particular trifurcation where the instrument tip is located within the central airway (prediction region) of a tubular network, and the three algorithm modules used include an EM-based algorithm, an image-based algorithm, and a robot-based algorithm. In this example, the probability distribution corresponding to the EM-based algorithm may be 20% in the first branch, 30% in the second branch, and 50% in the third (last) branch, and the confidence value applied to the EM-based algorithm and the central airway is 80%. For the same example, the probability distribution corresponding to the image-based algorithm may be 40%, 20%, 40% for the first branch, the second branch, and the third branch, and the confidence value applied to the image-based algorithm is 30%; while the probability distribution corresponding to the robot-based algorithm may be 10%, 60%, 30% for the first branch, the second branch, and the third branch, and the confidence value applied to the image-based algorithm is 20%. The difference in confidence values ​​applied to the EM-based algorithm and the image-based algorithm indicates that the EM-based algorithm may be a better choice for segment identification in the central airways than the image-based algorithm. An exemplary mathematical calculation of the final estimated state may be: for the first branch: 20%*80%+40%*30%+10%*20%=30%; for the second branch: 30%*80%+20%*30%+60%*20%=42%; and for the third branch: 50%*80%+40%*30%+30%*20%=58%.

[0128] In this example, the output estimated state of the instrument tip may be the resulting values ​​(eg, the resulting 30%, 42%, and 58%), or derivative values ​​from those resulting values, such as a determination that the instrument tip is in the third branch.

[0129] As described above, the estimated state can be represented in a variety of different ways. For example, the estimated state can also include an absolute depth from the airway to the position of the instrument tip, and a data set representing a set of branches traversed by the instrument within the tubular network, the set being a subset of the entire set of branches provided by, for example, a 3D model of the patient's lungs. Applying probability distributions and confidence values ​​to the estimated state allows for improved accuracy in estimating the position and / or orientation of the instrument tip within the tubular network.

[0130] VI.B.4. Strain-based algorithm module

[0131] VI.B.4.i. Elements of the Strain-Based Algorithm Module

[0132] The strain-based algorithm module 945 uses the strain data to determine an estimated state of the device within the tubular network. Figure 8B and Figures 9 to 11Modules that may be included in the strain-based algorithm module 945 are shown. Figure 8B As shown, the strain-based algorithm module 945 may include (i) a shape data determination module 906 for determining shape data based on strain data, (ii) a shape data comparison module 907 for comparing shape data with robot data, (iii) a shape data adjustment module 908 for adjusting shape data (or confidence in the shape data) based on the comparison between the shape data and the robot data; and (iv) a shape-based state estimation module 909 for determining shape-based estimated state data based on the adjusted shape data (or adjusted confidence in the shape data). Although shown as separate components, modules 906-909 may be implemented as one or more hardware components (e.g., a single component, separate components, or any number of components), one or more software components (e.g., a single component, separate components, or any number of components), or any combination thereof. Reference is made below to Figures 9 to 11 Modules 906-909 are described in more detail.

[0133] VI.B.4.ii. Determine shape data

[0134] Fig. 9 An exemplary shape data determination module that may be included in the strain-based algorithm module 945 is shown. Fig. 9 As shown, the shape data determination module 906 receives strain data from the strain data repository 901 and outputs the shape data to the shape data repository 902. The shape data determination module 906 may determine the shape data based on the strain data received from the strain data repository 901. Fig. 8A The strain data may include Figure 4C The shape data may include one or more measurements of strain along one or more optical fibers 456 (or one or more cores thereof) generated and / or stored by the shape detector 452. The shape data may include angles, coordinates, or a combination thereof that indicate the current shape of the instrument. In some cases, the shape data may include curvature information (e.g., curvature values ​​of one or more portions of the instrument), orientation information (e.g., roll, pitch, and / or yaw of one or more portions of the instrument), position information (e.g., the position of one or more portions of the instrument in a reference coordinate system that is used, for example, by the system to navigate the instrument), and / or other information that may be used to indicate the shape of the instrument.

[0135] VI.B.4.iii. Use Robotics Data to Improve Shape Data

[0136] Fig.10 An exemplary shape data comparison module and shape data adjustment module that may be included in the strain-based algorithm module 945 are shown. Fig.10As shown, the shape data comparison module 907 receives data from the plurality of data repositories 902-941. For example, the received data may include shape data from the shape data repository 902 and robot data from the robot data repository 930. The shape data comparison module 907 may compare the received shape data with the received robot data and determine whether the received shape data is consistent with the received robot data.

[0137] As described herein, the robot data may include, for example, kinematic model data that indicates the movement of the instrument expected to be caused by a given set of control commands. The shape data comparison module 907 may compare the movement indicated by the robot data with the movement indicated by the shape data received from the shape data determination module 906. Based on the comparison, the shape data comparison module 907 may output a comparison result indicating whether the shape data is consistent with the robot data and the degree of difference between the shape data and the robot data. For example, the comparison result may indicate that the curvature of the instrument indicated by the shape data is outside the range of acceptable curvatures indicated by the robot data (e.g., exceeds the highest acceptable curvature by a specific amount). For another example, the comparison result may indicate that the shape data corresponding to a specific portion of the instrument is inconsistent with the torque measurement value (e.g., the measurement value of the torque applied to the pull line) included in the robot data.

[0138] VI.B.4.iv. Improving shape data using data other than robot data

[0139] In other embodiments, the shape data comparison module 907 may compare the shape data with image data received from the image data repository 910, the shape data with EM data received from the EM data repository 920, the shape data with 3D model data received from the 3D model data repository 940, the shape data with other data received from other data repository 941, and / or any combination of data received from two or more of the data repositories 910-941.

[0140] For example, the shape data comparison module 907 can determine an expected orientation of the instrument (e.g., at or near the distal end of the instrument) based on the image data received from the image data repository 910. The shape data comparison module 907 can then determine whether the shape data is inconsistent with the expected orientation of the instrument (e.g., the image data indicates that the tip of the instrument is pointing in a direction parallel to the anatomical lumen, but the shape data indicates that the tip of the instrument is pointing toward the inner wall of the anatomical lumen).

[0141] In another example, the shape data comparison module 907 may determine that the anatomical lumen in which the instrument is located has a possible range of coordinate values ​​based on the 3D model data received from the 3D model data repository 940. The shape data comparison module 907 may then determine whether the shape data indicates that the instrument is located outside of the possible range of coordinate values, or whether the shape data indicates that the instrument is shaped in a manner that will not fit in the anatomical lumen.

[0142] In yet another example, the shape data comparison module 907 may determine a set of coordinate values ​​corresponding to the current position of the instrument in the reference coordinate system based on the EM data received from the EM data repository 920. The shape data comparison module 907 may then determine whether the shape data is inconsistent with an expected orientation of the instrument (e.g., the set of coordinate values ​​indicated by the shape data is different from the set of coordinate values ​​indicated by the EM data, or deviates from the set of coordinate values ​​indicated by the EM data by more than a threshold amount).

[0143] In yet another example, a fluoroscopic (X-ray) image may be analyzed by a computer vision algorithm to extract the outline of the instrument, and then the shape data comparison module 907 may determine whether the shape data is inconsistent with the extracted outline of the instrument.

[0144] In yet another example, different sensing modalities may be fitted into the working channel 438 and may be connected to work with the system. These sensing modalities include radial endobronchial ultrasound (REBUS) probes, multispectral imaging (spectroscopy), tomographic imaging (optical coherence tomography, confocal microscopy, two-photon excitation microscopy, etc.). Using the sensor data generated by these sensing modalities, the shape data comparison module 907 may determine whether the shape data is inconsistent with the sensor data.

[0145] VI.B.4.v. Other Examples of Shape Data Comparison

[0146] In some embodiments, the shape data comparison module 907 determines that a mismatch between the shape data and the robot data has been detected for more than a threshold amount of time, and outputs a warning indicating that the instrument may be damaged. For example, the shape data comparison module 907 may determine that the last five comparison results output to the shape data adjustment module 908 indicate that the shape data is inconsistent with the robot data, and output a warning (e.g., indicating that the instrument may be damaged, stuck, or otherwise malfunctioning).

[0147] although Fig.10Not shown, but additionally or alternatively, shape data comparison module 907 may compare shape data with estimated state data received from state estimator 980. In some cases, the shape of the sheath may be known (e.g., based on shape sensing using optical fibers within the sheath, or robotic data corresponding to the sheath). In such cases, shape data comparison module 907 may access shape data corresponding to the sheath surrounding the instrument and compare the shape data of the instrument with the shape data of the sheath.

[0148] In some cases, the shape data comparison module 907 determines that the robot data has a higher confidence value than the shape data, and based on this determination, compares the shape data to the robot data. Alternatively, in some cases, the shape data comparison module 907 determines that the robot data has a lower confidence value than the shape data, and based on this determination, avoids comparing the shape data to the robot data.

[0149] For example, at or near the distal end of the instrument, the confidence value assigned to the shape data or strain data may be lower than the confidence value assigned to the robotic data because, as described above, the strain measurements of the distal portion of the instrument may rely on shape / position data determined based on strain measurements of the proximal portion of the instrument (e.g., closer to the shape detector 452), and any errors in the strain measurements of the proximal portion may be amplified in the strain measurements of the distal portion. On the other hand, at or near the proximal end of the instrument, the confidence value assigned to the shape data or strain data may be higher than the confidence value assigned to the robotic data.

[0150] In some embodiments, the shape data is compared to the robotic data at or near the distal end of the instrument and adjusted as needed, but the shape data is not compared to the robotic data at or near the proximal end of the instrument. In other embodiments, the shape data is compared to the robotic data at or near the distal and proximal ends of the instrument and adjusted as needed.

[0151] VI.B.4.vi. Use comparison results to adjust shape data

[0152] The shape data comparison module 907 outputs the comparison results to the shape data adjustment module 908. The comparison results may indicate which portion of the shape data, if any, does not satisfy one or more conditions indicated by the data (e.g., robot data) to which the shape data is compared. Figure 4D As described above, the comparison result may indicate that the shape data corresponding to a portion of the endoscope 118 indicates that the portion exhibits a curvature value that is inconsistent with the robot data. Figure 4EAs described, the comparison may indicate that the shape data corresponding to a portion of the endoscope 118 indicates that the portion is moving at a speed that is inconsistent with the robot data.

[0153] In other cases, the comparison result may indicate that the direction of the instrument tip indicated by the shape data deviates from the direction of the instrument tip indicated by the robotics data by more than a threshold amount, that the shape data indicates a shape of the instrument such that a portion of the instrument will be outside of an anatomical lumen, or that the position of the instrument indicated by the shape data deviates from the position of the instrument indicated by the robotics data by more than a threshold amount. The comparison result may indicate an error or deviation from system expectations based on one or more of data from various sources and / or an estimated state from state estimator 980.

[0154] Based on the received comparison results, the shape data adjustment module 908 adjusts the shape data and outputs the adjusted shape data to the shape data repository 902. For example, when it is determined that the curvature value indicated by the shape data is inconsistent with the robot data, the shape data adjustment module 908 may modify the shape data so that the curvature value is within the acceptable curvature range indicated by the robot data. For another example, when it is determined that the current speed indicated by the shape data is inconsistent with the robot data, the shape data adjustment module 908 may modify the shape data so that the current speed is within the acceptable speed range indicated by the robot data. In yet another example, when it is determined that part or all of the shape data does not meet one or more conditions indicated by the robot data, the shape data adjustment module 908 may discard such shape data instead of adjusting the shape data. In yet another example, when it is determined that part or all of the shape data does not meet one or more conditions indicated by the robot data, the shape data adjustment module 908 may reduce the confidence of the shape data (e.g., by reducing the confidence value associated with the shape data) instead of adjusting the shape data. The adjusted shape data is stored in the shape data repository 902. In some cases, the adjusted shape data is stored in another data repository than shape data repository 902 .

[0155] In some cases, the shape data adjustment module 908 may make alternative or additional adjustments based on other factors. For example, the shape data adjustment module 908 may adjust the shape data (or adjust the confidence of the shape data) based on temperature changes. In such examples, the shape data adjustment module 908 may adjust the shape data (or adjust the confidence of the shape data) based on the thermal expansion and contraction characteristics of the optical fiber. The shape data adjustment module 908 may make such adjustments in response to determining that the received comparison result indicates that the shape data is inconsistent with at least one other data. In other cases, the shape data determination module 906 considers the current temperature when determining the shape data based on the received strain data, and the shape data adjustment module 908 does not make additional temperature-based adjustments to the shape data.

[0156] In some embodiments, the shape data adjustment module 908 may adjust the shape data (or adjust the confidence of the shape data) based on whether the tip of the instrument is undergoing articulation. Alternatively or in addition, the shape data adjustment module 908 may adjust the shape data (or adjust the confidence of the shape data) based on whether a non-shape changing strain (e.g., temperature, articulation pattern, etc.) is applied to the instrument. The shape data adjustment module 908 may make one or both of these adjustments in response to determining that the received comparison result indicates that the shape data is inconsistent with at least one other data.

[0157] Fig.11 An exemplary shape-based state estimation module that may be included in the strain-based algorithm module 945 is shown. Fig.11 As shown, the shape-based state estimation module 909 receives the adjusted shape data from the shape data repository 902 and determines the shape-based estimated state data based on the adjusted shape data and the previous estimated state data received from the estimated state data repository 985. The shape-based state estimation module 909 outputs the shape-based estimated state data to the estimated state data repository 985. This process can be repeated to generate estimated state data for future timestamps. In some cases, the shape-based state estimation module 909 determines an estimated state of a sheath covering the instrument based on the estimated instrument state.

[0158] VII.A. Overview of Shape Data Adjustment Based on Robot Data

[0159] Fig.12 Flowchart showing an exemplary method operable by a surgical robotic system or component thereof for determining and adjusting shape data based on other data available to the surgical robotic system (such as robot data), according to one embodiment. For example, Fig.12 The steps of the illustrated method 1200 may be performed by a processor and / or other components of a medical robotic system (e.g., surgical robotic system 500) or an associated system (e.g., strain-based algorithm module 945 of navigation configuration system 900). For convenience, the method 1200 is described as being performed by a surgical robotic system (also referred to as a "system" in conjunction with the description of the method 1200).

[0160] The method 1200 starts at block 1201. At block 1205, the system accesses robotic data about an instrument that is navigated (or to be navigated) within an internal region of a body. The robotic data may include data related to the control of an instrument (e.g., endoscope 118 and / or its sheath) and / or the physical movement of a portion of the instrument (e.g., instrument tip or sheath) within a tubular network. As described above, the robotic data may include command data, force and distance data, mechanical model data, kinematic model data, etc.

[0161] At block 1210, the system accesses strain data from an optical fiber positioned within the instrument. The strain data may indicate strain on a portion of the instrument positioned within the internal region of the body. In some cases, the strain data indicates one or both of strain on the distal end of the instrument and strain on the proximal end of the instrument. The strain data may be generated by the shape detector 452 and stored in the strain data repository 901, and the system may access the strain data from the strain data repository 901.

[0162] At box 1215, the system determines shape data based on the strain data. For example, based on the strain on a specific portion of the instrument indicated by the strain data, the system can predict the shape of the specific portion of the instrument. The shape data may include angles, coordinates, or a combination thereof indicating the current shape of the instrument. In some cases, the shape data may include curvature information (e.g., curvature values ​​of one or more portions of the instrument), orientation information (e.g., roll, pitch, and / or yaw of one or more portions of the instrument), position information (e.g., the position of one or more portions of the instrument in a reference coordinate system, which is used, for example, by the system to navigate the instrument), and / or other information that can be used to indicate the shape of the instrument.

[0163] At box 1220, the system compares the robot data and the shape data. In some embodiments, the comparison includes determining whether a specific value included in the shape data satisfies a corresponding condition indicated by the robot data. For example, the robot data accessed by the system may indicate that the instrument cannot be controlled in a manner that causes a curvature value greater than a maximum curvature value or outside a given curvature value range. In such examples, the system may determine whether the curvature value of a given portion of the instrument indicated by the shape data exceeds the maximum curvature value or is outside a given curvature value range indicated by the robot data of the given portion of the instrument. In another example, the robot data accessed by the system may indicate that the instrument cannot move faster than the maximum speed or move outside a specific range of movement. In such examples, the system may determine whether the movement (e.g., speed, movement path, or other time history data) of a given portion of the instrument indicated by the shape data satisfies a movement condition (e.g., maximum speed, movement speed range, etc.) indicated by the robot data of the given portion of the instrument. In other cases, similar techniques may be applied so that the system can determine whether any parameter value indicated by the shape data satisfies the corresponding shape condition indicated by the robot data (e.g., a minimum value, maximum value, and / or range of values ​​that may indicate whether a given parameter value in the shape data is or may be erroneous).

[0164] At box 1225, the system adjusts the shape data based on the comparison of the robot data and the shape data. In some embodiments, the adjustment includes modifying at least a portion of the shape data so that the determination of the estimated state of the instrument (at box 1230) is based on the modified portion of the shape data. For example, when it is determined that the curvature value indicated by the shape data exceeds the maximum curvature value indicated by the robot data, the system may modify the shape data so that the curvature value is less than or equal to the maximum curvature value indicated by the robot data. For another example, when it is determined that the current speed indicated by the shape data exceeds the maximum speed indicated by the robot data, the system may modify the shape data so that the current speed is less than or equal to the maximum speed indicated by the robot data. In other embodiments, the adjustment includes removing at least a portion of the shape data so that the determination of the estimated state of the instrument (at box 1230) is not based on the removed portion of the shape data. For example, when it is determined that a portion or all of the shape data does not meet one or more conditions indicated by the robot data, the system may discard such shape data or ignore such shape data when determining the estimated state at box 1230.

[0165] Adjusting the shape data may also include assigning a confidence value or weight to the shape data or adjusting such confidence value or weight assigned to the shape data. For example, upon determining that the shape data satisfies one or more conditions indicated by the robot data, the system may increase the confidence value or weight associated with the shape data. Alternatively, upon determining that the shape data does not satisfy one or more conditions indicated by the robot data, the system may decrease the confidence value or weight associated with the shape data.

[0166] At block 1230, the system determines an estimated state of the instrument based on the adjusted shape data. In some cases, the system may determine an estimated state of the instrument based on the adjusted shape data and the Fig. 8A Data from one or more data repositories shown and / or from Fig. 8A The system outputs the estimated state of the instrument using a combination of one or more estimated state data of the state estimator 980 in the apparatus. At block 1235, the system outputs the estimated state of the instrument. The method 1200 ends at block 1240.

[0167] VII.B. Overview of the Shape Data Adjustment Process

[0168] Fig.13 is a conceptual diagram illustrating an exemplary method operable by a surgical robotic system or component thereof for operating an instrument according to one embodiment. For example, Fig.13The steps shown in the illustrated 1300 may be performed by a processor and / or other components of a medical robotic system (e.g., surgical robotic system 500) or an associated system (e.g., image-based algorithm module 945 of navigation configuration system 900). For convenience, the process shown in 1300 is described as being performed by a surgical robotic system (also referred to simply as a "system").

[0169] like Fig.13 As shown, the shape data 1305 and the robot data 1310 are fed into a decision block 1315. At block 1315, the system determines whether the shape data 1305 is acceptable based on the robot data 1310. When it is determined that the shape data is not acceptable, the system proceeds to block 1320 and adjusts the shape data. When it is determined that the shape data is acceptable, the system proceeds to block 1325 to drive the instrument based on at least the shape data (or the adjusted shape data), and / or proceeds to block 1330 to navigate the instrument based on at least the shape data (or the adjusted shape data).

[0170] VII.C. Overview of the Shape Data Confidence Adjustment Process

[0171] Fig.14 is a conceptual diagram illustrating an exemplary method operable by a surgical robotic system or component thereof for operating an instrument according to one embodiment. For example, Fig.14 The steps shown in the illustrated 1400 may be performed by a processor and / or other components of a medical robotic system (e.g., surgical robotic system 500) or an associated system (e.g., image-based algorithm module 945 of navigation configuration system 900). For convenience, the process shown in 1400 is described as being performed by a surgical robotic system (also referred to simply as a "system").

[0172] like Fig.14As shown, the system obtains shape data 1405 and robot data 1410 and generates weighted data 1415. Weighted data 1415 may be a weighted sum of shape data 1405 and robot data 1410, where shape data 1405 and robot data 1410 are weighted based on their respective confidence values. If the confidence value associated with shape data 1405 is higher than the confidence value of robot data 1410, the shape represented by weighted data 1415 may be closer to the shape represented by shape data 1405. On the other hand, if the confidence value associated with robot data 1410 is higher than the confidence value of shape data 1405, the shape represented by weighted data 1415 may be closer to the shape represented by robot data 1410. At block 1420, the system determines whether shape data 1405 is acceptable based on robot data 1410. When it is determined that the shape data 1405 is not acceptable, the system proceeds to box 1425 and adjusts the confidence value associated with the shape data 1405. The system then proceeds to box 1430 to drive the instrument based on at least the weighted data reflecting the adjusted confidence value, and / or proceeds to box 1435 to navigate the instrument based on at least the weighted data reflecting the adjusted confidence value. On the other hand, when it is determined that the shape data 1405 is acceptable, the system proceeds to box 1430 to drive the instrument based on at least the weighted data 1415, and / or proceeds to box 1435 to navigate the instrument based on at least the weighted data 1415.

[0173] As just referenced Fig.14 As described, various embodiments described herein can adjust confidence values ​​to improve navigation or control of a medical device. For example, in some cases, a navigation system can use the adjusted confidence values ​​to reduce the weight given to strain-based shape data to determine the position of a medical device relative to a patient's anatomy, as represented by a preoperative model of the patient. As described elsewhere in this disclosure, a navigation system (see, e.g., Fig. 8A ) can receive a plurality of different state estimates of the medical device from the corresponding state estimator, and according to Fig.14 In the illustrated embodiment, the navigation system may reduce the weight given to the state derived from the strain-based shape data based on the comparison. Based on the adjusted confidence level, the navigation system may ignore the state estimate derived from the state estimator using the strain-based shape data, or may reduce the influence of the state estimator in determining the estimated state of the medical device.

[0174] It should be appreciated that the reverse is also possible for embodiments contemplated by the present disclosure. That is, if a comparison between the strain-based shape data and the robotics data-based shape data determines that the two types of data closely match (as may be determined by a threshold amount), then the navigation system may increase the confidence or weight of the state estimate derived from the strain-based shape data.

[0175] Similarly, in Fig.14 As described in , some embodiments may include a control system that controls actuation of a medical device based on a comparison between strain-based shape data and robotic data. Such a control system may use or ignore the strain-based shape data based on the comparison when controlling the posture of the medical device.

[0176] Although this article describes the robot data Figure 4D , Figure 4E and Figures 9 to 14 , but in other embodiments, other data may be used instead of or in combination with the robot data. In addition, although some of the techniques described herein are described with reference to surgical robotic systems, in other embodiments, such techniques may be applied to non-surgical systems, such as medical robotic systems and systems for controlling instruments within internal areas of the body that are not involved in surgical procedures.

[0177] VIII. Implementation System and Terminology

[0178] Implementations disclosed herein provide systems, methods, and devices for detecting physiological noise during navigation of luminal networks.

[0179] It should be noted that as used herein, the terms "couple", "coupling", "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0180] The functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM), or other optical disk storage devices, may include RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or may be used to store the desired program code in the form of an instruction or data structure and any other medium that can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code, or data that can be executed by a computing device or processor.

[0181] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0182] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0183] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0184] The foregoing specific embodiments of the disclosed specific implementations are provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device to at least: accessing robotic data regarding control of the instrument; accessing strain data from an optical fiber positioned within the device, the strain data indicating strain on a portion of the device positioned within an interior region of a body; determining shape data based on the strain data; comparing the robot data and the shape data; identifying differences between the robot data and the shape data based on the comparing; adjusting the shape data based on the identified differences so that the adjusted shape data matches the robot data; determining an estimated state of the instrument based on the adjusted shape data; as well as The estimated state of the instrument is output.

2. The non-transitory computer-readable storage medium of claim 1, wherein adjusting the shape data comprises modifying at least a portion of the shape data such that the determination of the estimated state of the instrument is based on the modified portion of the shape data.

3. The non-transitory computer-readable storage medium of claim 1, wherein adjusting the shape data comprises removing at least a portion of the shape data such that the determination of the estimated state of the instrument is not based on the removed portion of the shape data.

4. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, further cause the processor to: accessing electromagnetic (EM) data captured using (i) an EM sensor located proximal to a distal end of the instrument and (ii) an EM field generator located external to the body; comparing the EM data and the shape data; and The shape data is further adjusted based on the comparison of the EM data and the shape data.

5. The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, further cause the processor to: accessing image data captured by an imaging device located proximal to a distal end of the instrument; comparing the image data and the shape data; and The shape data is further adjusted based on the comparison of the image data and the shape data. 6 . The non-transitory computer-readable storage medium of claim 1 , wherein adjusting the shape data comprises adjusting a confidence value of the shape data.

7. The non-transitory computer readable storage medium of claim 1, wherein the strain data is generated based on a fiber Bragg grating (FBG) produced on a portion of the optical fiber.

8. The non-transitory computer-readable storage medium of claim 1, wherein the shape data comprises one of a curvature value of the portion of the instrument or time history data of the portion of the instrument.

9. The non-transitory computer-readable storage medium of claim 8, wherein the instructions, when executed, further cause the processor to adjust the shape data based on determining that the curvature value is greater than or equal to a threshold curvature value in the robot data.

10. The non-transitory computer-readable storage medium of claim 8, wherein the instructions, when executed, further cause the processor to adjust the shape data based on determining that the time history data satisfies a threshold time history condition in the robot data. 11 . The non-transitory computer-readable storage medium of claim 1 , wherein the instructions, when executed, further cause the processor to adjust the shape data based on a temperature change.

12. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to adjust the shape data based on a determination that a tip of the instrument is articulating.

13. The non-transitory computer readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to adjust the shape data based on determining that a non-shape changing strain is being applied to the instrument.

14. A non-transitory computer-readable storage medium according to claim 1, wherein the instructions, when executed, further cause the processor to assign a confidence value to the robot data corresponding to the first portion based on determining that the first portion of the instrument includes the distal end of the instrument, and the confidence value is higher than the confidence value assigned to the shape data corresponding to the first portion.

15. A non-transitory computer-readable storage medium according to claim 1, wherein the instructions, when executed, further cause the processor to assign a confidence value to the robot data corresponding to the first portion based on determining that the first portion of the instrument includes the proximal end of the instrument, and the confidence value is lower than the confidence value assigned to the shape data corresponding to the first portion.

16. The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to determine an estimated state of a sheath covering the instrument based on the estimated state of the instrument.

17. The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to assign a confidence value to the shape data based on a comparison of the shape data and additional data indicative of a shape of a sheath covering the instrument.

18. The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to: determining imminent damage to the instrument based on the estimated state of the instrument; and The instrument is controlled to avoid said damage.

19. The non-transitory computer-readable storage medium of claim 1, wherein the instructions, when executed, further cause the processor to: determining that a mismatch between the robot data and the shape data has been detected for at least a threshold amount of time; and A warning is output indicating that the instrument may be damaged.

20. A medical robot system, the system comprising: an instrument having an optical fiber positioned within the instrument; a sensor configured to generate strain data indicative of strain on a portion of the device positioned within the interior region of the body; an instrument positioning device attached to the instrument and configured to move the instrument; The non-transitory computer readable storage medium according to any one of claims 1-19.

21. The medical robotic system of claim 20, wherein adjusting the shape data comprises modifying at least a portion of the shape data such that the determination of the estimated state of the instrument is based on the modified portion of the shape data.

22. The medical robotic system of claim 20, wherein adjusting the shape data comprises removing at least a portion of the shape data such that the determination of the estimated state of the instrument is not based on the removed portion of the shape data.

23. The medical robotic system of claim 20, wherein the instructions when executed further cause the system to: accessing electromagnetic (EM) data captured using (i) an EM sensor located proximal to a distal end of the instrument and (ii) an EM field generator located external to the body; comparing the EM data and the shape data; and The shape data is further adjusted based on the comparison of the EM data and the shape data.

24. The medical robotic system of claim 20, wherein the instructions when executed further cause the system to: accessing image data captured by an imaging device located proximal to a distal end of the instrument; comparing the image data and the shape data; and The shape data is further adjusted based on the comparison of the image data and the shape data.

25. The medical robotic system of claim 20, wherein adjusting the shape data comprises adjusting a confidence value of the shape data.

26. The medical robotic system of claim 20, wherein the strain data is generated based on a fiber Bragg grating (FBG) generated on a portion of the optical fiber.

27. The medical robotic system according to claim 20, wherein the shape data comprises one of a curvature value of the portion of the instrument or time history data of the portion of the instrument.

28. The medical robotic system of claim 27, wherein the instructions, when executed, further cause the system to adjust the shape data based on determining that the curvature value is greater than or equal to a threshold curvature value in the robotic data.

29. The medical robotic system of claim 27, wherein the instructions, when executed, further cause the system to adjust the shape data based on determining that the time history data satisfies a threshold time history condition in the robotic data.

30. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to adjust the shape data based on temperature changes.

31. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to adjust the shape data based on a determination that a tip of the instrument is articulating.

32. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to adjust the shape data based on determining that a non-shape changing strain is being applied to the instrument.

33. A medical robotic system according to claim 20, wherein the instructions, when executed, further cause the system to assign a confidence value to the robotic data corresponding to the first part based on determining that the first part of the instrument includes the distal end of the instrument, and the confidence value is higher than the confidence value assigned to the shape data corresponding to the first part.

34. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to assign a confidence value to the robotic data corresponding to the first portion based on determining that the first portion of the instrument includes the proximal end of the instrument, the confidence value being lower than a confidence value assigned to the shape data corresponding to the first portion.

35. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to determine an estimated state of a sheath covering the instrument based on the estimated state of the instrument.

36. The medical robotic system of claim 20, wherein the instructions, when executed, further cause the system to assign a confidence value to the shape data based on a comparison of the shape data and additional data indicative of a shape of a sheath covering the instrument.

37. The medical robotic system of claim 20, wherein the instructions when executed further cause the system to: determining imminent damage to the instrument based on the estimated state of the instrument; and The instrument is controlled to avoid said damage.

38. The medical robotic system of claim 20, wherein the instructions when executed further cause the system to: determining that a mismatch between the robot data and the shape data has been detected for at least a threshold amount of time; and A warning is output indicating that the instrument may be damaged.

Images