User interface for navigating anatomical channels in a medical procedure - Patents.com

Abstract

The medical system includes a scope configured to capture at least one endoscopic image, a display device configured to display the endoscopic image, one or more processors, and a memory storing instructions executable by the one or more processors, the stored instructions including instructions for causing the one or more processors to determine an orientation of the scope associated with the endoscopic image. The stored instructions may include instructions for determining at least one reference axis associated with the lumen network. The reference axis may indicate a first anatomical direction and a second anatomical direction. The stored instructions may include instructions for generating an orientation indicator based on the reference axis. The stored instructions may include instructions for causing a display to present the endoscopic image and the orientation indicator, the orientation indicator being overlaid on the endoscopic image.

Description

[Technical Field]

[0001] The systems and methods disclosed herein relate to navigating a medical instrument through a patient's anatomy, and more particularly to a user interface that assists a user in navigating a medical instrument. [Background technology]

[0002] Certain robotic medical procedures may involve the use of shaft-type instruments, such as endoscopes, that can be inserted into a patient through an orifice (e.g., a natural orifice) and advanced to a target anatomical site. Such medical instruments may be articulatable so that the tip and / or other portion of the shaft can be deflected in one or more dimensions using robotic control. During navigation of the medical instrument, various graphical elements may be provided on the display to assist the operator in advancing the medical instrument to the desired target location. [Brief explanation of the drawings]

[0003] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. In addition, various features of different disclosed embodiments may be combined to form further embodiments that are part of this disclosure. Throughout the drawings, reference numerals may be reused to indicate correspondence between referenced elements. [Figure 1] 1 illustrates an embodiment of a cart-based robotic system configured for diagnostic and / or therapeutic bronchoscopy procedures. [Figure 2] 2 depicts a further aspect of the robotic system of FIG. 1. [Figure 3] 2 illustrates the embodiment of the robotic system of FIG. 1 configured for ureteroscopy. [Figure 4] 2 illustrates the embodiment of the robotic system of FIG. 1 positioned for a vascular procedure. [Figure 5]1 illustrates an embodiment of a table-based robotic system positioned for a bronchoscopy procedure. [Figure 6] 6 provides an alternative view of the robotic system of FIG. 5. [Figure 7] 1 illustrates an example system configured to accommodate a robotic arm. [Figure 8] 1 illustrates an embodiment of a table-based robotic system configured for a ureteroscopy procedure. [Figure 9] 1 illustrates an embodiment of a table-based robotic system configured for laparoscopic procedures. [Figure 10] 10 illustrates the table-based robotic system embodiment of FIGS. 5-9 with pitch or tilt adjustment. [Figure 11] 11 provides a detailed view of the table-column interface of the table-based robotic system of FIGS. 5-10. [Figure 12] 1 illustrates an alternative embodiment of a table-based robotic system. [Figure 13] 13 illustrates an end view of the table-based robotic system of FIG. 12. [Figure 14] FIG. 1 illustrates an end view of a table-based robotic system with a robotic arm attached. [Figure 15] 1 illustrates an exemplary instrument driver. [Figure 16] 1 illustrates an exemplary medical instrument having paired instrument drivers. [Figure 17] 10 illustrates an alternative design of the instrument driver and instrument, where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. [Figure 18] 1 illustrates an instrument having an instrument-based insertion architecture. [Figure 19] 1 illustrates an exemplary controller. [Figure 20] FIG. 16 shows a block diagram illustrating a localization system that estimates the position of one or more elements of the robotic system of FIGS. 1-10, such as the position of the implement of FIGS. 16-18, according to an exemplary embodiment. [Figure 21] 1 illustrates an example of a graphical user interface that provides either or both limited and expanded views of intraoperative images captured by an endoscope. [Figure 22] 1 illustrates an embodiment of a graphical user interface that provides various views. [Figure 23] 1 illustrates an example of a graphical user interface including multiple orientation indicators. [Figure 24] 1 illustrates an example of a graphical user interface, including an example of a two-dimensional (2D) compass. [Figure 25] 1 illustrates an example graphical user interface, including an example three-dimensional (3D) compass. [Figure 26] 10 illustrates an example articulation scenario associated with an articulation indicator. DETAILED DESCRIPTION OF THE INVENTION

[0004] The headings provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. While certain preferred embodiments and examples are disclosed below, the inventive subject matter extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses, as well as to modifications and equivalents thereof. Accordingly, the scope of claims that may arise from this specification is not limited by any of the specific embodiments described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable order and are not necessarily limited to any particular disclosed order. Various operations may be described sequentially as multiple separate operations in a manner that may be helpful in understanding a particular embodiment; however, the order of description should not be construed to imply that these operations are order-dependent. Furthermore, structures, systems, and / or devices described herein may be embodied as integrated or separate components. For purposes of comparing various embodiments, certain aspects and advantages of these embodiments will be described. Not necessarily all such aspects or advantages will be realized by any particular embodiment. Thus, for example, various embodiments may be performed in a manner that achieves or optimizes one advantage or group of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or suggested herein.

[0005] While certain spatially relative terms, such as “outer,” “inner,” “superior,” “lower,” “below,” “upper,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” and similar terms, are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for ease of description to describe positional relationships between elements / structures, such as with respect to the illustrated orientations of the drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures during use or operation in addition to the orientation shown in the drawings. For example, an element / structure described as being “above” another element / structure may represent a position below or to the side of such other element / structure, relative to the intended patient or alternative orientations of the element / structure, and vice versa. It should be understood that spatially relative terms, including those listed above, may be understood with respect to the illustrated orientation of each of the referenced figures.

[0006] Certain reference numbers are reused across different figures of a set of figures of the present disclosure for convenience of devices, components, systems, features, and / or modules having characteristics that may be similar in one or more respects. However, with respect to any of the embodiments disclosed herein, the reuse of a common reference number in a figure does not necessarily indicate that such features, devices, components, or modules are identical or similar. Rather, one skilled in the art may be informed by context as to the extent to which the use of a common reference number may imply similarity between the referenced subject matter. The use of a particular reference number in the context of the description of a particular figure may be understood to relate to the identified device, component, aspect, feature, module, or system in that particular figure, and not necessarily to any device, component, aspect, feature, module, or system identified by the same reference number in another figure. Furthermore, aspects of separate figures identified with a common reference number may be construed as sharing characteristics or as being entirely independent of one another. In some contexts, features associated with separate figures identified by a common reference number are unrelated and / or similar, at least with respect to certain aspects.

[0007] The present disclosure provides systems, devices, and methods for navigation of an instrument shaft, such as a medical endoscope, within a luminal network and monitoring of instrument shaft articulation. Articulation of an instrument according to the present disclosure can be implemented by tensioning one or more tendons, referred to herein as "pull wires," that traverse the instrument shaft. With respect to the medical instruments described in this disclosure, the term "instrument" is used according to its broad and ordinary meaning and may refer to any type of tool, device, assembly, system, subsystem, apparatus, component, etc. In some contexts herein, the term "device" may be used substantially interchangeably with the term "instrument." Furthermore, the term "shaft" is used herein according to its broad and ordinary meaning and may refer to any type of elongated cylinder, tube, scope (e.g., endoscope), prism (e.g., rectangular, oval, elliptical, or oval prism), wire, or the like, regardless of cross-sectional shape. It should be understood that any reference herein to a "shaft" or "instrument shaft" may be understood to refer to an endoscope.

[0008] 1. Overview. Aspects of the present disclosure may be integrated into a robotic-enabled medical system capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy, which may perform bronchoscopy, ureteroscopy, gastroscopy, and other endoscopic procedures.

[0009] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Furthermore, the system may provide the physician with the ability to perform procedures from an ergonomic position without requiring awkward arm movements and positions. Still further, the system may provide the physician with the ability to perform procedures with improved ease of use, such that a single user may control one or more of the system's instruments.

[0010] Various embodiments are described below in conjunction with the drawings for purposes of illustration. While many other implementations of the concepts of the present disclosure are possible, it should be understood that various advantages may be achieved by implementations of the present disclosure. Headings are included herein for reference and to aid in locating various sections. These headings do not limit the scope of the concepts described therein. Such concepts may be applied throughout the entire specification.

[0011] A. Robotic system - cart. Robot-enabled medical systems can be configured in various ways depending on the particular procedure. FIG. 1 illustrates an embodiment of a cart-based robot-enabled system 10 configured for a diagnostic and / or therapeutic bronchoscopy procedure. During a bronchoscopy procedure, the system 10 can include a cart 11 with one or more robotic arms 12 for delivering a medical instrument, such as a steerable endoscope 13, which may be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point (i.e., the mouth of a patient positioned on a table in this example) for delivering diagnostic and / or therapeutic tools. As shown, the cart 11 can be positioned adjacent to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 can be actuated to position a bronchoscope relative to the access point. The configuration of FIG. 1 can also be utilized when performing a gastrointestinal (GI) procedure using a gastroscope, an endoscope specialized for GI procedures. FIG. 2 illustrates an example cart embodiment in more detail.

[0012] With continued reference to FIG. 1 , once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or a combination thereof. As shown, the steerable endoscope 13 can include at least two telescoping parts, such as a portion of an inner leader (e.g., a scope) and a portion of an outer sheath, each portion coupled to a separate instrument driver from a set of instrument drivers 28, each coupled to the distal end of a respective robotic arm. This linear arrangement of the instrument drivers 28, which facilitates coaxial alignment of a portion of the leader with a portion of the sheath, creates a “virtual rail” 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual rails described herein are illustrated in the figures using dashed lines, and thus do not represent any physical structure of the system. Translation of the instrument driver 28 along the virtual rail 29 either nests a portion of the inner leader relative to a portion of the outer sheath or advances or retracts the endoscope 13 from the patient. The angle of the virtual rail 29 can be adjusted, translated, or pivoted based on clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 shown represents a compromise between providing the physician access to the endoscope 13 and minimizing friction resulting from bending the endoscope 13 into the patient's mouth.

[0013] After insertion, the endoscope 13 can be directed downstream of the patient's trachea and lungs using precise commands from the robotic system until it reaches the target destination or surgical site. To enhance navigation through the patient's pulmonary network and / or reach a desired target, the endoscope 13 can be manipulated to telescope a portion of the inner leader out of a portion of the outer sheath and to enhance articulation and increase bend radius. The use of a separate instrument driver 28 also allows the leader portion and sheath portion to be driven independently of one another.

[0014] For example, the endoscope 13 may be directed to deliver a biopsy needle to a target, such as a lesion or nodule within a patient's lung. The needle may be deployed downstream of a working channel extending the length of the endoscope to obtain a tissue sample that is analyzed by a pathologist. Depending on the results of the pathology, additional tools may be deployed downstream of the endoscope's working channel for additional biopsies. After identifying a nodule as malignant, the endoscope 13 may deliver tools endoscopically to remove potentially cancerous tissue. In some cases, diagnostic and therapeutic procedures may be delivered in separate procedures. In these situations, the endoscope 13 may also be used to deliver fiducials to "mark" the location of the targeted nodule. In other cases, diagnostic and therapeutic procedures may be delivered during the same procedure.

[0015] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable and provide support for control, electronics, fluidics, optics, sensors, and / or power to the cart 11. Placing such functionality in the tower 30 allows the form factor of the cart 11 to be smaller, allowing the surgeon and his or her staff to more easily adjust and / or reposition the cart 11. Furthermore, the division of functionality between the cart / table and the support tower 30 reduces clutter in the operating room and facilitates improved clinical workflow. The cart 11 may be positioned near the patient, while the tower 30 may be stored in a remote location so as not to get in the way during the procedure.

[0016] To support the robotic system described above, the tower 30 may include computer-based control system components that store computer program instructions in a non-transitory computer-readable storage medium, such as, for example, a persistent magnetic storage drive, a solid-state drive, or the like. Execution of these instructions, whether performed in the tower 30 or the cart 11, may control the entire system or subsystems thereof. For example, when executed by a processor in the computer system, the instructions may cause the robotic system components to actuate associated carriages and arm mounts, operate a robotic arm, and control a medical instrument. For example, in response to receiving control signals, motors in the joints of a robotic arm may position the arm in a particular pose.

[0017] Tower 30 may also include pumps, flow meters, valve controls, and / or fluid access to provide controlled irrigation and aspiration functions to a system that may be deployed through endoscope 13. These components may also be controlled using the computer system of tower 30. In some embodiments, irrigation and aspiration capabilities may be provided directly to endoscope 13 via separate cables.

[0018] The tower 30 may include voltage and surge protection designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components within the cart 11, making the cart 11 smaller and more mobile.

[0019] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In combination with a control system, such optoelectronic equipment may be used to generate real-time images for display on any number of consoles deployed throughout the system, including within the tower 30. Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical instruments.

[0020] The tower 30 may also include a console 31 in addition to other consoles available to the rest of the system, such as a console mounted on top of a cart. The console 31 may include a user interface and a display screen, such as a touch screen, for the physician operator. The consoles of the system 10 are typically designed to provide both robotic control and pre-operative and real-time procedure information, such as navigation and localization information for the endoscope 13. If the console 31 is not the only console available to the physician, a second operator, such as a nurse, may use the console 31 to monitor the patient's health or vital signs and system operation, as well as to provide procedure-specific data, such as navigation and localization information. In other embodiments, the console 30 is housed in a separate body from the tower 30.

[0021] The tower 30 may be coupled to the cart 11 and endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 through only one cable, simplifying and reducing clutter in the operating room. In other embodiments, certain functions may be combined in separate wiring and connections. For example, power may be provided to the cart through only one power cable, while support for control, optics, fluidics, and / or navigation may be provided through separate cables.

[0022] FIG. 2 provides a detailed view of an embodiment of a cart from the cart-based robot-enabled system shown in FIG. 1. The cart 11 generally includes an elongated support structure 14 (often referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more carriages, such as carriage 17 (alternatively, "arm supports"), for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2). The carriages 17 may include individually configurable arm mounts that rotate along orthogonal axes to adjust the base of the robotic arms 12 for better positioning relative to the patient. The carriages 17 also include a carriage interface 19 that allows the carriages 17 to translate vertically along the column 14.

[0023] Carriage interface 19 connects to column 14 through slots, such as slots 20 positioned on either side of column 14 to guide the vertical translation of carriage 17. Slots 20 contain vertical translation interfaces for positioning and holding the carriage at various vertical heights relative to cart base 15. The vertical translation of carriage 17 allows cart 11 to adjust the reach of robotic arm 12 to meet various table heights, patient sizes, and physician preferences. Similarly, individually configurable arm mounts on carriage 17 allow robotic arm base 21 of robotic arm 12 to be angled in various configurations.

[0024] In some embodiments, a slot cover that is flush and parallel with the slot surface may be added to the slot 20 to prevent dirt and fluids from entering the internal chamber of the column 14 and the vertical translation interface as the carriage 17 translates vertically. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of the slot 20. The cover is coiled within the spools until it deploys from a coiled state to extend and retract as the carriage 17 translates vertically up and down. The spring loading of the spools provides a force that retracts the cover onto the spool as the carriage 17 translates toward the spool, while also maintaining a seal as the carriage 17 translates away from the spool. The cover may be connected to the carriage 17 using, for example, a bracket at the carriage interface 19 to ensure that the cover extends and retracts properly as the carriage 17 translates.

[0025] Column 14 may contain mechanisms therein, such as gears and motors, designed to use a vertically aligned leadscrew to mechanically translate carriage 17 in response to control signals generated in response to user input, such as input from console 16.

[0026] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of linkages 23 connected by a series of joints 24, each including an independent actuator, each including an independently controllable motor. Each separately controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints, thus providing seven degrees of freedom. A large number of joints provides a large number of degrees of freedom, allowing for "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its respective end effector 22 at a specific position, orientation, and trajectory in space using different linkage positions and joint angles. This allows the system to position and orient the medical instrument from a desired point in space, while also allowing the physician to move the arm joints to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0027] The cart base 15 balances the weight of the column 14, carriage 17, and arm 12 on the floor. Thus, the cart base 15 houses the heavier parts, such as the electronics, motor, and power supply, as well as components that allow the cart to be both mobile and / or immobilized. For example, the cart base 15 includes casters 25 in the form of rollable wheels that allow the cart to be easily moved around the room before a procedure. Once in the proper position, the casters 25 can be immobilized using wheel locks to hold the cart 11 in place during a procedure.

[0028] The console 16, positioned at the vertical end of the column 14, provides both a user interface and a display screen (or dual-purpose device, such as a touch screen 26) for receiving user input, providing both pre-operative and intra-operative data to the physician user. Potential pre-operative data on the touch screen 26 may include pre-operative planning, navigation and mapping data derived from a pre-operative computed tomography (CT) scan, and / or notes from a pre-operative patient interview. Intra-operative data on the display may also include vital patient statistics such as respiration, heart rate, and / or pulse, along with optical information provided by tools, sensor information from sensors, and coordinate information. The console 16 can be positioned and tilted to allow a physician access to the console from the side of the column 14 opposite the carriage 17. From this position, the physician can view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 to aid in maneuvering and stabilizing the cart 11.

[0029] FIG. 3 shows an embodiment of the robot-enabled system 10 configured for ureteroscopy. In a ureteroscopy procedure, the cart 11 can be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to follow the patient's urethra and ureters, to the patient's lower abdominal region. During ureteroscopy, it may be desirable for the ureteroscope 32 to be aligned directly with the patient's urethra to reduce friction and force on the sensitive anatomical structures in that area. As shown, the cart 11 can be aligned at the table legs to enable the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. From the table legs, the robotic arm 12 can insert the ureteroscope 32 along a virtual rail 33 directly through the urethra and into the patient's lower abdomen.

[0030] After insertion into the urethra using control techniques similar to those used in bronchoscopy, the ureteroscope 32 can be navigated to the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications. For example, the ureteroscope 32 can be directed into the ureters and kidneys to fragment formed kidney stones using a laser lithotriptor or ultrasonic lithotriptor deployed downstream of the working channel of the ureteroscope 32. After stone fragmentation is complete, the resulting stone fragments can be removed using a basket deployed downstream of the ureteroscope 32.

[0031] FIG. 4 shows an embodiment of a robotic-enabled system similarly positioned for a vascular procedure. In a vascular procedure, the system 10 can be configured so that the cart 11 can deliver a medical instrument 34, such as a steerable catheter, to an access point in the femoral artery in a patient's leg. The femoral artery presents both a larger diameter for navigation and a less circuitous and tortuous path to the patient's heart, simplifying navigation. As in a ureteroscopy procedure, the cart 11 can be positioned toward the patient's leg and lower abdomen to allow the robotic arm 12 to provide direct linear access to the femoral artery access point in the patient's thigh / hip region on a virtual rail 35. After insertion into the artery, the medical instrument 34 can be oriented and inserted by translating the instrument driver 28. Alternatively, the cart can be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0032] B. Robot system - table. Embodiments of a robotic-enabled medical system may also incorporate a patient table. Incorporation of a table reduces the amount of capital equipment in the operating room by removing the cart and allows for better access to the patient. FIG. 5 shows an embodiment of such a robotic-enabled system deployed for a bronchoscopy procedure. The system 36 includes a support structure or column 37 for supporting a platform 38 (shown as a “table” or “bed”) across the floor. Much like a cart-based system, the end effector of the robotic arm 39 of the system 36 includes an instrument driver 42 designed to manipulate an elongated medical instrument, such as the bronchoscope 40 of FIG. 5, through or along a virtual rail 41 formed from the linear positioning of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging can be positioned across the patient's upper abdominal region by placing emitters and detectors around the table 38.

[0033] FIG. 6 provides an alternative view of system 36 without the patient and medical instruments for discussion purposes. As shown, column 37 may include one or more carriages 43, shown as ring-shaped in system 36, which may serve as a base for one or more robotic arms 39. Carriage 43 may translate along a vertical column interface 44 extending the length of column 37 to provide different vantage points from which robotic arms 39 may be positioned to reach the patient. Carriage 43 may rotate about column 37 using mechanical motors positioned within column 37, allowing robotic arms 39 to have access to multiple sides of table 38, such as both sides of the patient. In embodiments with multiple carriages, carriages may be separately positioned on the column and may translate and / or rotate independently of the other carriages. Carriage 43 need not surround column 37 or even be circular, although the illustrated ring shape facilitates rotation of carriage 43 about column 37 while maintaining structural balance. Rotation and translation of carriage 43 allows the system to align medical instruments, such as endoscopes and laparoscopes, with different access points on the patient. In other embodiments (not shown), system 36 may include a patient table or bed having adjustable arm supports in the form of bars or rails extending alongside it. One or more robotic arms 39 may be attached to the adjustable arm supports (e.g., via shoulders with elbow joints) that can be adjusted vertically. By providing vertical adjustment, robotic arms 39 may advantageously be stored compactly beneath a patient table or bed and then raised during a procedure.

[0034] Arm 39 may be attached to the carriage via a set of arm mounts 45 that include a series of joints that may be independently rotated and / or telescopically extended to provide additional configurability to robotic arm 39. Additionally, arm mounts 45 may be positioned on carriage 43 such that, when carriage 43 is appropriately rotated, arm mounts 45 may be positioned on the same side of table 38 (as shown in FIG. 6), on opposite sides of table 38 (as shown in FIG. 9), or on adjacent sides of table 38 (not shown).

[0035] The column 37 structurally provides support for the table 38 and a path for the vertical translation of the carriage. Internally, the column 37 may be equipped with a lead screw for guiding the vertical translation of the carriage and a motor for mechanizing the translation of that carriage based on the lead screw. The column 37 may also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0036] The table base 46 functions similarly to the cart base 15 of the cart 11 shown in FIG. 2, housing the heavier components to counterbalance the table / bed 38, column 37, carriage 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. The casters extend from the bottom of the table base 46 in opposite directions on either side of the base 46 and can be retracted when the system 36 needs to be moved.

[0037] Continuing with FIG. 6 , system 36 may also include a tower (not shown) that divides the functionality of system 36 between the table and tower to reduce the form factor and bulk of the table. As seen in previously disclosed embodiments, the tower may provide various support functions to the table, such as processing, computing, and control capabilities, power, fluidics, and / or optics and sensor processing. The tower may also be movable so that it can be positioned away from the patient to improve physician access and reduce clutter in the operating room. Furthermore, locating components in the tower allows for increased storage space in the table base for possible accommodation of a robotic arm. The tower may also include a master controller or console that provides both a user interface for user input, such as a keyboard and / or pendant, and a display screen (or touch screen) for pre-operative and intra-operative information, such as real-time imaging, navigation, and tracking information. In some embodiments, the tower may also include a holder for a gas tank used for venting.

[0038] In some embodiments, the table base can house and store the robotic arm when not in use. FIG. 7 illustrates a system 47 for housing the robotic arm in a table-based system embodiment. In system 47, carriage 48 can be vertically translated into base 49 to house robotic arm 50, arm mount 51, and carriage 48 within base 49. Base cover 52 can be translated and retracted open to allow carriage 48, arm mount 51, and arm 50 to deploy about column 53, and base cover 52 can be closed to house and protect them when not in use. Base cover 52 can be sealed with a membrane 54 along the edge of its opening to prevent dirt and fluid intrusion when closed.

[0039] FIG. 8 illustrates an embodiment of a robotic table-based system configured for a ureteroscopy procedure. For ureteroscopy, the table 38 may include a swivel portion 55 for positioning the patient at an off-angle from the column 37 and table base 46. The swivel portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) to position the bottom of the swivel portion 55 away from the column 37. For example, pivoting the swivel portion 55 may allow a C-arm (not shown) to be positioned over the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the carriage 35 (not shown) about the column 37, the robotic arm 39 may insert the ureteroscope 56 directly into the patient's groin area along a virtual rail 57 to reach the urethra. For ureteroscopy, stirrups 58 may also be secured to the swivel portion 55 of the table 38 to support the patient's leg position during the procedure and allow clear access to the patient's groin area.

[0040] In laparoscopic procedures, minimally invasive instruments may be inserted into a patient's anatomy through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include an elongated, rigid member, such as a shaft, that is used to access anatomy within the patient. After distension of the patient's abdominal cavity, the instruments may be oriented to perform surgical or medical tasks, such as grasping, cutting, ablation, suturing, etc. In some embodiments, the instruments may include a scope, such as a laparoscope. FIG. 9 illustrates an embodiment of a robotic-compatible table-based system configured for laparoscopic procedures. As shown in FIG. 9 , the carriage 43 of the system 36 may be rotated and vertically adjusted to position a pair of robotic arms 39 on either side of the table 38, allowing instruments 59 to be positioned using arm mounts 45 to reach the patient's abdominal cavity through minimal incisions on either side of the patient.

[0041] To accommodate laparoscopic procedures, the robotic table system may also tilt the platform to a desired angle. FIG. 10 illustrates an embodiment of a pitch- or tilt-adjustable robotic medical system. As shown in FIG. 10, the system 36 can accommodate the tilt of the table 38 to position one portion of the table higher from the floor than another portion. Additionally, the arm mount 45 can rotate to match the tilt such that the arm 39 maintains the same planar relationship as the table 38. To accommodate steeper angles, the column 37 may also include a telescoping portion 60 that allows the column 37 to extend vertically to prevent the table 38 from contacting the floor or colliding with the base 46.

[0042] FIG. 11 provides a detailed illustrative diagram of the interface between the table 38 and the column 37. The pitch mechanism 61 may be configured to vary the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch mechanism 61 may be enabled by positioning orthogonal axes 1, 2 at the column-table interface, each actuated by a separate motor 3, 4 in response to an electrical pitch command. Rotation along one screw 5 will allow tilt adjustment in one axis 1, while rotation along the other screw 6 will allow tilt adjustment along the other axis 2. In some embodiments, ball joints may be used to vary the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0043] For example, pitch adjustment is particularly useful when attempting to position the table in Trendelenburg position, i.e., positioning the patient's lower abdomen higher off the floor than the patient's lower abdomen for lower abdominal surgery. The Trendelenburg position allows the force of gravity to slide the patient's internal organs down into the patient's upper abdomen, emptying the abdominal cavity for entry of minimally invasive tools to perform lower abdominal surgical or medical procedures, such as laparoscopic prostatectomy.

[0044] 12 and 13 show isometric and end views of another embodiment of a table-based surgical robotic system 100. The surgical robotic system 100 includes one or more adjustable arm supports 105 that can be configured to support one or more robotic arms relative to the table 101 (see, for example, FIG. 14 ). In the illustrated embodiment, only one adjustable arm support 105 is shown, but additional arm supports can be provided on the opposite side of the table 101. The adjustable arm support 105 can be configured to move relative to the table 101 to adjust and / or change the position of the adjustable arm support 105 and / or any robotic arms attached thereto relative to the table 101. For example, the adjustable arm support 105 can be adjusted with one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides the system 100 with great versatility, including the ability to easily accommodate one or more adjustable arm supports 105 and any robotic arms attached thereto beneath the table 101. The adjustable arm support 105 can be raised from a stowed position to a position below the top surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from a stowed position to a position above the top surface of the table 101.

[0045] The adjustable arm support 105 can provide several degrees of freedom, including lift, lateral translation, tilt, etc. In the illustrated embodiment of FIGS. 12 and 13, the arm support 105 is configured with four degrees of freedom, which are illustrated by arrows in FIG. 12. A first degree of freedom allows adjustment of the adjustable arm support 105 in the z-direction ("Z-lift"). For example, the adjustable arm support 105 can include a carriage 109 configured to move up and down along or relative to the column 102 that supports the table 101. A second degree of freedom can allow the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 can include a rotational joint, which can allow the adjustable arm support 105 to align with a bed in Trendelenburg position. A third degree of freedom allows the adjustable arm support 105 to "pivot up," which can be used to adjust the distance between the side of the table 101 and the adjustable arm support 105. A fourth degree of freedom may allow translation of the adjustable arm support 105 along the longitudinal length of the table.

[0046] The surgical robotic system 100 of Figures 12 and 13 may include a table supported by a column 102 mounted to a base 103. The base 103 and column 102 support the table 101 against a support surface. A bed axis 131 and a support axis 133 are shown in Figure 13.

[0047] An adjustable arm support 105 can be mounted to the column 102. In other embodiments, the arm support 105 can be mounted to the table 101 or the base 103. The adjustable arm support 105 can include a carriage 109, a bar or rail connector 111, and a bar or rail 107. In some embodiments, one or more robot arms mounted to the rail 107 can translate and move relative to each other.

[0048] The carriage 109 can be attached to the column 102 by a first joint 113, which allows the carriage 109 to move relative to the column 102 (e.g., up and down about a first or vertical axis 123). The first joint 113 can provide a first degree of freedom ("Z-lift") for the adjustable arm support 105. The adjustable arm support 105 can include a second joint 115 that provides a second degree of freedom (tilt) for the adjustable arm support 105. The adjustable arm support 105 can include a third joint 117 that can provide a third degree of freedom ("pivot up") for the adjustable arm support 105. An additional joint 119 (shown in FIG. 13) can be provided that mechanically constrains the third joint 117 to maintain the orientation of the rail 107 as the rail connector 111 is rotated about a third axis 127. The adjustable arm support 105 may include a fourth joint 121 that may provide a fourth degree of freedom (translation) to the adjustable arm support 105 along a fourth axis 129 .

[0049] 14 illustrates an end view of a surgical robotic system 140A with two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101. A first robotic arm 142A is attached to a bar or rail 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A, which can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B, which can be configured to attach to one or more robotic medical instruments or tools.

[0050] In some embodiments, one or more of the robotic arms 142A, 142B comprise arms with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B can include eight degrees of freedom, including an insertion axis (one degree of freedom including insertion), a wrist (three degrees of freedom including wrist pitch, yaw, and roll), an elbow (one degree of freedom including elbow pitch), a shoulder (two degrees of freedom including shoulder pitch and yaw), and a base 144A, 144B (one degree of freedom including translation). In some embodiments, the insertion degree of freedom can be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides the insertion via an instrument-based insertion architecture.

[0051] C. Instrument Drivers and Interfaces. The end effector of the system's robotic arm includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates electromechanical means for actuating the medical instrument, and (ii) a removable or detachable medical instrument that may lack any electromechanical components such as a motor. This dichotomy can be caused by the need to sterilize medical instruments used in medical procedures and the inability to adequately sterilize expensive capital equipment due to the medical instrument's complex mechanical assembly and sensitive electronics. Therefore, medical instruments can be designed to be detached, removed, and replaced from the instrument driver (and thus the system) during individual sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be replaced or sterilized and can be draped for protection.

[0052] FIG. 15 illustrates an exemplary instrument driver. The instrument driver 62, positioned at the distal end of the robotic arm, is composed of one or more drive units 63 arranged on parallel axes to provide controlled torque to the medical instrument via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gearhead 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the motor shaft speed and providing feedback to the control circuitry, and a control circuit 68 for receiving control signals and operating the drive unit. Each drive unit 63 is controlled and motorized independently of the others, and the instrument driver 62 can provide multiple (four shown in FIG. 15 ) independent drive outputs to the medical instrument. In operation, the control circuit 68 receives the control signals, sends a motor signal to the motor 66, compares the resulting motor speed measured by the encoder 67 to the desired speed, and modulates the motor signal to generate the desired torque.

[0053] For procedures requiring a sterile environment, the robotic system may incorporate a drive interface, such as a sterile adapter connected to a sterile drape, located between the instrument driver and the medical instrument. The primary purpose of the sterile adapter is to transfer angular motion from the instrument driver's drive shaft to the instrument's drive input while maintaining physical separation, and therefore sterility, between the drive shaft and the drive input. Thus, an example sterile adapter may consist of a series of rotational inputs and outputs intended to mate with the instrument driver's drive shaft and the drive input to the instrument. The sterile drape connected to the sterile adapter is constructed of a thin, flexible material, such as transparent or translucent plastic, and is designed to cover the instrument driver, robotic arm, and capital equipment, such as a cart (in cart-based systems) or table (in table-based systems). The use of the drape allows the capital equipment to be positioned adjacent to the patient while still located in an area not requiring sterility (i.e., the non-sterile field). On the other side of the sterile drape, the medical instrument may interact with the patient in an area requiring sterility (i.e., the sterile field).

[0054] D. Medical equipment. FIG. 16 illustrates an exemplary medical instrument with a mated instrument driver. Similar to other instruments designed for use in a robotic system, the medical instrument 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its design intended for manual interaction by a physician, may include a rotary drive input 73, e.g., a receptacle, pulley, or spool, designed to mate with a drive output 74 that passes through a drive interface on an instrument driver 75, typically at the distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 may share an axis of rotation with the drive output 74 in the instrument driver 75, allowing for the transfer of torque from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline designed to mate with a receptacle on the drive input 73.

[0055] The elongate shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in endoscopy, or a minimally invasive incision, such as in laparoscopy. The elongate shaft 71 can be either flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or can include a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongate shaft can be connected to an end effector extending from an articulating wrist formed from a clevis having at least one degree of freedom, and to a surgical tool or medical instrument, such as a grasper or scissors, that can be actuated based on force from a tendon as the drive input rotates in response to torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongate shaft can include a steerable or controllable bend that can be articulated and bent based on torque received from the drive output 74 of the instrument driver 75.

[0056] Torque from the instrument driver 75 is transmitted downstream of the elongate shaft 71 using tendons along the shaft 71. These individual tendons, such as pull wires, may be individually anchored to individual drive inputs 73 in the instrument handle 72. From the handle 72, the tendons travel along the elongate shaft 71 through one or more pull lumens and are anchored to a distal portion of the elongate shaft 71 or to a wrist at a distal portion of the elongate shaft. During a surgical procedure, such as a laparoscopic, endoscopic, or hybrid procedure, these tendons may be coupled to a distally attached end effector, such as a wrist, grasper, or scissors. In such an arrangement, torque exerted on the drive input 73 will transmit tension to the tendons, thereby actuating the end effector in some manner. In some embodiments, the tendons may rotate a joint about an axis to move the end effector in one direction or another during a surgical procedure. Alternatively, the tendon may be connected at the distal end of the elongate shaft 71 to one or more jaws of a grasping instrument that closes under tension from the tendon.

[0057] In endoscopy, the tendons may be coupled to a bending or articulating section positioned along the elongate shaft 71 (e.g., at the distal end) via adhesive, control rings, or other mechanical fixation. When fixedly attached to the distal end of a bending section, torque exerted on the drive input 73 is transmitted to the tendon, causing the softer bending section (sometimes referred to as the articulating section or articulating region) to bend or articulate. Along non-bending sections, it may be advantageous to helical or spiral the individual pull lumens that direct the individual tendons along (or within) the wall of the endoscope shaft to counterbalance the radial forces resulting from tension in the pull wires. The angle of the helix and / or spacing between them can be varied or designed for specific purposes; narrower helices provide poor shaft compression under load forces, while fewer helices provide good shaft compression under load forces but also limit bending. At the other end of the spectrum, orienting the pull lumen parallel to the longitudinal axis of elongate shaft 71 may allow for controlled articulation at a desired flexion or articulation.

[0058] In endoscopy, the elongated shaft 71 houses several components that assist in robotic procedures. The shaft may be configured with a working channel for placing surgical tools (or medical instruments), irrigation, and / or suction into the surgical field at the distal end of the shaft 71. The elongated shaft 71 may also house wires and / or optical fibers that transmit signals to and from an optical assembly at the distal tip, which may include an optical camera. The shaft 71 may also house optical fibers for carrying light from a proximally located light source, such as a light emitting diode, to the distal end of the shaft.

[0059] At the distal end of instrument 70, the distal tip may also include a working channel opening for delivering tools to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. The distal tip may also include a port for a camera, such as a fiberscope or digital camera, to capture images of the internal anatomical space. In this regard, the distal tip may also include a port for a light source to illuminate the anatomical space when using the camera.

[0060] 16, the drive shaft axis, and therefore the drive input axis, is perpendicular to the axis of the elongate shaft. However, this arrangement complicates the roll ability of the elongate shaft 71. Rolling the elongate shaft 71 along its axis while holding the drive input 73 stationary results in undesirable entanglement of the tendons as they exit the drive input 73 and enter the pull lumen within the elongate shaft 71. Such resulting entanglement of tendons can interfere with any control algorithm aimed at predicting the movement of a flexible elongate shaft during an endoscopic procedure.

[0061] FIG. 17 illustrates an alternative design of an instrument driver and instrument in which the axes of the drive units are parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units with their drive outputs 81 aligned parallel at the end of a robotic arm 82. The drive units and their respective drive outputs 81 are housed in a rotating assembly 83 of the instrument driver 80, which is driven by one of the drive units in assembly 83. In response to torque provided by the rotary drive units, the rotating assembly 83 rotates along a circular bearing connecting the rotating assembly 83 to a non-rotating portion 84 of the instrument driver. Power and control signals may be transmitted from the non-rotating portion 84 of the instrument driver 80 to the rotating assembly 83 through electrical contacts and may be maintained throughout rotation by a brushed slip-ring connection (not shown). In other embodiments, the rotating assembly 83 may be integrated into the non-rotating portion 84 and therefore respond to a separate drive unit that is not parallel to the other drive units. Rotation mechanism 83 enables instrument driver 80 to rotate the drive units and their respective drive outputs 81 as a single unit about instrument driver axis 85 .

[0062] Similar to the previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown with a transparent exterior skin for purposes of illustration) that includes a plurality of drive inputs 89 (such as receptacles, pulleys, and spools) configured to receive the drive outputs 81 in the instrument driver 80. Unlike the previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis being substantially parallel to the axis of the drive inputs 89, rather than orthogonal as seen in the design of FIG.

[0063] When coupled to the rotation assembly 83 of the instrument driver 80, the medical instrument 86, including the instrument base 87 and the instrument shaft 88, rotates with the rotation assembly 83 about the instrument driver axis 85. Because the instrument shaft 88 is positioned in the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Rotation of the rotation assembly 83 therefore causes the instrument shaft 88 to rotate about its own longitudinal axis. Also, because the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 at the instrument base 87 do not entangle during rotation. Thus, the parallelism of the axes of the drive output 81, drive input 89, and instrument shaft 88 allows shaft rotation without entangling any of the controlling tendons.

[0064] FIG. 18 illustrates an instrument having an instrument-based insertion architecture, according to some embodiments. The instrument 150 can be coupled to any of the instrument drivers discussed above. The instrument 150 includes an elongate shaft 152, an end effector 162 connected to the elongate shaft 152, and a handle 170 coupled to the elongate shaft 152. The elongate shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongate shaft 152 includes one or more channels or grooves 158 along its outer surface. The grooves 158 are configured to receive one or more wires or cables 180 passing therethrough. Thus, the one or more cables 180 run along the outer surface of the elongate shaft 152. In other embodiments, the cables 180 can run through the elongate shaft 152. Manipulation of the one or more cables 180 (e.g., via the instrument driver) results in actuation of the end effector 162.

[0065] The instrument handle 170, sometimes referred to as the instrument base, may include a mounting interface 172 having one or more mechanical inputs 174, such as receptacles, pulleys, or spools, typically designed to intermate with one or more torque couplers on the mounting surface of the instrument driver.

[0066] In some embodiments, instrument 150 comprises a series of pulleys or cables that allow elongated shaft 152 to translate relative to handle 170. In other words, instrument 150 itself comprises an instrument-based insertion architecture that accommodates the insertion of instruments, thereby minimizing reliance on a robotic arm to effect the insertion of instrument 150. In other embodiments, the robotic arm may be primarily responsible for the insertion of instruments.

[0067] E. Controller. Any of the robotic systems described herein can include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the instrument and controller can be coupled (e.g., communicatively, electronically, wirelessly, and / or mechanically) such that manipulation of the controller causes a corresponding manipulation of the instrument via, for example, master-slave control.

[0068] 19 is a perspective view of an embodiment of controller 182. In this embodiment, controller 182 includes a hybrid controller that can include both impedance control and admittance control. In other embodiments, controller 182 may utilize only impedance control or passive control. In other embodiments, controller 182 may utilize only admittance control. By being a hybrid controller, controller 182 has the advantage of providing a lower perceived inertia during use.

[0069] In the illustrated embodiment, the controller 182 is configured to enable operation of two medical instruments and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0070] 19 , each positioning platform 188 includes a SCARA arm (selective compliance assembly robot arm) 198 coupled to a column 194 by a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along a rail 197) to allow each of the handles 184 to translate in the z-direction, providing a first degree of freedom. The SCARA arm 198 is configured to allow movement of the handles 184 in the x-y plane, providing an additional two degrees of freedom.

[0071] In some embodiments, one or more load cells are positioned within the controller. For example, in some embodiments, a load cell (not shown) is positioned on the body of each of the gimbals 186. Providing load cells allows a portion of the controller 182 to operate under admittance control, thereby advantageously reducing the perceived inertia of the controller during use. In some embodiments, the positioning platform 188 is configured for admittance control while the gimbals 186 are configured for impedance control. In other embodiments, the positioning platform 188 is configured for impedance control while the gimbals 186 are configured for admittance control. Thus, in some embodiments, the translational or positional degree of freedom of the positioning platform 188 can rely on admittance control, while the rotational degree of freedom of the gimbals 186 relies on impedance control.

[0072] F. Navigation and Control. Traditional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered through a C-arm) and other forms of radiation-based imaging modalities to provide intraluminal guidance to the operating physician. In contrast, the robotic systems contemplated by the present disclosure may provide non-radiation-based navigation and localization means to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "localization" may refer to determining and / or monitoring the position of an object within a reference coordinate system. Techniques such as preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to achieve a radiation-free surgical environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robot command data may be used individually or in combination to improve information obtainable solely through radiation-based imaging modalities.

[0073] FIG. 20 is a block diagram illustrating a localization system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an exemplary embodiment. The localization system 90 may be a set of one or more computing devices configured to execute one or more instructions. The computing devices may be embodied by a processor (or processors) and computer-readable memory in one or more of the components discussed above. By way of example, and not limitation, the computing devices may be the tower 30 shown in FIG. 1, the cart shown in FIGS. 1-4, the bed shown in FIGS. 5-14, etc.

[0074] 20, the localization system 90 may include a localization module 95 that processes the input data 91-94 to generate position data 96 of the distal tip of the medical instrument. The position data 96 may be data or logic that represents the position and / or orientation of the distal tip of the instrument relative to a frame of reference. The frame of reference may be a frame of reference relative to the patient's anatomy or relative to a known object such as an EM field generator (see the discussion of EM field generators below).

[0075] The various input data 91-94 will now be described in more detail. Preoperative mapping can be accomplished through the use of low-dose CT scan acquisition. Preoperative CT scans, for example, are reconstructed into three-dimensional images visualized as cutaway "slices" of the patient's internal anatomy. When analyzed as a whole, image-based models can be generated that cover the anatomical cavities, spaces, and structures of the patient's anatomy, such as the patient's pulmonary network. Features such as centerline geometry can be determined and approximated from the CT images to create a three-dimensional volume of the patient's anatomy, referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suited to bronchoscopy.

[0076] In some embodiments, the instrument may be equipped with a camera to provide visual data 92. A localization module 95 may process the visual data to enable one or more vision-based position tracking. For example, pre-operative model data may be used in conjunction with the visual data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advancing through the working channel of an endoscope). For example, using the pre-operative model data 91, the robotic system may generate a model-based predicted endoscopic image library based on the expected path of travel of the endoscope, with each image linked to a location within the model. During surgery, this library may be referenced by the robotic system to compare real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) with those in the image library to aid in localization.

[0077] Other computer vision-based tracking techniques use feature tracking to determine the motion of the camera, and thus the endoscope. Some features of the localization module 95 may identify circular geometric shapes in the preoperative model data 91 that correspond to anatomical lumens and track changes in that geometry to determine which anatomical lumens have been selected and the relative rotational and / or translational motion of the camera. The use of a phase map may further enhance vision-based algorithms or techniques.

[0078] Optical flow, another computer vision-based technique, may analyze the displacement and translation of image pixels in a video sequence in visual data 92 to infer camera movement. Examples of optical flow techniques may include motion detection, object segmentation calculations, luminance, motion compensated coding, stereo disparity measurements, etc. By comparing multiple frames over multiple iterations, the movement and position of the camera (and therefore the endoscope) may be determined.

[0079] The localization module 95 can use real-time EM tracking to generate a real-time position of the endoscope within a global coordinate system that can be registered to the patient's anatomy represented by the preoperative model. In EM tracking, an EM sensor (or tracker), consisting of one or more sensor coils embedded in a medical instrument (e.g., an endoscopic instrument) at one or more locations and orientations, measures variations in an EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed near the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces small currents in the EM sensor's sensor coils, which can be analyzed to determine the spacing and angle between the EM sensor and the EM field generator. These spacings and orientations can be intraoperatively "registered" to the patient's anatomy (e.g., the preoperative model) to determine a geometric transformation that aligns a single location in the coordinate system with a location in the preoperative model of the patient's anatomy. Once registered, an EM tracker embedded at one or more locations on the medical instrument (e.g., the distal tip of an endoscope) can provide a real-time display of the medical instrument's progression through the patient's anatomy.

[0080] The robot command and kinematic data 94 may also be used by a localization module 95 to provide localization data 96 for the robotic system. During pre-operative calibration, the device pitch and yaw resulting from the joint movement commands may be determined. Intra-operatively, these calibration measurements may be used in combination with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations may be analyzed in combination with EM, vision, and / or phase modeling to estimate the position of the medical instrument within the network.

[0081] As Figure 17 shows, several other input data may be used by the localization module 95. For example, although not shown in Figure 17, an instrument that utilizes shape-sensing fibers may provide shape data that the localization module 95 can use to determine the position and shape of the instrument.

[0082] The localization module 95 may use a combination of the input data 91-94. In some cases, such a combination may use a probabilistic approach in which the localization module 95 assigns confidence weights to the position determined from each of the input data 91-94. Thus, if the EM data is unreliable (e.g., in the presence of EM interference), which may reduce the reliability of the position determined by the EM data 93, the localization module 95 may rely more heavily on the vision data 92 and / or the robot command and kinematics data 94.

[0083] As discussed above, the robotic systems discussed herein may be designed to incorporate a combination of one or more of the above techniques. The computer-based control system of a tower-, bed-, and / or cart-based robotic system may store, for example, in a non-transitory computer-readable storage medium such as a persistent magnetic storage drive, solid-state drive, or the like, computer program instructions that, when executed, cause the system to receive and analyze sensor data and user commands, generate system-wide control signals, and display navigation and localization data such as instrument position within a global coordinate system, anatomical maps, and the like.

[0084] 2. User interfaces for navigation in medical procedures. Figure 21 shows an example of a graphical user interface (GUI) 2100 that provides either or both a limited field of view 2102 and an expanded field of view 2104 of intraoperative images captured by an endoscope. In the left half of Figure 21, the GUI 2100 presents the expanded field of view 2102, and in the right half of Figure 21, the GUI 2100 presents the limited field of view 2104 on the viewer / display. Both fields of view 2102, 2104 can present intraoperative images captured at the anatomical site.

[0085] In some embodiments, the GUI 2100 may present images side-by-side in both the expanded field of view 2102 and the limited field of view 2104, as shown. In other embodiments, the GUI may include only one selected field of view of the two fields of view 2102, 2104. For example, the GUI 2100 may present only the limited field of view 2104, provided as a circular cutout shape.

[0086] The expanded field of view 2102 can present a superset of the image presented in the limited field of view 2104 to provide a better appreciation of the position and orientation of the endoscope relative to surrounding features presented in the expanded field of view 2102 but not in the limited field of view 2104. In some embodiments, the expanded field of view 2102 can be an uncropped endoscope feed captured by an imaging device of the endoscope with its full field of view. The shape of the image is not limited to the square shape of the expanded field of view 2102 as shown, but can be any shape provided by the imaging device.

[0087] The expanded view 2102 of a live endoscope can provide the operator with more camera data on the screen as they navigate during a procedure. A traditional livescope view is a circular crop of the entire endoscopic camera. However, the camera typically collects more data than is currently displayed to the operator. Providing the operator with more of the available image captured by the imaging device can be beneficial in scenarios where an airway or other anatomical structure is visible but has previously been cropped from the view. In some embodiments, the operator has the ability to toggle off the crop to obtain the full image from the imaging device. This allows for improved navigation and the ability to better correct the trajectory of the endoscope.

[0088] In some embodiments, the expanded field of view 2102 may extend beyond its full field of view and supplement its captured endoscopic feed with predicted or otherwise simulated portions to present uncaptured features within the expanded field of view 2102. For example, based on the current position and orientation of the endoscope relative to the target anatomy, various image generation algorithms may process preoperative model data (e.g., CT scan data) to predict or simulate features that surround the endoscopic feed but are not visible in the endoscopic feed. As another example, artificial intelligence or trained machine learning algorithms may be used to predict or simulate such features. The predicted or simulated image portions may be provided with brighter brightness, lower contrast, or less emphasis to (i) distinguish between the actual captured portions and portions of the generated image and (ii) avoid distracting the operator.

[0089] The limited field of view 2104 may present a subset of the image presented in the expanded field of view 2102 to (i) better focus on features near the center of the endoscope feed or (ii) provide more instinctive navigation guidance to the operator of the endoscope. For example, a circular field of view, such as that shown in the limited field of view 2104, may often be more easily associated with the circular shape of the endoscope, allowing the operator to more instinctively navigate the endoscope based on the image in the limited field of view 2104. Note that the expanded field of view 2102 and the limited field of view 2104 are not limited by their respective shapes, but rather by being supersets or subsets of the respective fields of view that they present. That is, although the expanded field of view 2102 is shown as a square field of view and the limited field of view 2104 is shown as a circular field of view, it should be understood that any other shapes may be used. For example, any polygonal, curved, or any other shape may be applied as a mask or filter to the expanded field of view 2102 and the limited field of view 2104, as long as the limited field of view 2104 is a subset of the expanded field of view 2102.

[0090] In some implementations where the two fields of view 2102, 2104 are not presented together, but only one field of view, the operator may manually switch between the expanded field of view 2102 and the limited field of view 2104. Switching can be performed by manually selecting a switch button 2106. While the switch button 2106 is shown as a slidable switch, the switch button 2106 may be any selectable graphical control element that can capture the selection or deselection of a field of view, such as a radio button, a checkbox, or the like. In some embodiments, the switch button 2106 may not be a graphical control element, but rather a physical control element of the medical system. Some examples of control elements include a physical button, a touch-sensitive button, a touch screen icon, a joystick, a foot pedal, and other elements of an input device that can be used to provide input to the system.

[0091] In some embodiments, switching between the expanded field of view 2102 and the limited field of view 2104, and vice versa, may occur automatically based on a detected condition. For example, a detected condition may include detecting an obstacle or obstruction in the navigation path of the endoscope, detecting mucus in the navigation path of the endoscope, or detecting an adjacent airway that is blocking navigation or that may be of interest to the operator and / or is otherwise not currently displayed within the limited field of view 2104. Additionally, switching between the expanded field of view 2102 and the limited field of view 2104 may depend on the location of the target or lesion, the position and orientation of the surgical needle relative to the location of the target or lesion, for example, the expanded field of view 2102 may be automatically switched on as the needle approaches the lesion. Furthermore, airway configuration, airway size, and airway position may affect the automatic switching on and off of the limited field of view 2102 and the expanded field of view 2104.

[0092] It should be appreciated that machine learning and artificial intelligence may be utilized to train algorithms to predict or detect other situations during surgery in which the magnified field of view 2102 would be useful to a physician and to switch on the magnified field of view 2102 in response. For example, it may be predicted or determined that a particular location, orientation, or junction within the lumen network may pose some challenges for accurate navigation (e.g., a particularly confusing set of branches), and once the endoscope is determined to be at that location, orientation, or junction, the magnified field of view 2102 may be automatically switched on. In some embodiments, instead of automatically turning on the magnified field of view 2102, a recommendation (e.g., hint text or icon) for turning on the magnified field of view 2102 may be presented. Similarly, in some implementations, the magnified field of view 2102 may be automatically switched off when the endoscope is moved away from the location, orientation, or junction.

[0093] In some embodiments, one or more visual indicators or effects may be applied to the expanded field of view 2102 to distinguish that expanded image portion from the portion of the limited image in the limited field of view 2104. For example, an outline of the shape of the limited field of view 2104 may be superimposed on the expanded field of view 2102 when the expanded field of view 2102 is turned on. With reference to GUI 2100, for example, a circular outline of the limited field of view 2104 may be superimposed on the expanded field of view. As another example, a portion of the expanded image may be shaded, grayed out, blurred, presented in a lighter color, etc.

[0094] FIG. 22 shows an example of a GUI 2200 including various views. A live endoscopy view 2202 presented in the left half of the GUI 2200 may default to the limited view 2104 of FIG. 21 . At the top right is a pathway view 2204, and at the bottom right is a CT view 2206. As shown, the pathway view 2204 may present a view of a 3D model of a luminal network (e.g., lungs), and the CT view 2206 may present a view of a CT image. The pathway view 2204 may use a model generated based on multiple CT images as its 3D model. For example, the 3D model may be preoperative model data generated based on the model data 91 of FIG. 20 . As shown, the pathway view 2204 and the CT view 2206 may present tip indicators 2208 a-b, pathway indicators 2210 a-b, and target indicator 2212 to assist in endoscopic navigation.

[0095] FIG. 23 shows an example GUI 2300 including multiple orientation indicators. The GUI 2300 may include a selectable user interface (UI) element (e.g., a toggle button) 2302 that turns the presentation of the orientation indicators on or off. When the UI element 2302 is configured to present orientation indicators, each of the presented orientation indicators may be displayed to inform the operator of the current orientation of the endoscope. In this and other examples described herein, the orientation of the scope may be determined based on one or more tracking systems associated with the scope that can detect the orientation of the scope as it traverses the patient's anatomy. Examples of tracking systems include electromagnetic (EM) tracking systems, fiber optic shape sensors, image sensors, and other systems that include sensors for determining the orientation of the scope (or the tip of the scope) relative to the patient's anatomy. A tracking system may also use a combination of sensor inputs to determine the orientation of the sensor within the patient's anatomy. An example of a tracking system is described above with reference to FIG. 20 and may be used to determine the orientation of the scope to facilitate the orientation indicators or compasses described herein.

[0096] The first orientation indicator 2310 may be presented outside or adjacent to the endoscopic view (e.g., endoscopic view 2202 in FIG. 22 ). The endoscopic view may be a live image feed of the patient's anatomy captured by the scope as it traverses the patient's anatomy. The first orientation indicator 2310 may provide a reference axis / plane 2312 for one or more anatomical directions, such as the anterior / posterior axis (denoted by “A” and “P”) or the medial / lateral axis (denoted by “M” and “L”). The first orientation indicator 2310 may further provide a direction indicator 2314, which may be 2D, such as an arrow or directional needle, or 3D, such as a cone as shown, to indicate the orientation of the endoscope associated with the endoscopic view. For example, in the GUI 2300, the first orientation indicator 2310 may inform the operator that the top of the endoscopic view is toward the front, the bottom is toward the back, the left is toward the inside, and the right is toward the outside. In some other embodiments, when the endoscope adjusts its orientation, the direction indicator 2314, the reference axis / plane 2312, or both may adjust a corresponding amount in the display. For example, when the endoscope "rolls" along its longitudinal axis, the direction indicator 2314 may roll a corresponding roll amount in the display while the reference axis / plane 2312 remains fixed.

[0097] The reference axes / planes 2312 and their one or more anatomical directions may be determined pre-operatively or intra-operatively. For example, pre-operatively, a known placement of the endoscope relative to a known positioning of the patient on a support platform (e.g., support platform 38 of FIG. 5 ) may inform the GUI 2300 of a default set of reference axes / planes 2312. As another example, pre-operatively or intra-operatively, an operator may manually input or adjust the reference axes / planes 2312. As yet another example, pre-operatively or intra-operatively, the reference axes / planes 2312 may be automatically determined by performing a calibration procedure with the endoscope.

[0098] An exemplary calibration or alignment procedure can include advancing the endoscope into and then out of at least two known fiducial branches of the lumen network. This procedure can facilitate alignment of the frame of reference of the tracking system with the frame of reference of the patient's anatomy. For example, an operator may advance the bronchoscope into the left bronchus, retract it from the left bronchus, and advance the bronchoscope into the right bronchus. This calibration procedure can provide multiple fiducial points that can be mapped to at least one axis and provide a zero point on the axis and an anatomical orientation based on the zero point. Use of the calibration procedure can advantageously enable accurate determination of the reference axis / plane 2312 and its anatomical orientation without the errors often associated with default or manual fiducial configurations.

[0099] The second orientation indicator 2320 shows a 2D orientation indicator represented as a box. The second orientation indicator 2320 can inform the operator that the path view (e.g., path view 2204 of FIG. 22) is viewed from front to back by presenting an "A" while hiding a "P." In some implementations, the second orientation indicator 2320 can be tilted a corresponding amount when the path view as shown is rotated clockwise or counterclockwise. Note that while a box is shown, any indicating shape can be used.

[0100] The third orientation indicator 2330 shows a 3D orientation indicator represented as a cube. By presenting an upside-down "A" and an upright "S," the third orientation indicator 2330 can inform the operator that the simulated field of view 2306 predicted based on the preoperative model is viewed from anterior to posterior and superior to inferior. Additionally, the subtle display of an outer "L" informs the operator that the simulated field of view 2306 is viewed from an angle that is slightly lateral toward an inward orientation. Thus, the angulation of each face of the cube and which letter is visible at which orientation can reflect the orientation of the endoscope within the simulated field of view 2306. Note that while a cube is shown, any referent can be used.

[0101] As shown by the orientation indicators 2310, 2320, 2330 and their respective fields of view, each field of view may be presented with a corresponding orientation indicator that independently signals the orientation of the image presented within the field of view. For example, the second orientation indicator 2320 reflects a first orientation of the path field of view, and the third orientation indicator 2330 reflects a second orientation of the simulated field of view 2330 that is different from the first orientation.

[0102] In this disclosure, orientation indicators, including the illustrated orientation indicators 2310, 2320, 2330, may be referred to as a "compass." While the structure and elements of an orientation indicator may differ from a traditional physical compass, this term should be interpreted broadly herein to include any UI element that can assist an operator in determining the orientation of the endoscope relative to the displayed field of view. In some embodiments, all or a portion of the compass may be presented adjacent to or overlaid within the peripheral boundary of the endoscopic view to provide an instinctive association between the compass and the endoscopic view, as shown and described in more detail with respect to FIGS. 24-25 .

[0103] In some embodiments, an articulation indicator 2340 may be presented adjacent to or within the endoscopic view. The articulation indicator 2340 may indicate in which anatomical direction the endoscope is currently articulating or toward which anatomical direction it is articulating. If the endoscope has multiple separately articulatable components, such as a sheath portion and a leader portion, a separate articulation indicator 2342, 2346 may be provided for each articulatable component. As shown, the separate articulation indicators 2342, 2346 may be presented as variable length arcs having a center positioned toward the direction of articulation and a length corresponding to the magnitude of articulation. The maximum available articulation may be indicated by a first set of endpoints 2344a-b for the sheath articulation indicator 2342 and a second set of endpoints 2348a-b for the leader articulation indicator 2346. Although arcs of various lengths are shown as an example, other graphical elements, such as arrows of various thicknesses, can represent joint motion vectors (i.e., the direction and magnitude of joint motion). Another embodiment of the joint motion indicator 2340 is described in further detail with respect to FIG.

[0104] 24 shows an example GUI 2400 that includes an example 2D compass 2402. The 2D compass 2402 can surround the periphery of an endoscopic view (e.g., endoscopic view 2202 of FIG. 22) to provide the operator with a better sense of direction as the operator navigates intraoperatively. The 2D compass 2402 can include letters at different positions along the border of the compass to indicate corresponding anatomical directions such as superior / inferior, anterior / posterior, medial / lateral, proximal / distal, central / peripheral, superficial / deep, and dorsal / ventral. In the exemplary 2D compass 2402, the anterior / posterior and medial / lateral anatomical directions are provided with the letters A / P and M / L, respectively, and are referred to herein as anatomical direction indicators 2406a-d. Each pair of anatomical direction indicators 2406a-d may be presented at opposite ends of the peripheral boundary. The 2D compass 2402 may provide an angle index / scale 2404 around the peripheral boundary of the endoscopic view to indicate regular angular intervals, such as 5 degrees, 10 degrees, 15 degrees, 20 degrees, 30 degrees, etc. In some embodiments, all or at least a portion of the 2D compass 2402, such as the anatomical direction indicators 2406a-d or the angle index / scale 2404, may be superimposed on the endoscopic view.

[0105] The anatomical direction indicators 2406a-d and the angle index / scale 2404 can help the operator understand the current orientation of the endoscope capturing the endoscopic view image. The anatomical direction indicators 2406a-d of the 2D compass 2402 can inform the operator that "up" articulates the endoscope forward, "down" articulates it backward, "left" articulates it toward the inside of a bronchus, and "right" articulates it toward the outside of another bronchus. When the endoscope is "rolled," in some embodiments, the image may rotate clockwise or counterclockwise based on the angular change in "roll," while the anatomical direction indicators 2406a-d and the angle index / scale 2404 remain fixed. In other embodiments, the image may remain fixed while the anatomical direction indicators 2406a-d and the angle index / scale 2404 rotate based on the angular change in "roll."

[0106] In one example, the 2D compass 2402 is always displayed (e.g., always on) for the operator over the endoscopic view during the procedure. In another example, the compass 2404 can be manually switched on or off by the operator using the switch 2410, for example, by selecting "off" or "2D." In another example, the compass 2402, or portions thereof, may be automatically switched on or off depending on the orientation of the endoscope. For example, when the endoscope or camera is detected to be exactly or approximately parallel to a particular axis (e.g., parallel to an axis in Cartesian space), one or more other perpendicular or parallel axes may automatically appear or disappear. In some embodiments, the 2D compass 2402 may automatically be switched on when the endoscope or camera is detected to be exactly or approximately parallel to a particular axis.

[0107] Although four letters are shown on the 2D compass 2402 in Figure 24, in another example, any other number of letters may be shown. For example, three letters may be shown if the operator navigates the endoscope to a corner position and there is a dead end in a particular navigation direction. In another example, a fifth letter may appear / disappear in the center of the endoscopic view with softer or less accentuated shading as the operator moves in the direction of an otherwise invisible parallel axis; for example, in Figure 24, an "M" may appear as the operator moves inward, and an "L" may appear as the operator moves outward.

[0108] FIG. 25 shows an example GUI 2500 including an example 3D compass 2502. The 3D compass 2502 may include indicators of a 2D compass (e.g., 2D compass 2402 of FIG. 24) and some or all of its features. The 3D compass 2502 may be provided around the peripheral boundary of an endoscopic view (e.g., endoscopic view 2202 of FIG. 22) or overlaid on the endoscopic view to provide the operator with a better sense of direction as the operator navigates the endoscope intraoperatively. For example, the 3D compass 2502 may include 2D anatomical direction indicators (e.g., anatomical direction indicators 2406a-d of FIG. 24) at different positions along the peripheral boundary of the compass to indicate corresponding anatomical directions such as anterior / posterior, medial / lateral, etc. Additionally, the 3D compass 2502 may include 3D anatomical direction indicators 2506a-b at different positions within the endoscopic view to indicate corresponding anatomical directions such as upper / lower, indicated by the letters “S” and “I,” respectively. In the illustrated example, a 3D compass is overlaid on the endoscopic view so that the endoscopic view can be seen through the compass.

[0109] 3D anatomical direction indicators 2506a-b may be positioned on the endoscopic view based on the pitch and yaw of the tip of the endoscope. In some embodiments, bendable lines 2504a-b may be presented to indicate the pitch and yaw of the endoscope. For example, one or more bendable horizontal lines, which may also be referred to as "lines of latitude," may be generated to connect each latitude of a 2D axis, such as the M / L axis as shown, to indicate the yaw of the endoscope. Similarly, one or more bendable vertical lines, which may also be referred to as "lines of longitude," may be generated to connect each longitude of another 2D axis, such as the A / P axis as shown, to indicate the pitch of the endoscope. Note that while the bendable lines are "bendable," they need not necessarily be curved in all cases. For example, if the pitch or yaw of the endoscope is precisely aligned with the 2D axis, the bendable horizontal line 2504a or the bendable vertical line 2504b may be straight.

[0110] For simplicity, the exemplary 3D compass 2502 as shown is shown showing only a bendable horizontal line 2504a and a bendable vertical line 2504b. Each of the bendable lines 2504a-b can surround the endoscopic view as if the endoscopic view were a sphere. In some embodiments, each of the bendable lines 2504a-b can represent a proximal portion of the sphere and a distal portion of the sphere by distinguishing various aspects of the bendable lines 2504a-b. As an example, the bendable lines 2504a-b are shown as solid lines for the proximal portions and dashed lines for the distal portions. In some other implementations, other distinguishing characteristics may be used, such as line thickness (e.g., a thicker proximal portion compared to the distal portion), line color, line contrast, line softness, or line presence (e.g., only one of the proximal portion or the distal portion is presented). In one example, one or more characters displayed in the actual field of view that are further from the user may be de-emphasized or provided with lower brightness or contrast.

[0111] The 3D anatomical direction indicators 2506a-b may be positioned at the intersections of the bendable lines 2504a-b. As shown, the bendable horizontal line 2504a and the bendable vertical line 2504b intersect at two points: a proximal intersection point on a proximal portion of the sphere and a distal intersection point on a distal portion of the sphere. The axis connecting the two points may be the upper / lower axis of the 3D compass 2502. Thus, an anatomical direction indicator (e.g., "S") may be positioned at the proximal intersection point, and another anatomical direction indicator (e.g., "I") may be positioned at the distal intersection point.

[0112] Although anatomical indicators (S / I, A / P, M / L) are used in FIGS. 24-25, it should be understood that other symbols or indicators may be used to inform the user of anatomical directions, and in some cases other anatomical indicators along other axes may be used, including proximal / distal, central / peripheral, superficial / deep, dorsal / ventral, etc.

[0113] 24-25 , in addition to the anatomical direction indicators (S / I, A / P, M / L) 2406a-d, 2506a-b on the boundaries of the 2D compass 2402 and the 3D compass 2502, respectively, a target site indicator 2408, FIG. 24 , may also be displayed on the compass 2402, 2502 to indicate the location of the target site within the endoscopic view. This target indicator 2408 may be an icon, shape, mask, symbol, letter, or some other indicator on the peripheral boundary of the compass 2402, 2502, or may be displayed within the actual field of view of the endoscopic view. In some embodiments, the target site indicator 2408 may become larger, dimmer, brighter, or may change color, contrast, etc., based on the distance of the endoscope to the target site. The target site indicator 2408 can assist the operator in identifying the direction in which to advance the endoscope to reach the target anatomical site (e.g., a nodule or legion).

[0114] In some embodiments, one or more additional visual indicators may be presented to indicate to the operator whether they are properly centering the endoscope. By way of non-limiting example, the visual indicator may be a dashed or solid line (not shown), an icon, a shape, a symbol, text, or other graphical element. It may be advantageous for the navigation of the endoscope to be centered relative to the airway or anatomical channel through which the tool or endoscope is being driven, referred to as centering. An indicator such as a bull's eye may show the user how best to drive the endoscope or tool to achieve centering.

[0115] 24-25, the compass is designed to provide users with a better sense of direction as they navigate and adjust the scope tip position during a procedure. For example, when navigating around the lungs, users can become disoriented and face significant cognitive strain in determining the anatomical direction the scope tip is pointing. This is especially important for clients who use advanced imaging and need to know how to adjust the scope position based on the imaging results. The purpose of the compass is to reduce the cognitive strain on the user so that they can easily recognize which anatomical directions—up / down / left / right / insertion / retraction—correspond on the controller. This is achieved by providing users with the corresponding anatomical directions (up / down, anterior / posterior, medial / lateral) in the live endoscopic view.

[0116] In some embodiments, an orientation indicator (e.g., orientation indicators 2310, 2320, 2330 in FIG. 22) or a compass (e.g., compass 2402, 2502 in FIGS. 24-25) may always be presented to the user. In other embodiments, the orientation indicator or compass can be manually switched on or off using a selectable UI element (e.g., toggle button 2302 in FIG. 23 or switch 2410 in FIG. 24). The selectable UI element may also be used to transition the GUI from presenting an overlaid compass as shown in FIGS. 24-25 to an adjacent orientation indicator 2310 as shown in FIG. 23, or vice versa.

[0117] 26 illustrates an example articulation scenario 2600 associated with articulation indicators 2604, 2606. The articulation indicators 2600 may be presented adjacent to a peripheral boundary (e.g., of the articulation indicator 2340 of FIG. 23 ), surround a peripheral boundary as shown in example articulation scenarios 2600(a)-(e), or be overlaid on an endoscopic view (e.g., the endoscopic view 2202 of FIG. 22 ). The articulation indicators 2600 may inform the operator of various articulation conditions, including at least one selected articulatable component, a current articulation of the articulatable component, a maximum available articulation of the articulatable component, an articulation potential of the articulatable component, etc.

[0118] The articulation indicator may include a component selector 2602 that may select one or more articulatable components of the endoscope for articulation. In a first scenario 2600(a), the component selector 2602 selects "paired" (both a sheath portion and a leader portion) for articulation. In a second scenario 2600(b), the component selector 2602 selects "sheath" (sheath portion only) for articulation. In a third scenario 2600(c), a fourth scenario 2600(d), and a fifth scenario 2600(e), the component selector 2602 selects "scope" (leader portion only) for articulation.

[0119] Each of the scenarios 2600(a)-(e) can have a sheath articulation indicator 2604 and a scope articulation indicator 2606. The illustrated example scenarios 2600(a)-(e) present the sheath articulation indicator 2604 as surrounding the scope articulation indicator 2606, which may be more instinctive but may be reversed in some implementations. As shown in scenarios 2600(a)-(e), one or more selected articulation indicators may be highlighted with greater contrast, highlighting, gradients, patterns, colors, etc., compared to one or more non-selected articulation indicators. For example, in the first scenario 2600(a), both the sheath articulation indicator 2604 and the scope articulation indicator 2606 are highlighted. As another example, in the second scenario 2600(b), only the sheath articulation indicator 2604 is highlighted. As yet another example, in the third, fourth, and fifth scenarios 2600(c)-(e), only the scope articulation indicator 2606 is highlighted.

[0120] The joint motion indicators 2604, 2606 can be divided into multiple sections (e.g., quadrants as shown) and positioned near the peripheral boundary of a compass (e.g., the 2D compass 2402 in FIG. 23 or the 3D compass 2502 in FIG. 25). In some implementations, the joint motion indicators 2604, 2606 can be divided into more or fewer sections, such as five, six, eight, twelve, or any number of sections. Each section can represent joint motion toward a direction encompassed by the section. Furthermore, each section can represent a magnitude of the joint motion (e.g., a degree of joint motion). Thus, a section can represent a joint motion vector having a direction and magnitude relative to the operator.

[0121] Continuing with the exemplary scenarios 2600(a)-(e) divided into quadrants, each section can be represented by a corresponding arc. In the exemplary scenarios 2600(a)-(e), each arc is aligned with an anatomical direction (e.g., A / P, M / L), but it should be noted that the alignment is for ease of explanation. The positioning of the arcs may depend on the endoscope control scheme, and the positioning of the anatomical directions may depend on one or more reference axes determined in the calibration procedure described in FIG. 23 . In other words, the positioning of the arcs can be based on the control characteristics of the endoscope, and the positioning of the anatomical directions can be based on the placement of the patient or the lumen network within the patient. In some embodiments, when the endoscope is “rolled,” the arcs may roll a corresponding amount. In some other embodiments, the arcs may remain fixed during endoscope roll.

[0122] The length of the arc can indicate the maximum available articulation of the endoscope along the arc. As the endoscope is articulated along the arc, the magnitude of articulation can be represented by filling the arc until the maximum available articulation is reached, as shown. For example, in first scenario 2600(a), both the sheath articulation indicator 2604 and the scope articulation indicator 2606 indicate articulation toward the forward and outward direction, as indicated by the approximately half-filled articulation indicators 2604a, 2604a′, 2606a, 2606a′. However, the endoscope is not articulated toward the rear, as indicated by the empty articulation indicators 2604a″, 2606a″. Similarly, the endoscope is not articulated toward the medial direction. The approximately half-filled articulation indicators 2604a, 2604a′, 2606a, 2606a′ indicate that the endoscope is articulated toward the forward and outward direction to approximately half of the maximum available articulation. Thus, the articulation indicators 2604, 2606 can inform the operator of the current articulation vectors of a portion of the sheath and a portion of the leader.

[0123] In second scenario 2600(b), only the sheath portion has been selected for articulation, as indicated by highlighted sheath articulation indicator 2604b, as opposed to de-highlighted scope articulation indicator 2606b. Second scenario 2600(b) has the same articulation vector as first scenario 2600(a). If the endoscope is articulated further forward based on the selection, sheath articulation indicator 2604b will fill further, while scope articulation indicator 2606b remains unchanged.

[0124] In a third scenario 2600(c), only a portion of the leader has been selected for articulation, as indicated by the highlighted leader articulation indicator 2606c, as opposed to the de-highlighted sheath articulation indicator 2604c. The third scenario 2600(c) has a first articulation vector of a portion of the sheath pointing rearward and inward, and a second articulation vector of a portion of the leader pointing forward and outward. If the endoscope is articulated further forward based on the selection, the scope articulation indicator 2606c will fill further, while the sheath articulation indicator 2604c remains unchanged.

[0125] In a fourth scenario 2600(d), the first fully filled scope articulation indicator 2606d may indicate maximum articulation, so that a portion of the leader may not be articulated further forward. Similarly, the second fully filled scope articulation indicator 2606d' may indicate maximum articulation, so that a portion of the leader may not be articulated further outward. In some embodiments, when maximum articulation is reached, one or more characteristics (e.g., color, thickness, pattern, etc.) of the articulation indicator may be changed to indicate that the articulatable component is at its maximum articulation.

[0126] In a fifth scenario 2600(e), a portion of the leader has not yet reached its maximum forward articulation because the scope articulation indicator 2606e is not fully filled. However, here, the portion of the leader cannot continue to articulate forward because such articulation may be obstructed by a wall or object in the forward anatomical direction. In such a scenario, attempting to articulate the portion of the leader further forward may apply excessive force, causing the endoscope to buckle or the pull wires to break under stress. Undesirable stress may be indicated by the scope articulation indicator 2606e exhibiting enhancement near its outer edge (e.g., a change in a characteristic of the articulation indicator 2606e). For example, the articulation indicator 2606e is shown as changing color near its outer edge.

[0127] Note that while Figure 26 and its scenarios 2600(a)-(e) show arc-based joint motion indicators 2604, 2606 representing joint motion vectors, other implementations are possible. For example, a vector centered on the joint motion indicator 2340 of Figure 23 or the endoscopic view may be considered.

[0128] Additionally, up to this point, orientation indicators (e.g., orientation indicator 2310 in FIG. 23 and compasses 2402, 2502 in FIGS. 24-25) have been described as being applicable to live endoscopic views. However, it is contemplated that orientation indicators and their various features may be applied to previously captured endoscopic images. For example, the present disclosure may be applied to a sequence of previously captured and stored endoscopic images to add orientation indicators to the post-capture endoscopic images.

[0129] 3. Implementation systems and terminology. The implementations disclosed herein provide systems, methods, and devices for a user interface for navigating anatomical channels in a medical procedure.

[0130] It should be noted that, as used herein, the terms "couple," "coupled," "coupled," or other variations of the word coupled, can indicate either an indirect connection or a direct connection. For example, when a first component is "coupled" to a second component, the first component can be either indirectly connected to the second component through another component, or directly connected to the second component.

[0131] The functions described herein may be stored as one or more instructions on a processor-readable medium or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), or other optical, magnetic, or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Note that computer-readable media may be tangible and non-transitory. As used herein, the term "code" may refer to software, instructions, code, or data that is executable by a computing device or processor.

[0132] The methods disclosed herein include one or more steps or actions for achieving the described method. Method steps and / or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for the proper operation of the described method, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0133] As used herein, the term "plurality" refers to two or more. For example, a plurality of components refers to two or more components. The term "determining" encompasses a wide variety of acts, and thus, "determining" can include calculating, computing, processing, deriving, investigating, looking up (e.g., referring to a table, database, or another data structure), ascertaining, and the like. "Determining" can also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" can also include resolving, selecting, electing, establishing, and the like.

[0134] The phrase "based on" does not mean "based only on," unless expressly specified otherwise. In other words, the phrase "based on" describes both "based only on" and "based at least on."

[0135] The foregoing description of the disclosed implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the scope of the present invention. For example, those skilled in the art will recognize that many corresponding alternative and equivalent structural details may be employed, such as similar methods for fastening, mounting, coupling, or engaging tool components, equivalent mechanisms for producing specific actuation motions, and equivalent mechanisms for delivering electrical energy. Thus, the present invention is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0136] Further embodiments Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein may be performed in a different order, added, merged, or omitted entirely. Thus, in a particular embodiment, not all of the described acts or events are necessary to perform a process.

[0137] In particular, conditional language used herein, such as "can," "could," "might," "may," "eg," and the like, unless specifically stated otherwise or understood otherwise within the context in which it is used, is intended to have its ordinary meaning and is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not. Thus, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps are included in or performed in any particular embodiment, with or without author input or prompting. Terms such as "comprising," "including," "having," and the like are synonymous and used in their ordinary sense, inclusively in a non-limiting manner, and do not exclude additional elements, features, acts, operations, etc. Also, when the term "or" is used, for example, to connect a list of elements, the term "or" is used in its inclusive sense (and not its exclusive sense) to mean one, some, or all of the listed elements. Unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood in the context as it is commonly used to convey that an item, term, element, etc. can be either X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z, each be present.

[0138] In the foregoing description of the embodiments, it should be understood that various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in understanding one or more of the various inventive aspects. However, this method of disclosure should not be interpreted as reflecting an intention that any claim requires more features than are expressly recited in that claim. Moreover, any component, feature, or step illustrated and / or described in a particular embodiment herein may be applied to or used in conjunction with any other embodiment. Moreover, no component, feature, step, or group of components, features, or steps is necessary or essential for each embodiment. Accordingly, it is intended that the scope of the invention(s) disclosed herein and claimed below should not be limited by the specific embodiments described above, but should be determined solely by a fair reading of the following claims.

[0139] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply any physical characteristics or ordering. Thus, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not necessarily indicate a priority or order of the element with respect to any other elements, but rather may generally distinguish the element from other elements having a similar or identical name (apart from the use of the ordinal terminology). Additionally, as used herein, the indefinite articles ("a" and "an") may indicate "one or more" rather than "one." Furthermore, an action performed "based on" a condition or event may also be performed based on one or more other conditions or events not expressly recited.

[0140] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It is further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and should not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

[0141] Spatially relative terms such as "outside," "inside," "upper," "lower," "below," "upper," "vertical," "horizontal," and similar terms may be used herein for ease of description to describe the relationship between one element or component and another element or component as illustrated in the figures. It should be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device shown in the figures were inverted, a device positioned "below" or "under" another device would be disposed "above" another device. Thus, the illustrative term "below" can include both lower and upper positions. Devices may also be oriented in other directions, and thus spatially relative terms may be interpreted differently depending on the orientation.

[0142] Unless otherwise specified, comparative and / or quantitative terms such as "less," "more," "greater than," etc. are intended to encompass the notion of equality. For example, "less" can mean "less than" in the strict mathematical sense as well as "less than or equal to."

[0143] [Embodiment] (1) A health care system: Scope and The display and one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions causing the one or more processors to: displaying an endoscopic view of the anatomical site derived from the scope on the display; and a memory for displaying a compass overlaid on the endoscopic view. (2) A medical system as described in embodiment 1, wherein the compass is configured to inform the user of the orientation of the scope relative to the patient's anatomical structure. (3) The medical system of embodiment 1, wherein the stored instructions further cause the one or more processors to determine orientation based on a tracking sensor coupled to the scope. (4) A medical system as described in embodiment 1, wherein the compass is configured to be manually turned on or off using an input device coupled to the display. (5) A medical system as described in embodiment 1, wherein the compass is configured to be automatically switched on or off depending on at least one of the position, orientation, or direction of movement of the scope.

[0144] (6) Execution of the instructions further causes the one or more processors to: A medical system as described in embodiment 1, wherein in response to detecting that the movement of the scope is approximately parallel to an axis in Cartesian space, the compass is automatically switched on and displays other axes in the Cartesian space that are approximately perpendicular to the movement of the scope. (7) The medical system of embodiment 1, wherein displaying the compass includes displaying three indicators in response to the scope being at a corner of a path. (8) A medical system as described in embodiment 2, wherein displaying the compass includes displaying five indicators, at least one of the five indicators being configured to appear and disappear as the scope moves. (9) A medical system as described in embodiment 2, wherein displaying the compass includes displaying six indicators, at least one of the six indicators appearing with lower brightness or contrast than the other indicators of the six indicators. (10) The medical system of embodiment 1, wherein the instructions further cause the one or more processors to display a target site indicator on or adjacent to the compass that indicates the direction to navigate to reach the target.

[0145] (11) The instructions further cause the one or more processors to display an indicator on or adjacent to the compass that indicates to a user a center position within a path, which, when followed, provides navigation of the scope centered relative to the path. (12) A health care system, a scope configured to capture at least one endoscopic image; a display device configured to display the endoscopic image; and one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions causing the one or more processors to: determining an orientation of the scope relative to the endoscopic image; determining at least one reference axis associated with the lumen network, the reference axis indicating a first anatomical direction and a second anatomical direction; generating an orientation indicator based on the reference axis; and a memory containing instructions to cause the display to present the endoscopic image and the orientation indicator, the orientation indicator being overlaid on the endoscopic image. (13) The stored instructions, when executed by the one or more processors, cause the one or more processors to: generating anatomical direction indicators for the first anatomical direction and the second anatomical direction; 13. The medical system of claim 12, further comprising instructions to cause the anatomical direction indicators to be presented at opposite ends of the peripheral boundary of the orientation indicator. (14) The orientation indicator is a two-dimensional orientation indicator having a first axis and a second axis, the first axis is associated with an anterior / posterior anatomical direction and the second axis is associated with a medial / lateral anatomical direction; A medical system as described in embodiment 13, wherein the anterior / posterior anatomical directions and the medial / lateral anatomical directions are represented by anatomical direction indicators positioned on the peripheral boundaries of the two-dimensional orientation indicator. (15) The medical system of embodiment 14, wherein the anatomical direction indicator is a letter or symbol.

[0146] (16) The medical system of embodiment 14, wherein the orientation indicator is a three-dimensional orientation indicator further having a third axis, the third axis including an upper / lower anatomical direction based on the yaw and pitch of the scope. (17) A medical system as described in embodiment 16, wherein an anatomical direction indicator representing an upper / lower anatomical direction is superimposed on the endoscopic image. (18) The medical system of embodiment 17, wherein the three-dimensional orientation indicator includes a curved horizontal line connecting the anterior / posterior anatomical directions and a curved vertical line connecting the medial / lateral anatomical directions, and the intersection of the curved horizontal line and the curved vertical line represents the anatomical direction of the superior / inferior anatomical direction. (19) The medical system of embodiment 12, wherein the orientation of the scope is referenced based on a calibration procedure that includes navigating the scope to the left and right bronchi. (20) The stored instructions, when executed by the one or more processors, cause the one or more processors to: 13. The medical system of claim 12, further comprising instructions for switching the orientation indicator on or off from the display device.

[0147] (21) The medical system of embodiment 12, wherein the stored instructions further include instructions that, when executed by the one or more processors, cause the one or more processors to select between a two-dimensional orientation indicator or a three-dimensional orientation indicator. (22) The stored instructions, when executed by the one or more processors, cause the one or more processors to: generating a target site indicator; 13. The medical system of claim 12, further comprising instructions for overlaying the target site indicator on the endoscopic image. (23) The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining a distance from the tip of the scope to the target site; 23. The medical system of claim 22, further comprising instructions for adjusting the size of the target site indicator based on the distance. (24) The stored instructions, when executed by the one or more processors, cause the one or more processors to: further comprising instructions for generating an articulation indicator for an articulatable component of the scope; a length of the articulation indicator corresponds to a degree of articulation of the scope; A medical system as described in embodiment 12, wherein the position of the articulation indicator at the peripheral boundary of the endoscopic image indicates the direction of articulation of the scope. (25) The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining that the degree of joint motion is maximum joint motion; 25. The medical system of claim 24, further comprising instructions for modifying at least one characteristic associated with the joint motion indicator based on the determination.

[0148] (26) The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining that the articulatable component is prevented from further articulation; 25. The medical system of claim 24, further comprising instructions for modifying at least one characteristic associated with the lateral edge of the articulation motion indicator based on the determination. (27) A robot system, a robotic arm coupled to the scope; a viewer for displaying a view of an anatomical site derived from the scope; one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions comprising: displaying an endoscopic view on the viewer that includes the field of view; and a memory containing instructions for providing an electrical signal to display a two-dimensional compass or a three-dimensional compass configured to be switchable between being overlaid on the field of view or being provided as an icon adjacent to the field of view. (28) The robot system of embodiment 28, wherein the two-dimensional compass or the three-dimensional compass is configured to be resized and repositioned in response to being switched from being overlaid on the field of view to being presented as an icon. (29) A robot system comprising: a robotic arm coupled to the scope; a viewer for displaying a view of an anatomical site derived from the scope; one or more processors; a memory that stores instructions for execution by the one or more processors, the stored instructions including instructions for providing electrical signals for displaying an endoscopic view on the viewer that includes the field of view; A robotic system, wherein the field of view displayed on the endoscopic view can be switched between an enlarged view of the field of view and a limited view of the field of view. (30) The robotic system of embodiment 30, wherein the enlarged view is an uncropped view of the endoscope feed and the restricted view is a cropped view of the endoscope feed.

[0149] (31) The robotic system of embodiment 30, wherein the stored instructions further include instructions for providing an electrical signal to display a selectable icon adjacent to the displayed endoscopic view, the icon switching the displayed endoscopic view between the expanded field of view and the limited field of view in response to being selected by a user. (32) The robotic system of claim 30, wherein the memory instructions further include instructions for automatically switching the displayed endoscopic view between the limited field of view and the expanded field of view in response to a detected condition. (33) The detected state is Detecting an obstacle or blockage in a navigation path of the scope; detecting mucus within the navigation path of the scope; Detecting adjacent paths that are blocking navigation; 34. The robot system of claim 33, further comprising at least one of detecting that a route of interest to the user is not displayed. (34) The robotic system of embodiment 30, wherein the memory instructions further include instructions for simulating a portion of an image and including the portion of the simulated image in the expanded field of view to modify the shape of the endoscopic view.

Claims

1. 1. A healthcare system comprising: Scope and The display and one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions causing the one or more processors to: displaying an endoscopic view of the anatomical site derived from the scope on the display; and a memory for displaying a compass overlaid on the endoscopic view.

2. The medical system of claim 1 , wherein the compass is configured to inform a user of the orientation of the scope relative to a patient's anatomy.

3. The medical system of claim 1 , wherein the stored instructions further cause the one or more processors to determine an orientation based on a tracking sensor coupled to the scope.

4. The medical system of claim 1 , wherein the compass is configured to be manually turned on or off using an input device coupled to the display.

5. The medical system of claim 1 , wherein the compass is configured to be automatically switched on or off depending on at least one of a position, an orientation, or a direction of movement of the scope.

6. Execution of the instructions further causes the one or more processors to:

10. The medical system of claim 1, wherein in response to detecting that movement of the scope is approximately parallel to an axis in Cartesian space, the compass is automatically switched on to display another axis of the Cartesian space that is approximately perpendicular to the movement of the scope.

7. The medical system of claim 1 , wherein the displaying the compass includes displaying three indicators in response to the scope being at a corner of a path.

8. 3. The medical system of claim 2, wherein displaying the compass includes displaying five indicators, at least one of the five indicators configured to appear and disappear as the scope moves.

9. 3. The medical system of claim 2, wherein displaying the compass includes displaying six indicators, at least one of the six indicators appearing with a lower brightness or contrast than other indicators of the six indicators.

10. 10. The medical system of claim 1, wherein the instructions further cause the one or more processors to display a target site indicator on or adjacent to the compass that indicates a direction to navigate to reach the target.

11. 10. The medical system of claim 1, wherein the instructions further cause the one or more processors to display an indicator on or adjacent to the compass that indicates to a user a center position within a path that, when followed, provides navigation of the scope centered relative to the path.

12. 1. A health care system comprising: a scope configured to capture at least one endoscopic image; a display device configured to display the endoscopic image; and one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions causing the one or more processors to: determining an orientation of the scope relative to the endoscopic image; determining at least one reference axis associated with the lumen network, the reference axis indicating a first anatomical direction and a second anatomical direction; generating an orientation indicator based on the reference axis; and a memory containing instructions to cause the display to present the endoscopic image and the orientation indicator, the orientation indicator being overlaid on the endoscopic image.

13. The stored instructions, when executed by the one or more processors, cause the one or more processors to: generating anatomical direction indicators for the first anatomical direction and the second anatomical direction; The medical system of claim 12 , further comprising instructions to cause the anatomical direction indicators to be presented at opposite ends of a peripheral boundary of the orientation indicator.

14. the orientation indicator is a two-dimensional orientation indicator having a first axis and a second axis; the first axis is associated with an anterior / posterior anatomical direction and the second axis is associated with a medial / lateral anatomical direction; 14. The medical system of claim 13, wherein the anterior / posterior anatomical directions and the medial / lateral anatomical directions are represented by anatomical direction indicators located on a peripheral boundary of the two-dimensional orientation indicator.

15. The medical system of claim 14 , wherein the anatomical direction indicator is a letter or symbol.

16. 15. The medical system of claim 14, wherein the orientation indicator is a three-dimensional orientation indicator further having a third axis, the third axis including a superior / inferior anatomical direction based on yaw and pitch of the scope.

17. The medical system of claim 16 , wherein an anatomical direction indicator representing superior / inferior anatomical direction is superimposed on the endoscopic image.

18. 18. The medical system of claim 17, wherein the three-dimensional orientation indicator includes a curved horizontal line connecting the anterior / posterior anatomical directions and a curved vertical line connecting the medial / lateral anatomical directions, and an intersection of the curved horizontal line and the curved vertical line represents the anatomical direction of the superior / inferior anatomical direction.

19. 13. The medical system of claim 12, wherein the orientation of the scope is referenced based on a calibration procedure that includes navigating the scope to a left bronchus and a right bronchus.

20. The stored instructions, when executed by the one or more processors, cause the one or more processors to: The medical system of claim 12 , further comprising instructions to cause the orientation indicator to be switched on or off from the display device.

21. 13. The medical system of claim 12, wherein the stored instructions further comprise instructions that, when executed by the one or more processors, cause the one or more processors to select between a two-dimensional orientation indicator or a three-dimensional orientation indicator.

22. The stored instructions, when executed by the one or more processors, cause the one or more processors to: generating a target site indicator; The medical system of claim 12 , further comprising instructions for overlaying the target site indicator on the endoscopic image.

23. The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining a distance from the tip of the scope to the target site; 23. The medical system of claim 22, further comprising instructions for adjusting a size of the target site indicator based on the distance.

24. The stored instructions, when executed by the one or more processors, cause the one or more processors to: further comprising instructions for generating an articulation indicator for an articulatable component of the scope; a length of the articulation indicator corresponds to a degree of articulation of the scope; The medical system of claim 12 , wherein the position of the articulation indicator at a peripheral boundary of the endoscopic image indicates a direction of articulation of the scope.

25. The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining that the degree of joint motion is maximum joint motion; 25. The medical system of claim 24, further comprising instructions for altering at least one characteristic associated with the articulation indicator based on the determination.

26. The stored instructions, when executed by the one or more processors, cause the one or more processors to: determining that the articulatable component is prevented from further articulation; 25. The medical system of claim 24, further comprising instructions for altering at least one characteristic associated with a lateral edge of the articulation indicator based on the determination.

27. 1. A robotic system comprising: a robotic arm coupled to the scope; a viewer for displaying a view of an anatomical site derived from the scope; one or more processors; a memory storing instructions for execution by the one or more processors, the stored instructions comprising: displaying an endoscopic view on the viewer that includes the field of view; a memory containing instructions for providing an electrical signal to display a two-dimensional compass or a three-dimensional compass configured to be switchable between being overlaid on the field of view or provided as an icon adjacent to the field of view.

28. 28. The robotic system of claim 27, wherein the two-dimensional compass or the three-dimensional compass is configured to resize and reposition in response to being switched from overlaying the field of view to being presented as an icon.

29. 1. A robotic system comprising: a robotic arm coupled to the scope; a viewer for displaying a view of an anatomical site derived from the scope; one or more processors; a memory that stores instructions for execution by the one or more processors, the stored instructions including instructions for providing electrical signals for displaying an endoscopic view on the viewer that includes the field of view; A robotic system, wherein the field of view displayed on the endoscopic view can be switched between an enlarged view of the field of view and a limited view of the field of view.

30. 30. The robotic system of claim 29, wherein the enlarged view is an uncropped view of the endoscope feed and the restricted view is a cropped view of the endoscope feed.

31. 31. The robotic system of claim 30, wherein the stored instructions further include instructions for providing an electrical signal to display a selectable icon adjacent to the displayed endoscopic view, the icon responsive to being selected by a user to toggle the displayed endoscopic view between the expanded field of view and the limited field of view.

32. 31. The robotic system of claim 30, wherein the memory instructions further comprise instructions for automatically switching the displayed endoscopic view between the limited field of view and the expanded field of view in response to a detected condition.

33. The detected condition is Detecting an obstacle or blockage in a navigation path of the scope; detecting mucus within the navigation path of the scope; Detecting adjacent paths that are blocking navigation; and detecting that a path of interest to the user is not being displayed.

34. 31. The robotic system of claim 30, wherein the memory instructions further comprise instructions for simulating a portion of an image and including the portion of the simulated image in the expanded field of view to modify the shape of the endoscopic view.

Images