User interface for navigating anatomical channels in medical procedures

Abstract

A medical system includes: a sight glass configured to capture at least one endoscopic image; a display device configured to display the endoscopic image; one or more processors; and a memory storing instructions for execution by the one or more processors, the stored instructions including instructions that cause the one or more processors to determine an orientation of the endoscope associated with the endoscopic image. The stored instructions may include instructions for determining at least one reference axis associated with the lumen network. The reference axis may indicate a first anatomical direction and a second anatomical direction. The stored instructions may include instructions for generating an orientation indicator based on the reference axis. The stored instructions may include instructions for causing the display to present the endoscopic image and the orientation indicator, the orientation indicator overlying the endoscopic image.

Description

Background Art Technical Field

[0001] The systems and methods disclosed herein relate to navigating a medical device through a patient's anatomy, and more particularly to a user interface that assists a user in navigating a medical device.

[0002] Related technologies

[0003] Certain robotic medical procedures may involve the use of a shaft-type instrument (such as an endoscope) that can be inserted into a patient through an orifice (e.g., a natural orifice) and advanced to a target anatomical site. Such medical instruments may be articulable such that the tip and / or other portions of the shaft may be deflected in one or more dimensions using robotic control. During navigation of the medical instrument, various graphical elements may be provided on a display to assist the operator in advancing the medical instrument to a desired target location. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Various embodiments are depicted in the accompanying drawings for illustrative purposes and should not be construed as limiting the scope of the present invention in any way. In addition, the various features of the different disclosed embodiments may be combined to form additional embodiments as part of the present disclosure. Throughout the accompanying drawings, reference numerals may be reused to indicate the corresponding relationship between reference elements.

[0005] Figure 1 Embodiments of a cart-based robotic system arranged for use in diagnostic and / or therapeutic bronchoscopy procedures are illustrated.

[0006] Figure 2 Depicted Figure 1 Other aspects of the robotic system.

[0007] Figure 3 An example of a device arranged for ureteroscopy is shown. Figure 1 Implementation scheme of a robotic system.

[0008] Figure 4 Illustrated is arranged for use in a vascular procedure Figure 1 Implementation scheme of a robotic system.

[0009] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is illustrated.

[0010] Figure 6 Provided Figure 5 Alternative view of the robotic system.

[0011] Figure 7 An example system configured to stow a robotic arm is illustrated.

[0012] Figure 8 An embodiment of a table-based robotic system configured for use in a ureteroscopy procedure is illustrated.

[0013] Fig. 9 An embodiment of a table-based robotic system configured for laparoscopic procedures is illustrated.

[0014] Fig.10 Illustrated Figures 5 to 9 Embodiment of a stage-based robotic system with pitch and tilt adjustment.

[0015] Fig.11 Provided Figures 5 to 10 Detailed illustration of the interface between the stage and the column of a stage-based robotic system.

[0016] Fig.12 An alternative embodiment of a table-based robotic system is illustrated.

[0017] Fig.13 Illustrated Fig.12 End view of a stage-based robotic system.

[0018] Fig.14 An end view of a table-based robotic system with a robotic arm attached thereto is illustrated.

[0019] Fig.15 An exemplary instrument driver is illustrated.

[0020] Fig.16 An exemplary medical instrument having a pair of instrument drivers is illustrated.

[0021] Fig.17 An alternative design of an instrument driver and instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0022] Fig.18 An instrument having an instrument-based insertion architecture is illustrated.

[0023] Fig.19 An exemplary controller is illustrated.

[0024] Fig. 20 Depicted is a block diagram illustrating a positioning system according to an example embodiment that estimates Figures 1 to 10 The position of one or more elements of a robotic system, such as Figures 16 to 18 The location of the device.

[0025] Fig.21 Examples of graphical user interfaces are illustrated that provide either or both of a restricted field of view and an expanded field of view of intraoperative images captured by an endoscope.

[0026] Fig. 22 An example of a graphical user interface providing various views is illustrated.

[0027] Fig.23 An example of a graphical user interface including a plurality of orientation indicators is illustrated.

[0028] Fig.24 An example of a graphical user interface including an example of a two-dimensional (2D) compass is illustrated.

[0029] Fig.25 An example of a graphical user interface including an example of a three-dimensional (3D) compass is illustrated.

[0030] Fig.26 An example joint scene illustrating the association of joint motion indicators. DETAILED DESCRIPTION

[0031] The titles provided herein are for convenience only and do not necessarily affect the scope or meaning of the claimed invention. Although specific preferred embodiments and examples are disclosed below, the subject matter of the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and / or uses and modifications and equivalents thereof. Therefore, the scope of the claims that may appear herein is not limited to any of the specific embodiments described below. For example, in any method or process disclosed herein, the actions or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any specific disclosed sequence. The various operations may then be described as multiple discrete operations in a manner that may help understand certain embodiments; however, the order of description should not be interpreted as implying that these operations are dependent on the order. In addition, the structures, systems and / or devices described herein may be embodied as integrated components or separate components. For the purpose of comparing various embodiments, certain aspects and advantages of these embodiments are described. All such aspects or advantages may not necessarily be achieved by any particular embodiment. Therefore, for example, various embodiments may be performed by achieving or optimizing an advantage or a set of advantages taught herein without necessarily achieving other aspects or advantages that may also be taught or proposed herein.

[0032] Although certain spatially relative terms such as "external," "internal," "upper," "lower," "below," "above," "vertical," "horizontal," "top," "bottom," "outside," and similar terms are used herein to describe the spatial relationship of one device / element or anatomical structure to another device / element or anatomical structure, it should be understood that these terms are used herein for descriptive convenience to describe the positional relationship between elements / structures, such as with respect to the orientation shown in the accompanying drawings. It should be understood that spatially relative terms are intended to encompass different orientations of elements / structures in use or operation other than the orientation depicted in the accompanying drawings. For example, an element / structure described as being "above" another element / structure may represent positioning below or beside such other element / structure relative to a subject patient or an alternative orientation of the element / structure, and vice versa. It should be understood that spatially relative terms, including those listed above, may be understood relative to the corresponding illustrated orientations of the referenced drawings.

[0033] For the convenience of having devices, components, systems, features and / or modules with similar features in one or more aspects, specific reference numerals are reused in different figures of the group of figures of the present disclosure. However, with respect to any of the embodiments disclosed herein, the reuse of common reference numerals in the figures does not necessarily indicate that such features, devices, components or modules are the same or similar. On the contrary, those of ordinary skill in the art can be informed by the context that the use of common reference numerals can imply the degree of similarity between the cited themes. The use of specific reference numerals in the context of the description of a specific figure can be understood as relating to the device, component, aspect, feature, module or system identified in the specific figure, and does not necessarily relate to any device, component, aspect, feature, module or system identified by the same reference numeral in another figure. In addition, the aspects of the separate figures identified with common reference numerals can be interpreted as sharing characteristics or being completely independent of each other. In some contexts, the features associated with the separate figures identified by common reference numerals are unrelated and / or similar in at least some aspects.

[0034] The present disclosure provides systems, devices and methods for navigating instrument shafts (such as medical endoscopes) within a lumen network and monitoring instrument shaft articulation. The articulation of an instrument according to the present disclosure can be achieved by tensioning one or more tendons passing through the shaft of the instrument, which are referred to herein as "pull wires". With respect to the medical instruments described in the present disclosure, the term "instrument" is used according to its broad and common meaning and may refer to any type of tool, device, assembly, system, subsystem, equipment, component, etc. In some contexts herein, the term "device" may be used substantially interchangeably with the term "instrument". In addition, the term "axis" is used herein according to its broad and common meaning and may refer to any type of elongated cylinder, tube, scope (e.g., endoscope), prism (e.g., rectangular, oval, elliptical or rectangular prism), wire or the like, regardless of the cross-sectional shape. It should be understood that any reference to "axis" or "instrument axis" herein may be understood to refer to an endoscope.

[0035] 1. Overview .

[0036] Aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. In an endoscopy procedure, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0037] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm motions and positioning. Additionally, the system may provide the physician with the ability to perform procedures with improved ease of use such that one or more of the system's instruments may be controlled by a single user.

[0038] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other specific implementations of the disclosed concepts are possible, and various advantages can be achieved using the disclosed specific implementations. Titles are included herein for reference and to help locate the various sections. These titles are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout the specification.

[0039] A. Robotic System-Cart .

[0040] Robotically enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1An embodiment of a cart-based, robotically enabled system 10 arranged for use in diagnostic and / or therapeutic bronchoscopy procedures is illustrated. During a bronchoscopy procedure, the system 10 may include a cart 11 having one or more robotic arms 12 to deliver a medical instrument such as a steerable endoscope 13 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., the oral cavity of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned proximate to the patient's upper torso to provide access to the entry point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the access point. When performing gastrointestinal (GI) procedures using a gastroscope (a specialized endoscope for GI procedures), the bronchoscope may also be utilized. Figure 1 The arrangement in . Figure 2 An example embodiment of a cart is depicted in greater detail.

[0041] Continue to refer Figure 1 , once the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or in a combination thereof. As shown, the steerable endoscope 13 may include at least two telescopic portions, such as an inner guide (e.g., scope) portion and an outer sheath portion, each portion being coupled to a separate instrument driver from a set of instrument drivers 28, each of which is coupled to the distal end of a separate robotic arm. This linear arrangement of instrument drivers 28 that facilitates coaxial alignment of the guide portion with the sheath portion creates a "virtual track" 29 that can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual tracks described herein are depicted using dashed lines in the accompanying drawings, and thus the dashed lines do not depict any physical structure of the system. Translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical applications or physician preferences. For example, in bronchoscopy, the angle and positioning of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.

[0042] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.

[0043] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as, for example, a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove potential cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered in separate procedures. In these cases, the endoscope 13 can also be used to deliver a benchmark to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0044] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that may be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 30 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location to be out of the way during a procedure.

[0045] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions may control the entire system or a subsystem thereof. For example, when executed by a processor of a computer system, the instructions may cause components of the robotic system to actuate associated brackets and arm mounts, actuate a robotic arm, and control a medical device. For example, in response to receiving a control signal, a motor in a joint of a robotic arm may position the arm into a particular posture.

[0046] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to the system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.

[0047] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more mobile cart 11.

[0048] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0049] The tower 30 may also include a console 31, in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the system 10 is typically designed to provide both preoperative information and real-time information for robotic control and surgery, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the health or vital signs of the patient and the operation of the system, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, the console 30 is housed in a body separate from the tower 30.

[0050] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions may be coupled in separate wiring and connections. For example, while power may be provided to the cart via a single cable, support for control, optics, fluidics, and / or navigation may also be provided via separate cables.

[0051] Figure 2 Provided by Figure 1 Detailed illustration of an embodiment of a cart for a cart-based robotic-enabled system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2 The cradle 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The cradle 17 also includes a cradle interface 19 that allows the cradle 17 to translate vertically along the column 14.

[0052] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 contains a vertical translation interface to position and maintain the bracket at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet a variety of table heights, patient sizes, and physician preferences. Similarly, the separately configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0053] In some embodiments, the slot 20 may be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the internal cavity of the column 14 and the vertical translation interface when the carriage 17 is vertically translated. The slot cover can be deployed by a pair of spring reels positioned near the vertical top and bottom of the slot 20. The cover is coiled in the reel before deployment, and only extends and retracts from the coiled state of the cover as the carriage 17 vertically translates up and down. When the carriage 17 translates toward the reel, the spring loading of the reel provides the force to retract the cover into the reel, while also maintaining a tight seal when the carriage 17 translates away from the reel. The cover can be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure appropriate extension and retraction of the cover when the carriage 17 translates.

[0054] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).

[0055] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints and therefore provides seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different linkage mechanism positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0056] The cart base 15 balances the weight of the column 14, bracket 17, and arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart to be moved and / or fixed. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart to be easily moved around the room prior to a procedure. After reaching the proper position, the casters 25 can be secured using wheel locks to keep the cart 11 in place during the procedure.

[0057] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or a dual-purpose device, such as, for example, a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors, and coordinate information from sensors, as well as important patient statistics, such as respiration, heart rate, and / or pulse. The console 16 may be positioned and tilted to allow the physician to approach the console from the side of the column 14 opposite to the bracket 17. From this positioning, the physician can observe the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 for helping to manipulate and stabilize the cart 11.

[0058] Figure 3 An embodiment of a robotically enabled system 10 arranged for ureteroscopy is shown. In a ureteroscopy procedure, a cart 11 can be positioned to deliver a ureteroscope 32 (a procedure-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. In a ureteroscopy, it may be desirable to align the ureteroscope 32 directly with the patient's urethra to reduce friction and forces on sensitive anatomical structures in this area. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra along a virtual track 33 from the foot of the table.

[0059] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.

[0060] Figure 4An embodiment of a robotically enabled system similarly arranged for vascular procedures is shown. In a vascular procedure, the system 10 may be configured so that the cart 11 can deliver a medical device 34 (such as a steerable catheter) to an access point in the femoral artery in the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's legs and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 may be guided and inserted by translating the instrument drive 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0061] B. Robotic system - unit .

[0062] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment within the operating room by removing a cart, which allows for greater access to the patient. Figure 5 An embodiment of such a robotically enabled system is illustrated arranged for use in bronchoscopic procedures. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.

[0063] Figure 6An alternative view of the system 36 without the patient and medical device for discussion purposes is provided. As shown, the column 37 may include one or more brackets 43 shown as annular in the system 36, on which one or more robot arms 39 may be based. The bracket 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robot arm 39 may be positioned to reach the patient. The bracket 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robot arm 39 to access multiple sides of the table 38, such as, for example, both sides of the patient. In an embodiment with multiple brackets, the brackets may be positioned individually on the column and may translate and / or rotate independently of the other brackets. Although the bracket 43 does not need to be around the column 37 or even circular, the annular shape as shown facilitates the rotation of the bracket 43 around the column 37 while maintaining structural balance. The rotation and translation of the bracket 43 allows the system to align medical devices such as endoscopes and laparoscopes to different entry points on the patient. In other embodiments (not shown), the system 36 may include a patient table or bed with an adjustable arm support in the form of a pole or track extending alongside the patient table or bed. One or more robotic arms 39 may be attached to the adjustable arm support (e.g., via a shoulder with an elbow joint) that may be vertically adjusted. By providing vertical adjustment, the robotic arms 39 are advantageously able to be compactly stowed beneath the patient table or bed and subsequently raised during a procedure.

[0064] The arm 39 may be mounted on the carriage via a set of arm mounts 45 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 so that when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned on the same side of the table 38 (e.g., Figure 6 As shown), on the opposite side of the platform 38 (as shown Fig. 9 shown) or on an adjacent side of stage 38 (not shown).

[0065] The column 37 provides structural support for the stage 38 and provides a path for the vertical translation of the carriage. Internally, the column 37 can be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage based on the lead screw. The column 37 can also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0066] The base 46 has Figure 2The cart base 15 in the illustrated cart 11 functions similarly, accommodating heavier components to balance the table / bed 38, column 37, bracket 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. Casters deployed from the bottom of the table base 46 may extend in opposite directions on both sides of the base 46 and retract when the system 36 needs to be moved.

[0067] continue Figure 6 , the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in the previously disclosed embodiments, the tower can provide various support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower can also be movable to be positioned away from the patient, thereby improving the physician's access and eliminating clutter in the operating room. In addition, placing the components in the tower allows more storage space in the table base for potential stowage of the robotic arm. The tower may also include a main controller or console that provides a user interface such as a keyboard and / or suspension for user input, and a display screen (or touch screen) for preoperative and intraoperative information such as real-time imaging, navigation and tracking information. In some embodiments, the tower may also include a holder for a gas tank to be used for insufflation.

[0068] In some embodiments, the table base can fold up and store the robotic arm when not in use. Figure 7 A system 47 for stowing a robotic arm in an embodiment of a table-based system is illustrated. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the arm 50 around a column 53, and closed to stow the carriage, the arm mount, and the arm to protect them when not in use. The base cover 52 can be sealed along the edges of its opening with a membrane 54 to prevent dust and fluids from entering when closed.

[0069] Figure 8An embodiment of a robotically enabled table-based system configured for a ureteroscopy procedure is shown. During ureteroscopy, the table 38 may include a rotating portion 55 for positioning the patient at an angle to the column 37 and the table base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) so as to position the bottom portion of the rotating portion 55 away from the column 37. For example, the pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the bracket 35 (not shown) about the column 37, the robotic arm 39 can insert the ureteroscope 56 directly into the patient's groin area along the virtual track 57 to reach the urethra. During ureteroscopy, stirrups 58 may also be fixed to the rotating portion 55 of the table 38 to support the positioning of the patient's legs during the procedure and allow full access to the patient's groin area.

[0070] In a laparoscopic procedure, minimally invasive instruments may be inserted into the patient's anatomical structure through one or more small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instrument comprises an elongated rigid member, such as a shaft, for accessing the anatomical structure within the patient. After the patient's abdominal cavity is inflated, the instrument may be guided to perform a surgical or medical task, such as grasping, cutting, ablating, suturing, etc. In some embodiments, the instrument may comprise a scope, such as a laparoscope. Fig. 9 Embodiments of a robotically enabled table-based system configured for laparoscopic procedures are illustrated. Fig. 9 As shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 so that instruments 59 can be positioned using the arm mounts 45 to pass through minimal incisions on both sides of the patient to reach his / her abdominal cavity.

[0071] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. Fig.10 Embodiments of a robotically enabled medical system with pitch or tilt adjustment are illustrated. Fig.10 As shown, system 36 can accommodate the tilt of table 38 to position one portion of the table at a greater distance from the floor than another portion. Additionally, arm mount 45 can be rotated to match the tilt so that arm 39 maintains the same planar relationship with table 38. To accommodate steeper angles, column 37 can also include telescoping portion 60 that allows vertical extension of column 37 to prevent table 38 from contacting the floor or colliding with base 46.

[0072] Fig.11A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-to-table interface, each axis being actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment in one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2. In some embodiments, a ball joint can be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0073] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position (i.e., positioning the patient's lower abdomen higher from the floor than the patient's lower abdomen) for lower abdominal surgery. The Trendelenburg position causes the patient's internal organs to slide toward his / her upper abdomen by gravity, thereby clearing the abdominal cavity to allow minimally invasive tools to enter and perform lower abdominal surgical or medical procedures, such as laparoscopic prostatectomy.

[0074] Fig.12 and Fig.13 An isometric view and an end view of another embodiment of a table-based surgical robotic system 100 are illustrated. The surgical robotic system 100 includes a table 101 that can be configured to support one or more robotic arms (see, e.g., Fig.14 ) is provided with one or more adjustable arm supports 105 of the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports may be provided on opposite sides of the table 101. The adjustable arm support 105 may be configured so that it can be moved relative to the table 101 to adjust and / or change the positioning of the adjustable arm support 105 and / or any robotic arm mounted thereto relative to the table 101. For example, the adjustable arm support 105 may be adjusted in one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides high flexibility to the system 100, including the ability to easily stow the one or more adjustable arm supports 105 and any robotic arm attached thereto below the table 101. The adjustable arm support 105 may be raised from the stowed position to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 may be raised from the stowed position to a position above the upper surface of the table 101.

[0075] The adjustable arm support 105 may provide several degrees of freedom, including lift, lateral translation, tilt, etc. Fig.12 and Fig.13 In the illustrated embodiment, the arm support 105 is configured to have four degrees of freedom. Fig.12105 . The first degree of freedom allows the adjustable arm support 105 to be adjusted in the z-direction ("Z lift"). For example, the adjustable arm support 105 may include a bracket 109 that is configured to move upward or downward along or relative to the column 102 of the support table 101. The second degree of freedom allows the adjustable arm support 105 to be tilted. For example, the adjustable arm support 105 may include a swivel joint that allows the adjustable arm support 105 to be aligned with the bed in a Trendelenburg position. The third degree of freedom allows the adjustable arm support 105 to "pivot up," which can be used to adjust the distance between one side of the table 101 and the adjustable arm support 105. The fourth degree of freedom allows the adjustable arm support 105 to translate along the longitudinal length of the table.

[0076] Fig.12 and Fig.13 The surgical robot system 100 in the embodiment of the present invention may include a table supported by a column 102 mounted to a base 103. The base 103 and the column 102 support the table 101 relative to the support surface. The floor axis 131 and the support axis 133 are located at Fig.13 Shown in.

[0077] An adjustable arm support 105 can be mounted to column 102. In other embodiments, arm support 105 can be mounted to table 101 or base 103. Adjustable arm support 105 can include bracket 109, rod or track connector 111, and rod or track 107. In some embodiments, one or more robotic arms mounted to track 107 can translate and move relative to each other.

[0078] The bracket 109 may be attached to the column 102 via a first joint 113 that allows the bracket 109 to move relative to the column 102 (e.g., such as up and down along a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that provides a third degree of freedom ("upward pivot") to the adjustable arm support 105. Additional joints 119 may be provided (in Fig.13 ), the additional joint mechanically constrains the third joint 117 to maintain the orientation of the track 107 when the track connector 111 rotates about the third axis 127. The adjustable arm support 105 may include a fourth joint 121, which may provide the adjustable arm support 105 with a fourth degree of freedom (translation) along a fourth axis 129.

[0079] Fig.14An end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101 is illustrated. A first robotic arm 142A is attached to a rod or track 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the track 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the track 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0080] In some embodiments, one or more of the robotic arms 142A, 142B include an arm with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B may include eight degrees of freedom, including an insertion axis (including 1 degree of freedom for insertion), a wrist (including 3 degrees of freedom for wrist pitch, yaw, and roll), an elbow (including 1 degree of freedom for elbow pitch), a shoulder (including 2 degrees of freedom for shoulder pitch and yaw), and a base 144A, 144B (including 1 degree of freedom for translation). In some embodiments, the insertion degrees of freedom may be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0081] C. Device Drivers and Interfaces .

[0082] The end effector of the system's robotic arm includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device that may not contain any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.

[0083] Fig.15An example instrument driver is illustrated. The instrument driver 62, positioned at the distal end of the robotic arm, includes one or more drive units 63 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., Fig.15 In operation, the control circuit 68 will receive the control signal, transmit the motor signal to the motor 66, compare the resulting motor speed measured by the encoder 67 to the desired speed, and modulate the motor signal to generate the desired torque.

[0084] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, the example sterile adapter may include a series of rotation inputs and rotation outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover composed of a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, medical devices can dock with patients in areas that require sterilization (i.e., sterile areas).

[0085] D. Medical devices .

[0086] Fig.16An example medical device with a paired instrument driver is illustrated. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its intended design for manual interaction by a physician, may typically include a rotatable drive input 73 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 extending through a distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 may share an axis of rotation with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include splines that are designed to mate with a receptacle on the drive input 73.

[0087] The elongated shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 71 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from a joined wrist formed by a connecting fork having at least one degree of freedom and a surgical tool or medical instrument (such as, for example, a grasper or scissors), which can be actuated based on the force from the tendon when the drive input rotates in response to the torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft can include a manipulable or controllable curved segment that can be articulated and bent based on the torque received from the drive output 74 of the instrument driver 75.

[0088] The torque from the instrument driver 75 is transmitted to the elongated shaft 71 down the shaft 71 using tendons. These individual tendons (such as pull wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are guided downward along one or more pull cavities of the elongated shaft 71 and anchored at the distal portion of the elongated shaft 71, or in the wrist at the distal portion of the elongated shaft. During surgical procedures such as laparoscopic, endoscopic, or hybrid procedures, these tendons can be connected to a distally mounted end effector, such as a wrist, a grasper, or a pair of scissors. In such an arrangement, the torque applied to the drive input 73 transmits tension to the tendons, causing the end effector to actuate in some manner. In some embodiments, during a surgical procedure, the tendons can cause the joint to rotate around an axis, causing the end effector to move in one direction or another. Alternatively, the tendons can be connected to one or more jaws of a grasper at the distal end of the elongated shaft 71, wherein the tension from the tendons causes the grasper to close.

[0089] In endoscopy, the tendon may be coupled to a bending or articulation segment located along the elongated shaft 71 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the bending segment, torque applied to the drive input 73 will be transmitted downward along the tendon, causing the softer bending segment (sometimes referred to as the segment or region capable of articulation) to bend or articulate. Along the non-bending segment, it may be advantageous to spiral or swirl a separate traction lumen that guides a separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the traction wire. For specific purposes, the angle of the spiral and / or the spacing therebetween may be varied or designed, wherein a tighter spiral presents less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but also presents limited bending. In another case, the traction lumen may be guided parallel to the longitudinal axis of the elongated shaft 71 to allow controlled articulation in the desired bending or articulation segment.

[0090] In endoscopy, the elongated shaft 71 houses a plurality of components to assist in robotic procedures. The shaft may include a working channel for deploying surgical tools (or medical devices), irrigation components, and / or suction components to the operating area at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers to transmit signals to / from an optical assembly at the distal tip, which may include an optical camera. The shaft 71 may also house an optical fiber to carry light from a proximally located light source (such as a light emitting diode) to the distal end of the shaft.

[0091] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, irrigation and suction to the surgical site. The distal tip may also include a port for a camera (such as a fiberscope or a digital camera) to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source, which is used to illuminate the anatomical space when the camera is used.

[0092] exist Fig.16 In the example of , the axis of the drive shaft, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Rolling the elongated shaft 71 along its axis while keeping the drive input 73 stationary can cause undesirable tangling of the tendons as they extend out of the drive input 73 and into the traction cavity within the elongated shaft 71. The resulting tangling of such tendons can disrupt any control algorithm designed to predict the movement of the flexible elongated shaft during an endoscopic procedure.

[0093] Fig.17 An alternative design of an instrument driver and an instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units, wherein the drive outputs 81 of these drive units are aligned in parallel at the end of a robot arm 82. The drive unit and its corresponding drive output 81 are contained in a rotating assembly 83 of the instrument driver 80, which is driven by a drive unit in the drive unit in the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along the circular bearing that connects the rotating assembly 83 to the non-rotating part 84 of the instrument driver. Electric power and control signals can be conveyed to the rotating assembly 83 from the non-rotating part 84 of the instrument driver 80 by electrical contact, and the electrical contact can be maintained by the rotation of the brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can be responsive to a separate drive unit integrated into the non-rotating part 84, and is therefore not parallel to other drive units. The rotating mechanism 83 allows the instrument driver 80 to rotate the drive unit and its corresponding drive output 81 around the instrument driver axis 85 as a single unit.

[0094] Similar to previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown as having a transparent outer surface for discussion purposes) that includes a plurality of drive inputs 89 (such as sockets, pulleys, and spools) configured to receive drive outputs 81 in the instrument driver 80. Unlike previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 89, rather than being substantially parallel to the axis of the drive inputs 89. Fig.16 as orthogonal in the design.

[0095] When coupled to the rotation assembly 83 of the instrument driver 80, the medical device 86, including the instrument base 87 and the instrument shaft 88, rotates in combination with the rotation assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is located at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Therefore, the rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. In addition, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not tangle during rotation. Therefore, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows the shaft to rotate without tangling any control tendons.

[0096] Fig.18 An instrument with an instrument-based insertion architecture according to some embodiments is shown. The instrument 150 can be coupled to any of the instrument drivers discussed above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 coupled to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The groove 158 is configured to receive one or more wires or cables 180 passing through the groove. Therefore, one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cable 180 may also extend through the elongated shaft 152. The manipulation of the one or more cables 180 (e.g., via the instrument driver) causes the actuation of the end effector 162.

[0097] The instrument handle 170 (also referred to as the instrument base) may generally include an attachment interface 172 having one or more mechanical input portions 174, such as a socket, pulley, or spool, which are designed to reciprocately engage with one or more torque connectors on the attachment surface of the instrument driver.

[0098] In some embodiments, the instrument 150 includes a series of pulleys or cables that enable the elongated shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that accommodates instrument insertion, thereby minimizing reliance on a robotic arm to provide insertion of the instrument 150. In other embodiments, the robotic arm may be largely responsible for instrument insertion.

[0099] E. Controller .

[0100] Any of the robotic systems described herein may include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller may be coupled to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) such that manipulation of the controller causes corresponding manipulation of the instrument, e.g., via master-slave control.

[0101] Fig.19 1 is a perspective view of an embodiment of the controller 182. In this embodiment, the controller 182 comprises a hybrid controller that can have both impedance and admittance control. In other embodiments, the controller 182 can utilize only impedance or passive control. In other embodiments, the controller 182 can utilize only admittance control. By being a hybrid controller, the controller 182 can advantageously have lower perceived inertia when in use.

[0102] In the illustrated embodiment, the controller 182 is configured to allow manipulation of two medical devices and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0103] like Fig.19 As shown, each positioning platform 188 includes a SCARA arm (selective compliant assembly robot arm) 198 coupled to the column 194 via a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along the track 197) to allow each of the handles 184 to translate in the z direction, thereby providing a first degree of freedom. The SCARA arm 198 is configured to allow the handles 184 to move in the xy plane, thereby providing two additional degrees of freedom.

[0104] In some embodiments, one or more load sensors are positioned in the controller. For example, in some embodiments, a load sensor (not shown) is positioned in the body of each gimbal in the gimbal 186. By providing a load sensor, a portion of the controller 182 can be operated under admittance control, thereby advantageously reducing the perceived inertia of the controller when in use. In some embodiments, the positioning platform 188 is constructed for admittance control, and the gimbal 186 is constructed for impedance control. In other embodiments, the gimbal 186 is constructed for admittance control, and the positioning platform 188 is constructed for impedance control. Therefore, for some embodiments, the translational degree of freedom or the azimuth degree of freedom of the positioning platform 188 can rely on admittance control, while the rotational degree of freedom of the gimbal 186 relies on impedance control.

[0105] F. Navigation and Control .

[0106] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operator physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the positioning of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve the information obtained only by radiation-based imaging modalities.

[0107] Fig. 20 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an example embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory in one or more of the components discussed above. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figures 1 to 4 The cart shown, Figures 5 to 14 The bed shown, etc.

[0108] like Fig. 20 As shown, the positioning system 90 may include a positioning module 95 that processes the input data 91-94 to generate position data 96 for the distal tip of the medical device. The position data 96 may be data or logic representing the position and / or orientation of the distal end of the instrument relative to a reference system. The reference system may be a reference system relative to the patient's anatomy or a known object such as an EM field generator (see below for a discussion of EM field generators).

[0109] The various input data 91-94 are now described in more detail. Preoperative mapping can be accomplished using a collection of low-dose CT scans. The preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of cross-sectional views of the patient's internal anatomy. When analyzed as a whole, image-based models of anatomical cavities, spaces, and structures for the patient's anatomical structures (such as the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated from CT images to form a three-dimensional volume of the patient's anatomical structure, which is referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suitable for bronchoscopy.

[0110] In some embodiments, the instrument can be equipped with a camera to provide visual data 92. The positioning module 95 can process the visual data to implement one or more vision-based position tracking. For example, preoperative model data can be used in conjunction with the visual data 92 to implement computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advanced through a working channel of an endoscope). For example, using the preoperative model data 91, the robotic system can generate a library of expected endoscopic images from the model based on the expected path of travel of the endoscope, each image connected to a location within the model. During surgery, the robotic system can reference the library to compare real-time images captured at a video camera (e.g., a camera at the distal end of an endoscope) with those in the image library to assist in positioning.

[0111] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera, and therefore the motion of the endoscope. Some features of the localization module 95 may identify circular geometric structures in the preoperative model data 91 that correspond to anatomical cavities and track changes in those geometric structures to determine which anatomical cavity is selected, as well as track relative rotational and / or translational motion of the camera. The use of topological maps may further enhance vision based algorithms or techniques.

[0112] Optical flow (another computer vision-based technique) can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, brightness, motion compensated encoding, stereo disparity measurements, etc. Through multiple iterations of multi-frame comparisons, the movement and position of the camera (and therefore the endoscope) can be determined.

[0113] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope in a global coordinate system that can be aligned to the patient's anatomical structure represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils in one or more positions and orientations embedded in a medical device (e.g., an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "aligned" to the patient's anatomical structure (e.g., a preoperative model) during surgery to determine a geometric transformation that aligns a single position in the coordinate system with a location in the preoperative model of the patient's anatomical structure. Once registered, an embedded EM tracker in one or more locations of a medical device (eg, the distal tip of an endoscope) can provide a real-time indication of the progress of the medical device through the patient's anatomy.

[0114] The robotic commands and kinematic data 94 may also be used by a positioning module 95 to provide orientation data 96 for the robotic system. The pitch and yaw of the device derived from the joint motion commands may be determined during preoperative calibration. In surgery, these calibration measurements may be used in conjunction with known insertion depth information to estimate the positioning of the instrument. Alternatively, these calculations may be analyzed in conjunction with EM, vision, and / or topological modeling to estimate the positioning of the medical device within the network.

[0115] like Fig.17 As shown, the positioning module 95 can use a variety of other input data. For example, although Fig.17 Not shown, but an instrument utilizing shape sensing fibers may provide shape data that positioning module 95 may use to determine the position and shape of the instrument.

[0116] The localization module 95 may use the input data 91-94 in combination. In some cases, such a combination may use a probabilistic approach, where the localization module 95 assigns a confidence weight to a position determined from each of the input data 91-94. Thus, in situations where the EM data may be unreliable (such as where EM interference may be present), the confidence in the position determined by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematics data 94.

[0117] As discussed above, the robotic systems discussed herein can be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart can store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.), which when executed causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the positioning of instruments in a global coordinate system, anatomical maps, etc.

[0118] 2. User interface for navigating medical procedures .

[0119] Fig.21 An example of a graphical user interface (GUI) 2100 is illustrated that provides either or both a restricted field of view 2102 and an expanded field of view 2104 of an intraoperative image captured by an endoscope. Fig.21 , GUI 2100 presents an expanded field of view 2102 on the viewer / display and Fig.21 , the GUI 2100 presents a limited field of view 2104 on the viewer / display. The two views 2102, 2104 may present intraoperative images captured at the anatomical site.

[0120] In some embodiments, the GUI 2100 may present images in both the extended field of view 2102 and the restricted field of view 2104 side by side as shown. In other embodiments, the GUI may include only one selected view of the two views 2102, 2104. For example, the GUI 2100 may present only the restricted field of view 2104 provided as a circular cropped shape.

[0121] The expanded field of view 2102 may present a superset of the view presented in the limited field of view 2104 to provide better identification of the position and orientation of the endoscope in relation to surrounding features that are in the expanded field of view 2102 but not presented in the limited field of view 2104. In some embodiments, the expanded field of view 2102 may be an uncropped endoscope feed captured by the imaging device of the endoscope with its full field of view. The shape of the image is not limited to the square shape of the expanded field of view 2102 as shown, but is any shape provided by the imaging device.

[0122] The extended field of view 2102 of the real-time endoscope view can provide the operator with more camera data on the screen as the operator navigates during surgery. The traditional real-time endoscope view is a circular cutout of the entire endoscope camera; however, the camera typically collects more data than is currently displayed to the operator. Providing the operator with more usable images captured by the imaging device can be beneficial in situations where the airway or other anatomical structures are visible but were previously cropped out of view. In some embodiments, the operator can toggle the cutout closed in order to obtain a full image from the imaging device. This allows for improved navigation and the ability to better correct the trajectory of the endoscope.

[0123] In some embodiments, the extended field of view 2102 can supplement its captured endoscope feed with predicted or otherwise simulated portions to extend beyond its full field of view and present uncaptured features in the extended field of view 2102. For example, based on the current positioning and orientation of the endoscope relative to the target anatomical site, various image generation algorithms can process preoperative model data (e.g., CT scan data) to predict or simulate features that are surrounding but not visible in the endoscope feed. As another example, artificial intelligence or trained machine learning algorithms can be used to predict or simulate such features. The predicted or simulated image portions can be presented with brighter brightness, lower contrast, or less emphasis in order to (i) distinguish the actual captured portion from the generated image portion, and (ii) not distract the operator.

[0124] The limited field of view 2104 may present a subset of the view presented in the extended field of view 2102 to (i) better focus on features near the center of the endoscope feed or (ii) provide a more intuitive navigation guide to the operator of the endoscope. For example, a circular view as shown in the limited field of view 2104 may generally be more easily associated with the circular shape of the endoscope, so that the operator may more intuitively navigate the endoscope based on the view of the limited field of view 2104. It is noted that the extended field of view 2102 and the limited field of view 2104 are not limited by their respective shapes, but rather by a superset or subset of the respective fields of view they present. That is, although the extended field of view 2102 is illustrated as a square field of view and the limited field of view 2104 is illustrated as a circular field of view, it should be understood that any other shape may be used. For example, any other shape (such as any polygonal, curved, or any other shape) may be applied as a mask or filter to the extended field of view 2102 and the restricted field of view 2104 as long as the restricted field of view 2104 is a subset of the extended field of view 2102.

[0125] In some specific implementations where the two fields of view 2102, 2104 are not presented together but only one field of view is presented, the operator can manually switch between the extended field of view 2102 and the limited field of view 2104. The switch can be performed by manually selecting a switch button 2106. Although the switch button 2106 is shown as a slideable switch, the switch button 2106 can be any selectable graphical control element that can capture the selection or non-selection of the field of view, such as a radio button, a check box, etc. In some embodiments, the switch button 2106 may not be a graphical control element, but a physical control element of the medical system. Some examples of control elements include physical buttons, touch-sensitive buttons, touch screen icons, joysticks, foot pedals, and other elements of an input device that can be used to provide input to the system.

[0126] In some embodiments, switching between the extended field of view 2102 and the restricted field of view 2104, and vice versa, may be automatically accomplished based on a detected condition. For example, the detected condition may include detection of an obstruction or blockage in the navigation path of the endoscope, detection of mucus in the navigation path of the endoscope, or detection of an adjacent airway that blocks navigation or may be of interest to the operator and / or is not currently displayed in the restricted field of view 2104. Additionally, switching between the extended field of view 2102 and the restricted field of view 2104 may depend on the location of the target or lesion, the positioning and orientation of the surgical needle relative to the location of the target or lesion, e.g., the extended field of view 2102 may be automatically turned on as the needle gets closer to the lesion. Additionally, airway configuration, airway size, and airway positioning may affect the automatic turning on and off of the restricted field of view 2102 and the extended field of view 2104.

[0127] It should be understood that machine learning and artificial intelligence can be used to train algorithms to predict or detect other situations in which the extended field of view 2102 may be useful to the physician during surgery and to switch the extended field of view 2102 in response. For example, it can be predicted or determined that certain locations, orientations, or junctions within the luminal network may pose some challenges to correct navigation (e.g., a set of particularly confusing branches), and when the endoscope is determined to be at that location, orientation, or junction, the extended field of view 2102 can be automatically turned on. In some embodiments, a recommendation (e.g., prompt text or icon) to turn on the extended field of view 2102 can be presented instead of automatically turning on the extended field of view 210. Similarly, in some specific implementations, the extended field of view 2102 can be automatically turned off when the endoscope has moved away from the location, orientation, or junction.

[0128] In some embodiments, one or more visual indicators or effects may be applied to the extended field of view 2102 to distinguish its extended image portion from the restricted image portion of the restricted field of view 2104. For example, when the extended field of view 2102 is turned on, an outline of the shape of the restricted field of view 2104 may be overlaid on the extended field of view 2102. For example, referring to the GUI 2100, a circular outline of the restricted field of view 2104 may be overlaid on the extended field of view. For another example, the extended image portion may be shaded, grayed out, blurred, rendered in a lighter color, etc.

[0129] Fig. 22 An example of a GUI 2200 including various views is illustrated. The live endoscopy view 2202 presented on the left half of the GUI 2200 may be defaulted to Fig.21 2104. At the upper right is a pathway view 2204 and at the lower right is a CT view 2206. As shown, pathway view 2204 may present a view of a 3D model of a luminal network (e.g., a lung) and CT view 2206 may present a view of a CT image. Pathway view 2204 may use a model generated based on multiple CT images as its 3D model. For example, the 3D model may be based on Fig. 20 The preoperative model data is generated from the model data 91. As shown, the path view 2204 and the CT view 2206 can present tip indicators 2208a-2208b, path indicators 2210a-2210b and target indicator 2212 to assist in the navigation of the endoscope.

[0130] Fig.23 An example of a GUI 2300 including multiple orientation indicators is illustrated. The GUI 2300 may include a selectable user interface (UI) element (e.g., a toggle button) 2302 that turns the presentation of the orientation indicators on or off. When the UI element 2302 is set to present the orientation indicators, each of the presented orientation indicators may be displayed to inform the operator of the current orientation of the endoscope. In this example and other examples described herein, the orientation of the endoscope may be determined based on one or more tracking systems associated with the endoscope, which may detect the orientation of the endoscope as it passes through the patient's anatomy. Examples of tracking systems include electromagnetic (EM) tracking systems, fiber optic shape sensors, image sensors, and other systems that include sensors for determining the orientation of the endoscope (or the tip of the endoscope) relative to the patient's anatomy. The tracking system may also use a combination of sensor inputs to determine the orientation of the sensor within the patient's anatomy. Reference above Fig. 20 One example of a tracking system is described and may be used to determine the orientation of a scope to facilitate the orientation indicator or compass described herein.

[0131] The first orientation indicator 2310 may be displayed in the endoscopic view (e.g., Fig. 22 The endoscopic view 2202 may be presented outside or near the endoscope. The endoscopic view may be a live image feed of the patient's anatomy captured by the scope as it passes through the patient's anatomy. The first orientation indicator 2310 may provide a reference axis / plane 2312 with respect to one or more anatomical directions, such as an anterior / posterior axis (represented by "A" and "P") or a medial / lateral axis (represented by "M" and "L"). The first orientation indicator 2310 may additionally provide a direction indicator 2314, which may be 2D (such as an arrow or a direction needle) or 3D (such as a cone as shown) to communicate the orientation of the endoscope associated with the endoscopic view. For example, in the GUI 2300, the first orientation indicator 2310 may notify the operator that the top of the endoscopic view is toward the front, the bottom is toward the back, the left is toward the inside, and the right is toward the outside. In some other embodiments, as the endoscope adjusts its orientation, the direction indicator 2314, the reference axis / plane 2312, or both may adjust in the display by a corresponding amount. For example, as the endoscope "rolls" along its longitudinal axis, the direction indicator 2314 may scroll in the display by a corresponding scroll amount while the reference axis / plane 2312 remains fixed.

[0132] The reference axis / plane 2312 and one or more anatomical directions thereof may be determined preoperatively or during surgery. For example, preoperatively, the endoscope is positioned relative to the patient on a support platform (e.g., Figure 5 The known placement of the endoscope at a known location on the support platform 38 of the endoscope may inform the GUI 2300 of a set of default reference axes / planes 2312. For another example, preoperatively or intraoperatively, the operator may manually input or adjust the reference axes / planes 2312. For another example, preoperatively or intraoperatively, the reference axes / planes 2312 may be automatically determined by performing a calibration process with the endoscope.

[0133] An example calibration or registration process may include advancing the endoscope into and then out of at least two known reference branches of the luminal network. The process may facilitate registration of the reference system of the tracking system with the reference system of the patient's anatomy. For example, an operator may advance the bronchoscope into the left bronchus, withdraw it from the left bronchus, and advance the bronchoscope into the right bronchus. Such a calibration process may provide multiple reference points that may be mapped onto at least one axis, providing a zero point on the axis, and an anatomical direction based on the zero point. Use of a calibration process may advantageously enable accurate determination of the reference axis / plane 2312 and its anatomical direction without the errors typically associated with a default or manual reference configuration.

[0134] The second orientation indicator 2320 illustrates a 2D orientation indicator represented as a box. The second orientation indicator 2320 can inform the operator of the path view (e.g., Fig. 22 The path view 2204 of the figure is seen from the front side toward the back side. In some specific implementations, when the path view as shown is rotated clockwise or counterclockwise, the second orientation indicator 2320 can tilt a corresponding amount. Although a frame is shown, it should be noted that any indicative shape can be used.

[0135] The third orientation indicator 2330 illustrates a 3D orientation indicator represented as a cube. The third orientation indicator 2330 can inform the operator that the simulated view 2306 based on the preoperative model prediction is seen from the front side toward the back side and from the upper side toward the lower side by presenting an inverted "A" and an upright "S". In addition, the slight display of the "L" on the outside informs the operator that the simulated view 2306 is seen from an angle oriented slightly outside toward the inside. Therefore, the angle of each face of the cube and which letters are visible in what orientation can reflect the orientation of the endoscope in the simulated view 2306. Although a cube is shown, it should be noted that any indicative object can be used.

[0136] As exemplified by orientation indicators 2310, 2320, 2330 and their respective views, each view may be presented with a corresponding orientation indicator that independently informs the orientation of the image presented in the view. For example, second orientation indicator 2320 reflects a first orientation of the path view, while third orientation indicator 2330 reflects a second orientation of the simulation view 2330 that is different from the first orientation.

[0137] In the present disclosure, orientation indicators, including the described orientation indicators 2310, 2320, 2330, may be referred to as a "compass". Although the structure and elements of the orientation indicator may differ from a traditional physical compass, the term is broadly interpreted herein to include UI elements that can assist an operator in determining the orientation of the endoscope relative to a shown view. In some embodiments, all or a portion of the compass may be presented near or overlaid within the peripheral boundary of the endoscopic view in order to provide an intuitive association between the compass and the endoscopic view, such as relative to the compass. Figure 24 to Figure 25 as shown and described in more detail.

[0138] In some embodiments, an articulation indicator 2340 may be presented near or within the endoscopic view. The articulation indicator 2340 may indicate in which anatomical direction the endoscope is currently articulating or towards which anatomical direction the endoscope is currently articulating. In the event that the endoscope has multiple components that are capable of individual articulation (such as a sheath portion and a guide portion), a separate articulation indicator 2342, 2346 may be provided for each component that is capable of individual articulation. As shown, the separate articulation indicators 2342, 2346 may be presented as variable length arcs having a center positioned toward the direction of articulation and a length corresponding to the magnitude of the articulation. The maximum available articulation may be indicated by a first set of endpoints 2344a-2344b of the sheath articulation indicator 2342 and a second set of endpoints 2348a-2348b of the guide articulation indicator 2346. Although a variable length arc is shown as an example, other graphical elements (such as variable thickness arrows) may represent an articulation vector (i.e., the direction and magnitude of the articulation). Relative to Fig.26 Another embodiment of the articulation indicator 2340 is described in greater detail.

[0139] Fig.24 An example of a GUI 2400 including an example of a 2D compass 2402 is illustrated. The 2D compass 2402 may be positioned around an endoscopic view (e.g., Fig. 22 The 2D compass 2402 may include letters at different locations along the border of the compass to indicate corresponding anatomical directions, such as superior / inferior, anterior / posterior, medial / lateral, proximal / distal, central / peripheral, surface / bottom, dorsal / ventral, etc. In the example 2D compass 2402, the anterior / posterior and medial / lateral anatomical directions are provided with letters A / P and M / L, respectively, referred to herein as anatomical direction indicators 2406a-2406d. Each pair of anatomical direction indicators 2406a-2406d may be presented at opposite ends of the peripheral border. The 2D compass 2402 may provide an angular indicator / ruler 2404 around the peripheral border of the endoscopic view to indicate regular angular intervals of 5 degrees, 10 degrees, 15 degrees, 20 degrees, 30 degrees, etc. In some embodiments, all or at least a portion of the 2D compass 2402 (such as the anatomical direction indicators 2406a-2406d or the angle indicator / ruler 2404) may be overlaid on the endoscopic view.

[0140] The anatomical direction indicators 2406a-2406d and the angle indicator / ruler 2404 can help the operator understand the current orientation of the endoscope capturing the image for the endoscopic view. The anatomical direction indicators 2406a-2406d of the 2D compass 2402 can inform the operator that "up" will move the endoscope to the front side, "down" will move the endoscope to the back side, "left" will move the endoscope inward into a bronchus, and "right" will move the endoscope outward into another bronchus. If the endoscope is "rolled", in some embodiments, the image can be rotated clockwise or counterclockwise based on the change in angle of "roll" while the anatomical direction indicators 2406a-2406d and the angle indicator / ruler 2404 remain fixed. In other embodiments, the image can remain fixed while the anatomical direction indicators 2406a-2406d and the angle indicator / ruler 2404 rotate based on the change in angle of "roll".

[0141] In one example, the 2D compass 2402 is always displayed on the endoscopic view for the operator during the procedure (e.g., always on). In another example, the compass 2404 can be manually turned on or off by the operator using the switch 2410 (e.g., by selecting "OFF" or "2D"). In another example, the compass 2402 or a portion thereof can be automatically turned on or off based on the orientation of the endoscope. For example, when the endoscope or camera is detected as being exactly or approximately parallel to a particular axis (e.g., parallel to an axis in Cartesian space), one or more other perpendicular or parallel axes can automatically appear or disappear. In some embodiments, the 2D compass 2402 can be automatically turned on when the endoscope or camera is detected as being exactly or approximately parallel to a particular axis.

[0142] Although in Fig.24 Four letters are shown on the 2D compass 2402 of FIG. 2 , but in another example, any other number of letters may be shown. For example, if the operator has navigated the endoscope to a corner location and there is a dead end in a particular navigation direction, three letters may be shown. In another example, as the operator moves in the direction of an additional parallel axis not shown, a fifth letter may appear / disappear in the middle portion of the endoscope view with a softer or weakened shading effect. For example, in Fig.24 , “M” may appear when the operator moves in the medial direction, and “L” may appear when the operator moves in the lateral direction.

[0143] Fig.25 An example of a GUI 2500 including an example of a 3D compass 2502 is illustrated. The 3D compass 2502 may include a 2D compass (e.g., Fig.24 2D compass 2402) and its features. 3D compass 2502 can be centered around the endoscopic view (e.g., Fig. 22 The 3D compass 2502 may include or overlay a peripheral border of the endoscopic view 2202 of the endoscopic view to provide the operator with a better sense of direction when navigating the endoscope during surgery. For example, the 3D compass 2502 may include 2D anatomical direction indicators at different locations along the peripheral border of the compass (e.g., Fig.24 3D compass 2502 may include 3D anatomical direction indicators 2406a-2406d) to indicate corresponding anatomical directions, such as anterior / posterior, medial / lateral. In addition, 3D compass 2502 may include 3D anatomical direction indicators 2506a-2506b at different locations within the endoscopic view to indicate corresponding anatomical directions, such as superior / inferior, represented by the letters "S" and "I", respectively. In the illustrated example, the 3D compass is overlaid on the endoscopic view so that the endoscopic view can be viewed through the compass.

[0144] 3D anatomical direction indicators 2506a-2506b can be positioned on the endoscopic view based on the pitch and yaw of the tip of the endoscope. In some embodiments, bendable lines 2504a-2504b can be presented to indicate the pitch and yaw of the endoscope. For example, one or more bendable horizontal lines, which may also be referred to as "latitudes", may be generated to connect each latitude of a 2D axis (such as the M / L axis as shown) to indicate the yaw of the endoscope. Similarly, one or more bendable vertical lines, which may also be referred to as "meridians", may be generated to connect each longitude of another 2D axis (such as the A / P axis as shown) to indicate the pitch of the endoscope. It should be noted that the bendable lines are "bendable", but they do not have to be bent in all cases. For example, when the pitch or yaw of the endoscope is precisely aligned with the 2D axis, the bendable horizontal line 2504a or the bendable vertical line 2504b may become a straight line.

[0145] For simplicity, the example 3D compass 2502 as shown in the figure is illustrated as only showing a flexible horizontal line 2504a and a flexible vertical line 2504b. Each line in the flexible lines 2504a-2504b can surround the endoscopic view, as if the endoscopic view is a sphere. In some embodiments, each line in the flexible lines 2504a-2504b can indicate the proximal part of the sphere and the distal part of the sphere by distinguishing the various aspects of the flexible lines 2504a-2504b. For example, the flexible lines 2504a-2504b are illustrated as solid lines for the proximal part, and are illustrated as dotted lines for the distal part. In some other specific implementations, other distinguishing characteristics can be used, such as line thickness (for example, compared with the distal part, the proximal part is thicker), line color, line contrast, line softness, line existence (for example, only one of the proximal part or the distal part) etc. In one example, one or more letters displayed in the actual field of view that are farther from the user may be faded or presented with lower brightness or contrast.

[0146] 3D anatomical direction indicators 2506a-2506b can be positioned at the intersection of bendable lines 2504a-2504b. As shown, bendable horizontal line 2504a and bendable vertical line 2504b intersect at two points, i.e., at the proximal intersection on the proximal part of the sphere and at the distal intersection on the distal part of the sphere. The axis connecting the two points can be the upper / lower axis of the 3D compass 2502. Therefore, an anatomical direction indicator (e.g., "S") can be positioned at the proximal intersection, and another anatomical direction indicator (e.g., "I") can be positioned at the distal intersection.

[0147] It should be understood that although Figure 24 to Figure 25 Anatomical indicators (S / I, A / P, M / L) are used in the illustration, but other symbols or indicators may be used to inform the user of anatomical direction, and in some cases, other anatomical indicators along other axes may be used, including proximal / distal, central / peripheral, surface / bottom, dorsal / ventral, etc.

[0148] See again Figure 24 to Figure 25 In addition to the anatomical direction indicators (S / I, A / P, M / L) 2406a-2406d, 2506a-2506b on the borders of the 2D compass 2402 and the 3D compass 2502, respectively, the anatomical direction indicators (S / I, A / P, M / L) 2406a-2406d, 2506a-2506b may also be displayed on the compasses 2402, 2502 Fig.242408, which shows the location of the target site in the endoscopic view. The target indicator 2408 can be an icon, shape, mask, symbol, letter, or some other indicator on the outer perimeter of the compass 2402, 2502, or otherwise displayed within the actual field of view of the endoscopic view. In some embodiments, the target site indicator 2408 can become larger, darker, brighter, change color or contrast, etc. based on the distance of the endoscope to the target site. The target site indicator 2408 can help the operator identify the direction in which they should advance the endoscope in order to reach the target anatomical site (e.g., a nodule or lesion).

[0149] In some embodiments, one or more additional visual indicators may be presented to indicate to the operator whether they are properly centered driving the endoscope. As non-limiting examples, the visual indicators may be dashed or solid lines (not shown), icons, shapes, symbols, letters, or other graphical elements. It may be advantageous to keep the endoscope centered relative to navigation of the airway or anatomical passage in which the tool or endoscope is being manipulated, an approach known as centered driving. Indicators such as a bull's-eye may indicate to the user how to best drive the endoscope or tool in order to achieve centered driving.

[0150] Still see Figure 24 to Figure 25 , the compass is designed to give the user a better sense of direction as they navigate and adjust the endoscope tip positioning during a procedure. For example, as the user navigates in the periphery of the lung, they may lose their sense of direction and face a large cognitive burden to determine the anatomical direction that the endoscope tip is pointing to. This is particularly important for customers who use advanced imaging and need to know how to adjust their endoscope positioning based on the imaging results. The purpose of the compass is to reduce the cognitive burden on the user, allowing them to easily identify the corresponding anatomical direction on the controller (up / down / left / right / insert / retract). This is achieved by providing the user with the corresponding anatomical direction (superior / inferior, anterior / posterior, medial / lateral) in the live endoscopic view.

[0151] In some embodiments, an orientation indicator (e.g., Fig. 22 orientation indicator 2310, 2320, 2330) or a compass (e.g., Figure 24 to Figure 25 The compass 2402, 2502) may always be presented to the user. In other embodiments, the orientation indicator or compass may be displayed using a selectable UI element (e.g., Fig.23 Toggle button 2302 or Fig.24 The selectable UI element can also be used to change the GUI from presenting Figure 24 to Figure 25 The superimposed compass shown is transformed into Fig.23 adjacent orientation indicator 2310 as shown, or vice versa.

[0152] Fig.26 An example joint scene 2600 is illustrated with associated joint motion indicators 2604, 2606. The joint motion indicator 2600 may be displayed in (e.g., Fig.23 2340), is presented near a peripheral boundary of the articulation indicator 2340, around the peripheral boundary, as shown in the example articulation scenes 2600(a)-(e), or overlaid on an endoscopic view (e.g., Fig. 22 The articulation indicator 2600 may inform the operator of various articulation states, including: at least one selected articulable component, current articulation of the articulable component, maximum available articulation of the articulable component, articulability of the articulable component, etc.

[0153] The articulation indicator may include a component selector 2602 that may select one or more articulatable components of the endoscope for articulation. In the first scenario 2600(a), the component selector 2602 has selected "pair" (both the sheath portion and the guide portion) for articulation. In the second scenario 2600(b), the component selector 2602 has selected "sheath" (sheath portion only) for articulation. In the third scenario 2600(c), the fourth scenario 2600(d), and the fifth scenario 2600(e), the component selector 2602 has selected "scope" (guide portion only) for articulation.

[0154] Each of the scenes 2600(a)-(e) may have a sheath articulation indicator 2604 and a scope articulation indicator 2606. The example scenes 2600(a)-(e) as shown present the sheath articulation indicator 2604 as surrounding the scope articulation indicator 2606, which may be more intuitive, but may be the opposite in some implementations. As shown in the scenes 2600(a)-(e), one or more selected articulation indicators may be emphasized, such as having greater contrast, highlights, gradients, patterns, colors, etc., compared to one or more unselected articulation indicators. For example, in the first scene 2600(a), both the sheath articulation indicator 2604 and the scope articulation indicator 2606 are emphasized. For another example, in the second scene 2600(b), only the sheath articulation indicator 2604 is emphasized. For another example, in the third, fourth, and fifth scenes 2600(c)-(e), only the scope articulation indicator 2606 is emphasized.

[0155] The articulation indicators 2604, 2606 may be divided into multiple segments (eg, quadrants as shown) and positioned on a compass (eg, Fig.23 2D Compass 2402 or Fig.25 2502). In some implementations, the joint motion indicators 2604, 2606 may be divided into more or fewer segments, such as five, six, eight, twelve, or any number of segments. Each segment may represent a joint motion toward a direction encompassed by the segment. In addition, each segment may represent a magnitude of the joint motion (e.g., the degree of the joint motion). Thus, the segments may represent to the operator a joint motion vector having a direction and a magnitude.

[0156] Continuing to divide the example scenes 2600(a)-(e) into quadrants, each segment can be represented by a corresponding arc. Note that in the example scenes 2600(a)-(e), each arc is aligned with an anatomical direction (e.g., A / P, M / L), but this alignment is for ease of description. The positioning of the arcs may depend on the endoscope control scheme, and the positioning of the anatomical direction may depend on the use of Fig.23 In other words, the positioning of the arc can be based on the control characteristics of the endoscope, and the positioning of the anatomical direction can be based on the placement of the patient or the luminal network in the patient. In some embodiments, when the endoscope is "rolled", the arc can roll with the corresponding amount. In some other embodiments, the arc can remain fixed during the rolling of the endoscope.

[0157] The length of the arc may indicate the maximum available articulation of the endoscope toward the arc. When the endoscope articulates toward the arc, the amplitude of the articulation may be indicated by filling the arc until the maximum available articulation is reached, as shown. For example, in the first scene 2600(a), both the sheath articulation indicator 2604 and the scope articulation indicator 2606 indicate articulation toward the front and the outside, as shown by the approximately half-filled articulation indicators 2604a, 2604a', 2606a, 2606a'. However, the endoscope does not articulate toward the rear, as shown by the empty articulation indicators 2604a", 2606a". Similarly, the endoscope does not articulate toward the inside. The approximately half-filled articulation indicators 2604a, 2604a', 2606a, 2606a' indicate that the endoscope articulates to approximately half of the maximum available articulation toward the front and the outside. Thus, the articulation indicators 2604, 2606 may inform the operator of the current articulation vectors of the sheath portion and the guide portion.

[0158] In the second scene 2600(b), only the sheath portion is selected for articulation as indicated by the emphasized sheath articulation indicator 2604b, as compared to the de-emphasized scope articulation indicator 2606b. The second scene 2600(b) has the same articulation vector as the first scene 2600(a). If the endoscope is further articulated toward the anterior side based on the selection, the sheath articulation indicator 2604b will be further filled, while the scope articulation indicator 2606b remains unchanged.

[0159] In the third scenario 2600(c), only the guide portion is selected for articulation as indicated by the emphasized guide articulation indicator 2606c, as compared to the de-emphasized sheath articulation indicator 2604c. The third scenario 2600(c) has a first articulation vector for the sheath portion toward the posterior and medial side and a second articulation vector for the guide portion toward the anterior and lateral side. If the endoscope is further articulated toward the anterior side based on this selection, the scope articulation indicator 2606c will be further filled, while the sheath articulation indicator 2604c remains unchanged.

[0160] In the fourth scenario 2600(d), the first fully filled speculum articulation indicator 2606d may indicate maximum articulation, and thus the guide portion may not articulate further toward the anterior side. Similarly, the second fully filled speculum articulation indicator 2606d' may indicate maximum articulation, and thus the guide portion may not articulate further toward the lateral side. In some embodiments, when maximum articulation has been reached, one or more characteristics (e.g., color, thickness, pattern, etc.) of the articulation indicator may be changed to indicate that the articulable component is at its maximum articulation.

[0161] In the fifth scenario 2600 (e), the guide portion has not yet reached its maximum articulation toward the anterior side because the endoscope articulation indicator 2606e is not fully filled. However, here, the guide portion may not continue to articulate toward the anterior side because such articulation may be impeded by a wall or object in the anterior anatomical direction. In this scenario, attempting to articulate the guide portion further toward the anterior side may apply excessive force, causing the endoscope to bend under stress or the pull wire to break under stress. Undesirable stress may be indicated by the endoscope articulation indicator 2606e being shown emphasized (e.g., changing the characteristics of the articulation indicator 2606e) near the outer edge. For example, the articulation indicator 2606e is shown as changing its color near its outer edge.

[0162] although Fig.26The scenes 2600 (a)-(e) illustrate arc-based joint motion indicators 2604, 2606 to represent joint motion vectors, but it should be noted that other specific implementations are possible. For example, it can be considered Fig.23 An articulation indicator 2340 or a vector centered on the endoscopic view.

[0163] In addition, so far, orientation indicators (e.g. Fig.23 The orientation indicator 2310 and Figure 24 to Figure 25 The compass 2402, 2502) has been described as being applicable to live endoscopic views. However, it is also contemplated that the orientation indicator and its various features may be applied to previously captured endoscopic images. For example, the present disclosure may be applied to a previously captured and stored sequence of endoscopic images to add an orientation indicator to the captured endoscopic images.

[0164] 3. Implement systems and terminology .

[0165] Implementations disclosed herein provide systems, methods, and apparatus for user interfaces for navigating anatomical pathways during medical procedures.

[0166] It should be noted that, as used herein, the term "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0167] The functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or processor.

[0168] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0169] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0170] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0171] The foregoing description of the disclosed specific implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

[0172] Additional Implementation

[0173] Depending on the implementation, some actions, events or functions of any one of the processes or algorithms described herein may be performed in a different order, added, merged or completely ignored. Therefore, in some embodiments, not all described actions or events are necessary for the practice of the process.

[0174] Unless otherwise specifically stated or understood in the context of use, conditional language used herein, such as "may", "can", "might", "could", "for example", etc., refers to its ordinary meaning and is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included in any particular embodiment or whether they will be performed in any particular embodiment with or without author input or prompting. The terms "include", "comprise", "have", etc. are synonymous and are used in their ordinary sense and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, etc. Moreover, the term "or" is used in its inclusive sense (rather than in its exclusive sense) so that when used, for example, to connect a series of elements, the term "or" refers to one, some, or all of the elements in the series. Unless specifically stated otherwise, conjunctive language such as the phrase "at least one of X, Y, and Z" is understood in the context of general usage to convey that an item, term, element, etc. can be X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that a particular embodiment requires that at least one of X, at least one of Y, and at least one of Z each be present.

[0175] It should be understood that in the above description of the embodiments, in order to simplify the present disclosure and assist in understanding one or more of the various inventive aspects, various features are sometimes grouped together in a single embodiment, figure, or description thereof. However, this method of the present disclosure should not be understood to reflect the following intention: any claim requires more features than the features explicitly described in that claim. In addition, any component, feature, or step illustrated and / or described in the specific embodiments herein may be applied to any other embodiment or used in conjunction with any other embodiment. In addition, for each embodiment, no component, feature, step, or group of components, features, or steps is necessary or indispensable. Therefore, it is expected that the scope of the present invention disclosed herein and claimed below is not limited to the specific embodiments described above, but should only be determined by a fair reading of the appended claims.

[0176] It should be understood that certain ordinal terms (e.g., "first" or "second") may be provided for ease of reference and do not necessarily imply physical characteristics or ordering. Therefore, as used herein, ordinal terms (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc. do not necessarily indicate the priority or order of the element relative to any other element, but may generally distinguish the element from another element with a similar or identical name (but used to use the ordinal term). In addition, as used herein, the indefinite article ("a") may indicate "one or more" rather than "one." In addition, an operation performed "based on" a condition or event may also be performed based on one or more other conditions or events that are not explicitly stated.

[0177] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the example embodiments belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and not interpreted in an idealized or overly formal sense unless expressly defined as such herein.

[0178] For ease of description, the spatially relative terms "external," "internal," "upper," "lower," "below," "above," "vertical," "horizontal," and similar terms may be used herein to describe the relationship between one element or component and another element or component illustrated in the drawings. It should be understood that the spatially relative terms are intended to cover different orientations of the device in use or operation other than the orientation depicted in the drawings. For example, where the device shown in the drawings is flipped, a device that is "below" or "under" another device may be placed "above" the other device. Thus, the illustrative term "below" may include both lower and upper positioning. The device may also be oriented in another direction, and thus the spatially relative terms may be interpreted differently depending on the orientation.

[0179] Unless otherwise expressly stated, comparative and / or quantitative terms such as "less", "more", "greater", etc., are intended to encompass equivalent concepts. For example, "less" may mean not only "less" in the strictest mathematical sense, but also "less than or equal to".

Claims

1. A medical system, comprising: Looking glass; monitor; one or more processors; and a memory storing instructions for execution by the one or more processors, the stored instructions causing the one or more processors to: displaying on the display an endoscopic view of the anatomy derived from the scope, and A compass is displayed overlaying the endoscopic view.

2. The medical system according to claim 1, wherein: The compass is configured to inform a user of the orientation of the scope relative to the patient's anatomy.

3. The medical system according to claim 1, wherein: The stored instructions further cause the one or more processors to determine the orientation based on a tracking sensor coupled to the scope.

4. The medical system according to claim 1, wherein: The t-compass is configured to be manually turned on or off using an input device coupled to the display.

5. The medical system according to claim 1, wherein: The compass is configured to be automatically turned on or off depending on at least one of a position, orientation, or direction of movement of the scope.

6. The medical system according to claim 1, wherein: Execution of the instructions further causes the one or more processors to: In response to detecting movement of the scope approximately parallel to an axis in Cartesian space, the compass is automatically turned on and displays other axes of the Cartesian space approximately perpendicular to the movement of the scope.

7. The medical system according to claim 1, wherein: The displaying the compass includes displaying three indicators in response to the scope being at a corner of a path.

8. The medical system according to claim 2, wherein: Displaying the compass includes displaying five indicators, wherein at least one of the five indicators is configured to appear and disappear when the scope is moved.

9. The medical system according to claim 2, wherein: Displaying the compass includes displaying six indicators, and at least one of the six indicators appears with less brightness or contrast than other indicators of the six indicators.

10. The medical system according to claim 1, wherein: The instructions further cause the one or more processors to display a target location indicator on or near the compass, the target location indicator showing a direction to navigate in order to reach a target.

11. The medical system according to claim 1, wherein: The instructions further cause the one or more processors to display an indicator on or near the compass that indicates to a user a center position within a path that, when followed, provides navigation of the scope centered relative to the path.

12. A medical system, comprising: a scope configured to capture at least one endoscopic image; a display device configured to display the endoscopic image; one or more processors; and a memory storing instructions for execution by the one or more processors, the stored instructions comprising instructions causing the one or more processors to: determining an orientation of the scope associated with the endoscopic image; determining at least one reference axis associated with the luminal network, the reference axis indicating a first anatomical direction and a second anatomical direction; generating an orientation indicator based on the reference axis; as well as The display is caused to present the endoscopic image and the orientation indicator, with the orientation indicator overlaid on the endoscopic image.

13. The medical system according to claim 12, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to: generating anatomical direction indicators for the first anatomical direction and the second anatomical direction; as well as The anatomical direction indicators are presented at opposite ends of a peripheral boundary of the orientation indicator.

14. The medical system according to claim 13, wherein: The orientation indicator is a two-dimensional orientation indicator having a first axis and a second axis, wherein: The first axis is associated with an anterior / posterior anatomical direction and the second axis is associated with a medial / lateral anatomical direction, and The anterior / posterior anatomical directions and the medial / lateral anatomical directions are indicated using anatomical direction indicators positioned at the peripheral boundaries of the two-dimensional orientation indicator.

15. The medical system of claim 14, wherein: The anatomical direction indicator is a letter or a symbol.

16. The medical system of claim 14, wherein: The orientation indicator is a three-dimensional orientation indicator additionally having a third axis including superior / inferior anatomical directions based on the yaw and pitch of the scope.

17. The medical system of claim 16, wherein: An anatomical direction indicator representing the superior / inferior anatomical direction is overlaid on the endoscopic image.

18. The medical system of claim 17, wherein: The three-dimensional orientation indicator includes a curved horizontal line connecting the anterior / posterior anatomical directions and a curved vertical line connecting the medial / lateral anatomical directions, and wherein the intersection of the curved horizontal line and the curved vertical line represents the anatomical direction in the superior / inferior anatomical direction.

19. The medical system of claim 12, wherein: The orientation of the scope is referenced based on a calibration process that includes navigating the scope to the left and right bronchus.

20. The medical system of claim 12, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to: The orientation indicator is turned on or off from the display device.

21. The medical system of claim 12, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to select between a two-dimensional orientation indicator or a three-dimensional orientation indicator.

22. The medical system of claim 12, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to: generating a target site indicator; and The target site indicator is overlaid on the endoscopic image.

23. The medical system of claim 22, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to: determining a distance from the tip of the scope to the target site; and The target site indicator is sized based on the distance.

24. The medical system of claim 12, wherein: The stored instructions also include instructions that, when executed by the one or more processors, cause the one or more processors to: generating an articulation indicator for an articulatable component of the scope, in: The length of the articulation indicator corresponds to the degree of articulation of the scope, and The positioning of the articulation indicator at the peripheral border of the endoscopic image indicates the direction of articulation of the scope.

25. The medical system of claim 24, the stored instructions further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determining that the extent of joint motion is at maximum joint motion; and At least one property associated with the articulation indicator is changed based on the determination.

26. The medical system of claim 24, the stored instructions further comprising instructions that, when executed by the one or more processors, cause the one or more processors to: determining that the articulatable component is prevented from further articulation; and At least one property associated with an outer edge of the articulation indicator is changed based on the determination.

27. A robot system, comprising: a robotic arm coupled to the endoscope; a viewer for displaying a field of view of an anatomical part derived from the scope; one or more processors; and a memory storing instructions for execution by the one or more processors, the stored instructions comprising instructions for providing an electrical signal for: displaying an endoscopic view including the field of view on the viewer, and A two-dimensional compass or a three-dimensional compass is displayed, the two-dimensional compass or the three-dimensional compass configured to be switched between overlaying the field of view and being provided as an icon proximate the field of view.

28. The robotic system of claim 28, wherein: In response to switching from overlaying the field of view to being provided as an icon, the two-dimensional compass or the three-dimensional compass is configured to be resized and repositioned.

29. A robot system, comprising: a robotic arm coupled to the endoscope; a viewer for displaying a field of view of an anatomical part derived from the scope; one or more processors; and a memory storing instructions for execution by the one or more processors, the stored instructions including instructions for providing electrical signals for displaying an endoscopic view including the field of view on the viewer, Therein, the field of view displayed in the endoscopic view can be switched between an expanded view of the field of view and a restricted view of the field of view.

30. The robotic system of claim 30, wherein: The expanded view is an uncropped view of the endoscope feed and the restricted view is a cropped view of the endoscope feed.

31. The robot system according to claim 30, wherein: The stored instructions also include instructions for providing an electrical signal for displaying a selectable icon adjacent to the displayed endoscopic view, the selectable icon switching the displayed endoscopic view between the expanded field of view and the restricted field of view in response to being selected by a user.

32. The robotic system of claim 30, wherein: The memory instructions also include instructions for automatically switching the displayed endoscopic view between the limited field of view and the expanded field of view in response to a detected condition.

33. The robotic system of claim 33, wherein: The detected condition includes at least one of the following: a blockage or obstruction is detected in a navigation path of the scope, detecting mucus in the navigation path of the scope, An adjacent path that blocks navigation is detected, It is detected that the path that the user is interested in is not displayed.

34. The robotic system of claim 30, wherein: The memory instructions also include instructions for simulating an image portion and including the simulated image portion in the extended field of view to change a shape of the endoscopic view.

Images