Dynamic Flexible Scope Driving Device and Method of Using the Same

Abstract

A surgical robot system can include a robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The surgical robot system can be configured to operate the flexible scope in a particular mode among a plurality of modes. In accordance with a determination that the flexible scope is operating in the particular mode, the surgical robot system can provide an electrical signal for presenting a first visual indicator corresponding to each mode on the viewer.

Description

Technical Field

[0001] The systems and methods disclosed herein are directed to devices and methods for indicating the position or orientation of a surgical tool, and more particularly, to a surgical robotic system for indicating the position or orientation of a flexible surgical tool.

Background Art

[0002] Robotic medical systems can perform a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy (e.g., bronchoscopy, ureteroscopy, gastroscopy, etc.).

[0003] Such robotic medical systems can include robotic arms configured to control the movement of surgical tool(s) during a given medical procedure. To achieve a desired posture of the surgical tool, the robotic arm may be placed in a specific posture during remote operation. Some robotic medical systems can include arm supports (e.g., bars) connected to and supporting each base of the robotic arms.

Summary of the Invention

Means for Solving the Problems

[0004] One or more instruments can be coupled to one or more robotic arms of a robotic medical system (e.g., a surgical robotic system) for a medical procedure. For example, the instrument can be coupled to the robotic arm as a starting instrument for performing the procedure or as an exchange instrument during the procedure.

[0005] In a robotic system including a plurality of robotic arms, at least one robotic arm can be coupled to a camera or scope that provides a surgical field of view. The scope is an important asset for laparoscopic or endoscopic surgery. There are a limited number of ways to control a straight and rigid robotic scope. For example, the scope can be inserted, removed, panned left and right, and rotated. However, a rigid scope can present challenges in a robotic system including a plurality of robotic arms. For example, the limited number of ways in which a rigid scope can be controlled can lead to an increase in collisions between robotic arms. Further, a rigid scope can be large in size, which can congest the surgeon's workspace or further limit the area where surgical instruments and ports can be located on the patient.

[0006] To mitigate these challenges, a robotic arm can be coupled to a flexible scope (e.g., a flexible laparoscope, a flexible endoscope, etc.) including a camera. The flexible scope introduces additional degrees of freedom and enables a surgeon to control the camera coupled to the flexible scope in more ways than are available with a rigid scope. For example, in addition to the insertion, removal, panning, and rotation operations achievable with a rigid scope, the user can also move and rotate (e.g., articulate, pivot, bend, flex, etc.) at least a portion (e.g., the distal portion) of the flexible scope. As a result, the user can visualize a target anatomical structure from viewpoints (e.g., angles) that are not achievable with a rigid scope. In some cases, the flexible scope can have a volume that is smaller than the volume of a rigid scope, thereby freeing up space within the surgeon's workspace. However, operating a flexible scope can present new challenges. For example, the user may not be able to easily determine the shape of the flexible scope, or the direction and / or degree of movement or rotation of the scope, or the way to return the scope from a bent configuration to a straight configuration.

[0007] Therefore, there is a need for a robotic medical system that can easily enable a user to visualize the position or orientation of a flexible scope in relation to the surgical field.

[0008] As disclosed herein, a robotic medical system (e.g., a surgical robotic system) can include a robotic arm coupled to a flexible scope. The robotic medical system can also include a viewer for displaying a view of the surgical site derived from the flexible scope. For example, the view of the surgical site can correspond to (or be determined based on) the view of a camera coupled to the flexible scope. The user interface displayed on the viewer may be configured to include an indicator indicating the position and / or orientation of the flexible scope relative to a reference position.

[0009] As disclosed herein, the flexible scope can be inserted into a patient through a port, and the reference position is determined based on the position of the port (e.g., corresponding to the position of the port).

[0010] As disclosed herein, the robotic medical system is configured to operate the flexible scope in a plurality of modes such as a joint movement mode, a trajectory mode, and an automatic insertion and / or withdrawal mode (also referred to as an automatic insertion withdrawal mode).

[0011] As disclosed herein, the robotic medical system has knowledge about the camera view (e.g., by storing information indicating the camera view). In some embodiments, the view is determined based on an image provided by the camera. In some embodiments, the view is determined based on the position and orientation of the camera (e.g., a predefined volume of space in front of the camera).

[0012] As disclosed herein, the robotic medical system is configured to enable movement of the robotic arm. In some embodiments, the robotic medical system can cause robotic movement of the robotic arm. In some embodiments, the robotic medical system can enable manual movement of the robotic arm.

[0013] As disclosed herein, the robotic medical system is configured to determine the position and / or orientation of the flexible scope. In some embodiments, the robotic medical system determines that the position or orientation of the scope has changed and updates an indicator to indicate the changed position or orientation of the scope. In some embodiments, the robotic medical system updates the indicator in real time (e.g., while the scope is moving) to indicate the changed position or orientation of the scope.

[0014] Accordingly, the systems and / or methods disclosed herein advantageously improve the operation of a robotic medical system during surgery. For example, the use of a flexible scope allows the user to visualize target anatomical structures from perspectives that may not be achievable using a rigid scope. The user interface displayed on the viewer may be configured to include an indicator of the position or orientation of the flexible scope, which leads to an improved user experience as the user can quickly determine the position and / or orientation of the flexible scope. The ability to determine the position or orientation of the scope can also improve the safety of the surgery.

[0015] The systems, methods, and devices of the present disclosure each have several innovative aspects, and none of them are involved only in the desirable attributes disclosed herein.

[0016] According to some embodiments of the present disclosure, a robotic system includes a first robotic arm coupled to a flexible scope. The robotic system also includes a viewer for displaying a view of a surgical site derived from the flexible scope. The robotic system further includes one or more processors and a memory storing instructions executed by the one or more processors. The stored instructions include instructions for operating the flexible scope in a particular one of a plurality of modes. The stored instructions include instructions for providing an electrical signal for presenting, on the viewer, a first visual indicator corresponding to each mode in accordance with a determination that the flexible scope is operating in a particular one of the plurality of modes. The first visual indicator indicates at least one of a position or an orientation of the flexible scope relative to a reference position.

[0017] In some embodiments, the flexible scope is inserted into a patient through a first port, and the reference position is determined based on the position of the first port.

[0018] In some embodiments, the robotic system causes a distal portion of the flexible scope to move from a first position or a first orientation to a second position different from the first position or a second orientation different from the first orientation while the robotic system operates the flexible scope in a first one of the plurality of modes.

[0019] In some embodiments, the stored instructions include instructions for updating the first visual indicator to indicate a second position or a second orientation of a distal portion of the flexible scope relative to the reference position.

[0020] In some embodiments, the first visual indicator includes one or more reference axes and graphical elements positioned relative to the one or more reference axes to identify at least one of a direction or an extent of movement.

[0021] In some embodiments, the stored instructions also include instructions for providing an electrical signal for presenting an updated graphical element indicating a change in the position and / or orientation of the flexible scope associated with the movement, according to a determination that the distal portion of the flexible scope has moved to at least one of (i) from a first position to a second position, or (ii) from a first orientation to a second orientation.

[0022] In some embodiments, the stored instructions also include instructions for determining the shape of the flexible scope and providing an image corresponding to the determined shape.

[0023] In some embodiments, the stored instructions also include instructions for providing an electrical signal for presenting an updated image indicating the changed shape, according to a determination that the shape of the flexible scope has changed from a first shape to a second shape.

[0024] In some embodiments, the image includes two or more links corresponding to two or more links of the flexible scope.

[0025] In some embodiments, the first visual indicator includes a plurality of graphical elements. The plurality of graphical elements includes a first graphical element and a second graphical element different from the first graphical element. The first graphical element represents a first component of the movement of the distal end of the flexible scope along a first direction. The second graphical element represents a second component of the movement of the distal end of the flexible scope along a second direction, and the first direction is different from the second direction.

[0026] In some embodiments, the stored instructions include instructions for providing an electrical signal to update the position of a first graphical element in accordance with a determination that movement of the distal end of the flexible scope along a first direction has changed from a first magnitude to a second magnitude. The stored instructions also include instructions for providing an electrical signal to update the position of a second graphical element in accordance with a determination that movement of the distal end of the flexible scope along a second direction has changed from a third magnitude to a fourth magnitude.

[0027] In some embodiments, the stored instructions include instructions for providing an electrical signal to update the length of a first graphical element in accordance with a determination that movement of the flexible scope in a first direction has changed from a first magnitude to a second magnitude. The stored instructions also include instructions for providing an electrical signal to update the length of a second graphical element in accordance with a determination that movement of the flexible scope in a second direction has changed from a third magnitude to a fourth magnitude.

[0028] In some embodiments, the view of the surgical site is derived from a camera coupled to the flexible scope. The stored instructions also include instructions for expanding a first visual indicator in accordance with a determination that the camera is active.

[0029] In some embodiments, the robotic system causes movement of the flexible scope about a target point in space while the robotic system is operating the flexible scope in a second one of a plurality of modes.

[0030] In some embodiments, the first visual indicator includes a first graphical element representing the target point.

[0031] In some embodiments, the stored instructions also include instructions for providing an electrical signal to overlay the first graphical element on the displayed view.

[0032] In some embodiments, the first visual indicator includes a second graphical element that represents a line of sight from the target point to the port of the flexible scope.

[0033] In some embodiments, the first graphical element has a first size. The stored instructions also include instructions for adjusting the size of the first graphical element from the first size to a second size in accordance with a change in the insertion depth of the flexible scope into the surgical site.

[0034] In some embodiments, the first visual indicator includes a second graphical element that represents the range of the orbital movement of the flexible scope.

[0035] In some embodiments, the first visual indicator includes a third graphical element that represents the position of the flexible scope.

[0036] In some embodiments, the stored instructions also include instructions for providing an electrical signal for overlaying the first graphical element, the second graphical element, and the third graphical element on the displayed field of view.

[0037] In some embodiments, the first visual indicator includes one or more axes and a graphical element that represents the insertion direction of the flexible scope with respect to the one or more axes.

[0038] In some embodiments, the robotic system automatically inserts and / or retracts the flexible scope when the robotic system is operating the flexible scope in a third mode of a plurality of modes.

[0039] In some embodiments, the stored instructions also include instructions for determining the shape of the flexible scope during activation of the third mode and for providing an image corresponding to the determined shape.

[0040] In some embodiments, the stored instructions also include instructions for updating an image while the flexible scope is automatically inserted or retracted.

[0041] In some embodiments, the first visual indicator includes a graphical element indicating at least one of the degree of insertion of the flexible scope, the degree of retraction, or the articulation movement.

[0042] In some embodiments, the robotic system includes a second robotic arm coupled to the surgical tool.

[0043] In some embodiments, the stored instructions include instructions for providing an electrical signal for presenting a second visual indicator corresponding to the surgical tool on the viewer.

[0044] In some embodiments, the first visual indicator includes information identifying a first robotic arm coupled to the flexible scope.

[0045] In some embodiments, the first visual indicator is displayed on the viewer simultaneously with the display of the view of the surgical site.

[0046] In some embodiments, the first visual indicator is displayed around the display of the view of the surgical site.

[0047] In some embodiments, the first visual indicator is displayed overlaid on the display of the view of the surgical site.

[0048] In some embodiments, the viewer is part of the surgeon console.

[0049] In some embodiments, the surgeon console includes an input device. The stored instructions include instructions for receiving an input on the input device and determining that the flexible scope is operating in a particular mode in accordance with the received input.

[0050] In some embodiments, the input device includes a foot pedal.

[0051] According to some embodiments of the present disclosure, a robotic system includes a first robotic arm coupled to a flexible scope. The robotic system includes a viewer for displaying a view of a surgical site derived from the flexible scope. The robotic system also includes one or more processors and a memory storing instructions executed by the one or more processors. The stored instructions include instructions for operating the flexible scope and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of a position or orientation of the flexible scope relative to a reference position.

[0052] According to some embodiments, an electronic device communicates with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The electronic device includes one or more processors and a memory. The memory stores instructions executed by the one or more processors. The stored instructions include instructions for operating the flexible scope in a particular mode of a plurality of modes. The stored instructions also include instructions for providing an electrical signal for presenting a first visual indicator corresponding to each mode on the viewer according to a determination that the flexible scope is operating in a particular mode of the plurality of modes. The first visual indicator indicates at least one of a position or orientation of the flexible scope relative to a reference position.

[0053] According to some embodiments, an electronic device communicates with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The electronic device includes one or more processors and a memory. The memory stores instructions executable by the one or more processors. The stored instructions include instructions for operating the flexible scope and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of a position or an orientation of the flexible scope relative to a reference position.

[0054] According to some embodiments, a computer-readable storage medium stores instructions executable by one or more processors that communicate with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The stored instructions include instructions for operating the flexible scope in a particular mode of a plurality of modes. The stored instructions include instructions for providing an electrical signal for presenting a first visual indicator corresponding to each mode on the viewer according to a determination that the flexible scope is operating in the particular mode of the plurality of modes. The first visual indicator indicates at least one of a position or an orientation of the flexible scope relative to a reference position.

[0055] According to some embodiments, a computer-readable storage medium stores instructions that are executed by one or more processors that communicate with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The stored instructions include instructions for operating the flexible scope in a particular one of a plurality of modes. The stored instructions include instructions for operating the flexible scope and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of a position or an orientation of the flexible scope relative to a reference position.

[0056] Note that the various embodiments described above can be combined with any other embodiments described herein. The features and advantages described herein are not all-inclusive, and in particular, many additional features and advantages will be apparent to those of ordinary skill in the art upon consideration of the drawings, specification, and claims. Further, note that the language used herein has been selected primarily for readability and for instructional purposes, and may not have been selected to delineate or circumscribe the subject matter of the invention.

Brief Description of the Drawings

[0057] The disclosed aspects are described below in conjunction with the accompanying drawings, which illustrate, but do not limit, the disclosed aspects, and like reference numerals represent like elements.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24A

Figure 24B

Figure 24C

Figure 25

Figure 26A

Figure 26B

Figure 27A

Figure 27B

Figure 27C

Figure 27D

Figure 28A

Figure 28B

Figure 28C

Figure 28D

Figure 29A

Figure 29B

Figure 29C

Figure 29D

Figure 30A

Figure 30B

Figure 30C

Figure 30D

Figure 30E

Figure 30F

Figure 31A

Figure 31B

Figure 31C

Figure 31D

Figure 31E

Figure 31F

Figure 32A

Figure 32B

Figure 32C

Figure 32D

Figure 32E

Figure 32F

Figure 33

[0058] 1. Overview. Aspects of the present disclosure can be integrated into a robotic medical system capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy. This system can perform bronchoscopy, ureteroscopy, gastroscopy, etc. among endoscopic procedures.

[0059] In addition to performing a wide range of procedures, the system can provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system can provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm movements and positions. Furthermore, the system can provide the physician with the ability to perform procedures with improved ease of use such that one or more of the instruments of the system can be controlled by the user alone.

[0060] The following describes various embodiments in conjunction with the drawings for illustrative purposes. It should be understood that many other embodiments of the disclosed concepts are possible and that various advantages can be achieved with the disclosed embodiments. This specification includes headings for reference and to assist in identifying the locations of various sections. These headings are not intended to limit the scope of the concepts described therein. Such concepts can be applied throughout this specification.

[0061] A. Robot System - Cart. A robotic medical system can be configured in various ways depending on the particular procedure. FIG. 1 illustrates one embodiment of a cart-based robotic system 10 arranged for a diagnostic and / or therapeutic bronchoscopy procedure. During bronchoscopy, the system 10 can include a cart 11 having one or more robotic arms 12 for delivering medical instruments, such as a maneuverable endoscope 13 that can be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point (i.e., the patient's mouth positioned on the table in this example) for delivering diagnostic and / or treatment tools. As shown, the cart 11 can be positioned proximate to the upper torso of the patient to provide access to the access point. Similarly, the robotic arm 12 can be actuated to position the bronchoscope relative to the access point. The arrangement of FIG. 1 can also be utilized when performing a gastrointestinal (GI) procedure using a gastroscope, which is an endoscope specialized for GI procedures. FIG. 2 illustrates an exemplary embodiment of the cart in more detail.

[0062] Continuing to refer to FIG. 1, when the cart 11 is properly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient either robotically, manually, or in combination thereof. As shown, the steerable endoscope 13 can include at least two nested portions, such as an inner leader portion and an outer sheath portion, each portion being coupled to a separate instrument driver from a set of instrument drivers 28, and each instrument driver being coupled to the distal end of an individual robotic arm. This linear arrangement of the instrument drivers 28, which facilitates aligning the leader portion coaxially with the sheath portion, creates a "virtual rail" 29 that can be repositioned in space by operating one or more robotic arms 12 at different angles and / or positions. The virtual rails described herein are shown in the figures using dashed lines, and thus the dashed lines do not represent any physical structure of the system. Translation of the instrument drivers 28 along the virtual rail 29 nests the inner leader portion within the outer sheath portion or advances or retracts the endoscope 13 from the patient. The angle of the virtual rail 29 can be adjusted, translated, or pivoted based on clinical use or physician preference. For example, in bronchoscopy, the angle and position of the virtual rail 29 shown provide a compromise between providing the physician access to the endoscope 13 and minimizing the friction resulting from bending the endoscope 13 into the patient's mouth.

[0063] The endoscope 13 can be directed downstream into the patient's trachea and lungs after insertion using precise commands from the robotic system until it reaches the target destination or surgical site. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be operated to extend the inner leader portion telescopically from the outer sheath portion to enhance articulation and increase the bending radius. The use of separate instrument drivers 28 also allows the leader portion and the sheath portion to be driven independently of each other.

[0064] For example, the endoscope 13 can be directed to deliver a biopsy needle to a target, such as a lesion or nodule in a patient's lung. The needle can be disposed downstream of the working channel over the length of the endoscope to obtain a tissue sample for a pathologist to analyze. Depending on the pathology results, additional tools can be disposed downstream of the working channel of the endoscope for additional biopsies. After identifying a nodule as malignant, the endoscope 13 can deliver tools endoscopically to excise potential cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered in separate procedures. In these situations, the endoscope 13 can also be used to deliver a fiducial to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0065] The system 10 can also include a movable tower 30 connected to the cart 11 via a support cable to provide support for control, electronics, fluidics, optics, sensors, and / or power to the cart 11. By placing such functions on the tower 30, the form factor of the cart 11 can be reduced, and the surgeon and their staff performing the surgery can more easily adjust and / or reposition the cart 11. Additionally, the functional split between the cart / table and the support tower 30 reduces clutter in the operating room and facilitates improvement of the clinical workflow. The cart 11 can be positioned near the patient, while the tower 30 can be housed in a remote location so as not to be obstructive during the procedure.

[0066] In the support of the robot system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions in a non-transitory computer-readable storage medium such as a permanent magnetic memory drive, a solid state drive, etc. Execution of these instructions may control the entire system or its subsystem(s), regardless of whether the execution occurs within the tower 30 or within the cart 11. For example, when executed by a processor of a computer system, the instructions may cause the components of the robot system to operate the associated carriage and arm mounts, operate the robot arm, and control the medical instrument. For example, in response to receiving a control signal, a motor within a joint of the robot arm may position the arm in a particular posture.

[0067] Also, the tower 30 may include pumps, flow meters, valve controls, and / or fluid access to provide a controlled perfusion function and a suction function to a system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the perfusion function and the suction function may be delivered directly to the endoscope 13 through separate cable(s).

[0068] The tower 30 may include voltage and surge protection devices designed to provide filtered and protected power to the cart 11, thereby avoiding placing a power transformer and other auxiliary power components within the cart 11, and making the cart 11 smaller and more movable.

[0069] Tower 30 may also include support equipment for sensors deployed throughout the robot system 10. For example, tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robot system 10. In combination with the control system, such optoelectronic equipment may be used to generate real-time images for display on any number of consoles deployed throughout the system, including within tower 30. Similarly, tower 30 may also include an electronic subsystem for receiving signals from deployed electromagnetic (EM) sensors and processing the received signals. Tower 30 may also house and be used to position an EM field generator for detection by EM sensors within or on the medical device.

[0070] Tower 30 may also include console 31, in addition to other consoles available to the rest of the system, such as a console mounted on top of a cart. Console 31 may include a user interface and a display screen, such as a touch screen, for a physician operator. The consoles of system 10 are generally designed to provide both robot control and pre-operative and real-time information for the procedure, such as navigation information and location information for endoscope 13. When console 31 is not the only console available to the physician, a second operator, such as a nurse, may use console 31 to monitor the patient's health or vital signs and the operation of the system and to provide procedure-specific data, such as navigation information and location information. In other embodiments, console 30 is housed within a separate body from tower 30.

[0071] Tower 30 can be coupled to cart 11 and endoscope 13 through one or more cables or connections (not shown). In some embodiments, the support functions from tower 30 can be provided to cart 11 through a single cable, simplifying and organizing the operating room. In other embodiments, specific functions can be coupled at separate wirings and connections. For example, power can be provided to the cart through a single power cable, but support for control, optics, fluidics, and / or navigation can be provided through separate cables.

[0072] FIG. 2 is a detailed illustration of one embodiment of a cart from the cart-based robotic-enabled system shown in FIG. 1. Cart 11 generally includes an elongated support structure 14 (often referred to as a “column”), a cart base 15, and a console 16 at the top of column 14. Column 14 can include one or more carriages, such as a carriage 17 (alternatively an “arm support”) for supporting the deployment of one or more robotic arms 12 (three are shown in FIG. 2). Carriage 17 can include individually configurable arm mounts that rotate along orthogonal axes to adjust the base of robotic arm 12 for better positioning relative to the patient. Carriage 17 also includes a carriage joint 19 that allows carriage 17 to translate vertically along column 14.

[0073] Carriage joint 19 is connected to column 14 through slots, such as slots 20, positioned on both sides of column 14 to guide the vertical translation of carriage 17. Slots 20 include vertical translation joints for positioning and holding the carriage at various vertical heights relative to cart base 15. The vertical translation of carriage 17 allows cart 11 to adjust the reach of robotic arm 12 to accommodate various table heights, patient sizes, and physician preferences. Similarly, the individually configurable arm mounts on carriage 17 allow the robotic arm base 21 of robotic arm 12 to be angled in various configurations.

[0074] In some embodiments, to prevent dirt and fluid from entering the internal chamber of column 14 and the vertical translation joint when carriage 17 translates vertically, a slot cover that is coplanar and parallel to the slot surface may be added to slot 20. The slot cover may be deployed through a pair of spring spools positioned near the vertical top and bottom of slot 20. The cover is coiled within the spool such that as carriage 17 translates vertically up and down, it extends from a coiled state and unfolds until it retracts. The spring mechanism of the spool provides a force to retract the cover to the spool when carriage 17 translates towards the spool, while also maintaining the seal when carriage 17 translates away from the spool. The cover may be connected to carriage 17, for example, using brackets at carriage joint 19, to ensure that the cover extends and retracts properly as carriage 17 translates.

[0075] Column 14 may include mechanisms such as gears and motors internally designed to use a lead screw aligned vertically to mechanically translate carriage 17 in response to a control signal generated in response to a user input, for example, an input from console 16.

[0076] The robotic arm 12 can generally include a robotic arm base 21 and an end effector 22 separated by a series of link mechanisms 23 connected by a series of joints 24. Each joint includes an independent actuator, and each actuator includes a separately controllable motor. Each separately controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints and thus provides seven degrees of freedom. The large number of joints provides a large number of degrees of freedom, enabling "redundant" degrees of freedom. The redundant degrees of freedom enable the robotic arm 12 to position each end effector 22 at a particular position, orientation, and trajectory in space using different link mechanism positions and joint angles. This enables the system to position and orient a medical instrument from a desired point in space, while allowing the physician to move the arm joints to a clinically advantageous position away from the patient, causing greater access while avoiding arm collisions.

[0077] The cart base 15 balances the weight of the column 14, carriage 17, and arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable either movement and / or immobilization of the cart. For example, the cart base 15 includes rollable wheel-shaped casters 25 that enable the cart to be easily moved around the room prior to the procedure. After reaching an appropriate position, the casters 25 can be immobilized using wheel locks to hold the cart 11 in place during the procedure.

[0078] The console 16 positioned at the vertical end of column 14 enables both a user interface for receiving user input and a display screen (or a dual-purpose device such as a touch screen 26, for example), providing both preoperative data and intraoperative data to the user, who is a doctor. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from preoperative computerized tomography (CT) scans, and / or notes from preoperative patient interviews. Intraoperative data on the display may also include essential patient statistics such as respiration, heart rate, and / or pulse, along with optical information provided by the tool, sensor information from the sensors, and coordinate information. The console 16 can be positioned and tilted so that the doctor can access the console from the side of column 14 opposite the carriage 17. From this position, the doctor can view the console 16, the robotic arm 12, and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 that helps operate and stabilize the cart 11.

[0079] Figure 3 illustrates an embodiment of a robot-compatible system 10 arranged for ureteroscopy. In a ureteroscopy procedure, the cart 11 can be positioned to deliver a ureteroscope 32, a procedure-specific endoscope designed to follow the patient's urethra and ureter, to the patient's lower abdominal region. In ureteroscopy, it may be desirable for the ureteroscope 32 to be directly aligned with the patient's urethra to reduce friction and force on sensitive anatomical structures within its range. As shown, the cart 11 can be aligned on the foot side of the table so that the robotic arm 12 can position the ureteroscope 32 to provide direct linear access to the patient's urethra. From the foot side of the table, the robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra along a virtual rail 33.

[0080] After being inserted into the urethra using a control technique similar to that in bronchoscopy, the ureteroscope 32 can be navigated to the bladder, ureter, and / or kidney for diagnostic and / or therapeutic purposes. For example, the ureteroscope 32 can be directed into the ureter and kidney, and a laser lithotripsy device or an ultrasonic lithotripsy device disposed downstream of the working channel of the ureteroscope 32 can be used to break up the formed kidney stones. After the lithotripsy is completed, a basket disposed downstream of the ureteroscope 32 can be used to remove the resulting stone fragments.

[0081] FIG. 4 illustrates one embodiment of a robot-compatible system similarly arranged for vascular procedures. In a vascular procedure, the system 10 can be configured such that the cart 11 can deliver a medical instrument 34, such as a steerable catheter, to an access point in the femoral artery in the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less tortuous path that bypasses the patient's heart, thereby facilitating navigation. As seen in ureteroscopy procedures, the cart 11 can be positioned towards the patient's leg and lower abdomen such that the robotic arm 12 provides direct linear access to the femoral artery access point in the patient's thigh / lumbar region to a virtual rail 35. After insertion into the artery, the medical instrument 34 can be directed and inserted by translating the instrument driver 28. Alternatively, the cart can be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid artery and brachial artery near the shoulder and wrist, for example.

[0082] B. Robot System - Table. Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating the table reduces the amount of capital equipment in the operating room by removing the cart and enables better access to the patient. FIG. 5 illustrates one embodiment of such a robotic-enabled system arranged for a bronchoscopy procedure. System 36 includes a support structure or column 37 for supporting a platform 38 (illustrated as a "table" or "bed") across the floor. Similar to a cart-based system, the end effector of the robotic arm 39 of system 36 includes an instrument driver 42 designed to manipulate an elongate medical instrument, such as the bronchoscope 40 of FIG. 5, through or along a virtual rail 41 formed from the linear alignment of the instrument driver 42. In practice, a C-arm for providing fluoroscopic imaging can be positioned across the entire upper abdominal region of the patient by disposing the emitter and detector around the table 38.

[0083] FIG. 6 is an alternative view of system 36 without a patient and medical instruments for purposes of discussion. As shown, column 37 may include one or more carriages 43 shown as ring-shaped in system 36 that can serve as a base for one or more robotic arms 39. The carriage 43 may translate along a vertical column joint 44 that follows the length of column 37 and provide different viewing positions from which the robotic arm 39 can be positioned to reach the patient. The carriage(s) 43 may rotate about column 37 using a mechanical motor positioned within column 37, enabling the robotic arm 39 to have access to multiple sides of table 38, such as both sides of the patient. In embodiments having multiple carriages, the carriages may be individually positioned on the column and may translate and / or rotate independently of other carriages. The carriage 43 need not surround column 37 and need not even be circular, although the illustrated ring shape facilitates rotation of the carriage 43 about column 37 while maintaining structural balance. The rotation and translation of the carriage 43 enables the system to align medical instruments, such as endoscopes and laparoscopes, to different access points on the patient. In other embodiments (not shown), system 36 may include a patient table or patient bed having an adjustable arm support in the form of a bar or rail that extends alongside it. One or more robotic arms 39 can be attached to the adjustable arm support that can be adjusted in the vertical direction (e.g., via a shoulder having an elbow joint). By providing vertical adjustment, the robotic arm 39 can advantageously be compactly housed under the patient table or patient bed and then raised during the procedure.

[0084] The arm 39 can be attached to the carriage through a set of arm mounts 45 that include a series of joints that can rotate individually and / or extend telescopically to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 can be positioned on the carriage 43 such that when the carriage 43 rotates appropriately, the arm mounts 45 can be positioned on the same side of the table 38 (as shown in FIG. 6), on both sides of the table 38 (as shown in FIG. 9), or on adjacent sides of the table 38 (not shown).

[0085] The column 37 structurally provides a path for supporting the table 38 and for the vertical translation of the carriage. Internally, the column 37 may be equipped with a lead screw for guiding the vertical translation of the carriage and a motor for mechanizing the translation of the carriage based on the lead screw. The column 37 can transmit power signals and control signals to the carriage 43 and to the robotic arm 39 mounted thereon.

[0086] The table base 46 serves a similar function to the cart base 15 of the cart 11 shown in FIG. 2 and houses heavier components to balance the table / bed 38, column 37, carriage 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during treatment. The casters deployed from the bottom of the table base 46 extend in opposite directions on both sides of the base 46 and can retract when it is necessary to move the system 36.

[0087] Continuing with reference to FIG. 6, the system 36 may also include a tower (not shown) for splitting the functions of the system 36 between the table and the tower to reduce the form factor and bulk of the table. As seen in the previously disclosed embodiments, the tower may provide various support functions to the table, such as processing power, computing power, and control power, power, fluidics, and / or optics and sensor processing. The tower may also be movable to be positioned away from the patient to improve physician access and to organize the operating room. Additionally, by placing components in the tower, it is possible to expand the storage space in the table base for potential accommodation of robotic arms. The tower may also include a master controller or console that provides both a user interface for user input, such as a keyboard and / or pendant, and a display screen (or touch screen) for preoperative and intraoperative information, such as real-time imaging, navigation, and tracking information. In some embodiments, the tower may also include a holder for a gas tank used for ventilation.

[0088] In some embodiments, the table base may accommodate and store the robotic arm when not in use. FIG. 7 illustrates a system 47 for accommodating a robotic arm in one embodiment of a table base system. In system 47, the carriage 48 can be translated vertically within the base 49 to accommodate the robotic arm 50, the arm mount 51, and the carriage 48 within the base 49. The base cover 52 can be translated and retracted to open, and the carriage 48, the arm mount 51, and the arm 50 can be deployed about the column 53, and when not in use, the base cover 52 can be closed to accommodate and protect them. The base cover 52 can be sealed with a membrane 54 along the edge of its opening to prevent dirt and fluid intrusion when closed.

[0089] FIG. 8 illustrates an embodiment of a robotic-enabled table-based system configured for ureteroscopy procedures. In ureteroscopy, table 38 may include a swivel portion 55 for positioning the patient at an off-angle from column 37 and table base 46. The swivel portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) to position the bottom portion of the swivel portion 55 away from column 37. For example, pivoting of the swivel portion 55 enables a C-arm (not shown) to be positioned across the patient's lower abdomen without interfering with the space of a column (not shown) below table 38. By rotating a carriage 35 (not shown) around column 37, robotic arm 39 may directly insert ureteroscope 56 along virtual rail 57 into the patient's groin to reach the urethra. In ureteroscopy, buttress 58 may be fixed to the swivel portion 55 of table 38 to support the position of the patient's legs during this procedure and enable clear access to the patient's groin.

[0090] In laparoscopic procedures, minimally invasive instruments may be inserted into a patient's anatomical structure through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include an elongate rigid member such as a shaft used to access anatomical structures within the patient. After inflation of the patient's abdomen, the instrument may be directed to perform surgical or medical tasks such as grasping, cutting, ablation, suturing, and the like. In some embodiments, the instrument may include a scope such as a laparoscope. FIG. 9 is an embodiment of a robotic-enabled table-based system configured for laparoscopic procedures. As shown in FIG. 9, the carriage 43 of system 36 may be rotated and adjusted vertically to position a pair of robotic arms 39 on either side of table 38 such that instrument 59 can be positioned using arm mount 45 to reach the patient's abdomen through minimal incisions on either side of the patient.

[0091] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. FIG. 10 illustrates one embodiment of a robotic-enabled medical system with pitch or tilt adjustment. As shown in FIG. 10, system 36 may be adapted to the tilt of table 38 and position a portion of the table further away from the floor than other portions. Additionally, arm mount 45 may rotate in accordance with the tilt so that arm 39 maintains the same planar relationship with table 38. To accommodate steep angles, column 37 may also include a nested portion 60 that allows for vertical extension of column 37 to prevent table 38 from contacting the floor or colliding with base 46.

[0092] FIG. 11 is a detailed illustrative view of the junction between table 38 and column 37. Pitch rotation mechanism 61 may be configured to vary the pitch angle of table 38 relative to column 37 with multiple degrees of freedom. Pitch rotation mechanism 61 may be actuated by positioning orthogonal axes 1, 2 at the junction between the column and the table, with each axis being operated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 allows for tilt adjustment along one axis 1, while rotation along the other screw 6 allows for tilt adjustment along the other axis 2. In some embodiments, a ball joint may be used to vary the pitch angle of table 38 relative to column 37 with multiple degrees of freedom.

[0093] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position, i.e., when attempting to position the patient's lower abdomen higher above the floor than the patient's upper abdomen for lower abdominal surgery. The Trendelenburg position slides the patient's internal organs towards the patient's upper abdomen by gravity, emptying the abdominal cavity when performing minimally invasive procedures or medical procedures in the lower abdomen, such as laparoscopic prostatectomy, using low-invasive tools.

[0094] Figures 12 and 13 are isometric and end views of an alternative embodiment of a table-based surgical robot system 100. The surgical robot system 100 includes one or more adjustable arm supports 105 (see, e.g., FIG. 14) configured to support one or more robot arms relative to a table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, although additional arm supports can be provided on the opposite side of the table 101. The adjustable arm support 105 is configured to move relative to the table 101 to adjust and / or change the position of the adjustable arm support 105 and / or any robot arm attached thereto relative to the table 101. For example, the adjustable arm support 105 can be adjusted relative to the table 101 with one or more degrees of freedom. The adjustable arm support 105 provides a high degree of versatility to the system 100, including the ability to easily accommodate one or more adjustable arm supports 105 and any robot arms attached thereto under the table 101. The adjustable arm support 105 can be raised from a storage position to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from a storage position to a position above the upper surface of the table 101.

[0095] The adjustable arm support 105 can provide several degrees of freedom, including rising, lateral translation, tilting, etc. In the illustrated embodiment of FIGS. 12 and 13, the arm support 105 is configured with four degrees of freedom, which are illustrated by arrows in FIG. 12. The first degree of freedom enables adjustment of the adjustable arm support 105 in the z - direction (“Z - rise”). For example, the adjustable arm support 105 can include a carriage 109 configured to move up and down along or relative to a column 102 that supports the table 101. The second degree of freedom can enable the adjustable arm support 105 to tilt. For example, the adjustable arm support 105 can include a swivel joint, which can enable alignment of the adjustable arm support 105 with a Trendelenburg - position bed. The third degree of freedom can enable the adjustable arm support 105 to be “pivoted up” and used to adjust the distance between the side of the table 101 and the adjustable arm support 105. The fourth degree of freedom can enable translation of the adjustable arm support 105 along the longitudinal length of the table.

[0096] The surgical robot system 100 of FIGS. 12 and 13 can include a table supported by a column 102 mounted to a base 103. The base 103 and the column 102 support the table 101 relative to a support surface. The floor axis 131 and the support axis 133 are shown in FIG. 13.

[0097] The adjustable arm support 105 can be mounted to the column 102. In other embodiments, the arm support 105 can be mounted to the table 101 or the base 103. The adjustable arm support 105 can include a carriage 109, a bar or rail connector 111, and a bar or rail 107. In some embodiments, one or more robot arms mounted to the rail 107 can translate and move relative to each other.

[0098] The carriage 109 can be attached to the column 102 by the first joint 113, thereby enabling the carriage 109 to move relative to the column 102 (e.g., move up and down along the first axis, i.e., the vertical axis 123). The first joint 113 can provide a first degree of freedom ( "Z lift") to the adjustable arm support 105. The adjustable arm support 105 can include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 can include a third joint 117 that can provide a third degree of freedom ( "pivot up") to the adjustable arm support 105. As the rail connector 111 is rotated about the third axis 127, a further joint 119 (shown in FIG. 13) can be provided to mechanically constrain the third joint 117 so as to maintain the orientation of the rail 107. The adjustable arm support 105 can include a fourth joint 121 that can provide a fourth degree of freedom (translation) to the adjustable arm support 105 along the fourth axis 129.

[0099] FIG. 14 is an end view of a surgical robot system 140A with two adjustable arm supports 105A, 105B mounted on both sides of a table 101. The first robotic arm 142A is attached to the bar or rail 107A of the first adjustable arm support 105A. The first robotic arm 142A includes a base 144A attached to the rail 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or robotic medical tools. Similarly, the second robotic arm 142B includes a base 144B attached to the rail 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or robotic medical tools.

[0100] In some embodiments, one or more of the robotic arms 142A, 142B include an arm having seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B include an insertion axis (one degree of freedom including insertion), a wrist (three degrees of freedom including wrist pitch, yaw, and roll), an elbow (one degree of freedom including elbow pitch), a shoulder (two degrees of freedom including shoulder pitch and yaw), and a base 144A, 144B (one degree of freedom including translation), and can include eight degrees of freedom. In some embodiments, the insertion degree of freedom can be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides the insertion via an insertion architecture of the instrument base.

[0101] C. Instrument Driver and Junction. The end effector of the robotic arm of the system includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates electromechanical means for actuating a medical instrument, and (ii) a removable or detachable medical instrument that may lack any electromechanical components such as motors. This dichotomy may be caused by the need to sterilize the medical instruments used in medical procedures and the inability to adequately sterilize expensive capital equipment due to the complex mechanical assembly and sensitive electronics of the medical instruments. Thus, the medical instrument can be designed to be detached from, removed from, and replaced from the instrument driver (and thus its system) upon individual sterilization or disposal by a physician or the physician's staff. In contrast, the instrument driver need not be replaced or sterilized and can be draped for protection.

[0102] FIG. 15 illustrates an exemplary instrument driver. The instrument driver 62 positioned at the distal end of the robotic arm is composed of one or more drive units 63 arranged on parallel axes to provide controlled torque to the medical instrument via a drive shaft 64. Each drive unit 63 includes an individual drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and operating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide a plurality (shown as four in FIG. 15) of independent drive outputs to the medical instrument. During operation, the control circuit 68 receives control signals, transmits motor signals to the motor 66, compares the motor speed obtained as a result measured by the encoder 67 with the desired speed, and modulates the motor signals to generate the desired torque.

[0103] For procedures that require a sterile environment, the robotic system can incorporate a drive junction, such as a sterile adapter connected to a sterile drape, positioned between the instrument driver and the medical instrument. The main purpose of the sterile adapter is to physically separate the drive shaft and the drive input, and as a result, transmit angular motion from the drive shaft of the instrument driver to the drive input of the instrument while maintaining sterility. Thus, an exemplary sterile adapter can be composed of a series of rotary input and output parts intended to mate with the drive shaft of the instrument driver, and a drive input to the instrument. The sterile drape connected to the sterile adapter is composed of a thin flexible material such as transparent plastic or semi-transparent plastic, and is designed to cover capital equipment such as instrument drivers, robotic arms, and carts (in cart-based systems) or tables (in table-based systems). The use of the drape makes it possible to position the capital equipment close to the patient while still being located in an area that does not require sterilization (i.e., the non-sterile field). On the other side of the sterile drape, the medical instrument can come into contact with the patient in an area that requires sterilization (i.e., the sterile field).

[0104] D. Medical Instrument. FIG. 16 illustrates an exemplary medical instrument having a paired instrument driver. Similar to other instruments designed for use in a robotic system, medical instrument 70 includes an elongate shaft 71 (or elongate body) and an instrument base 72. The instrument base 72, also referred to as an “instrument handle” by virtue of its design intended for manual interaction by a physician, generally includes a rotary drive input portion 73 designed to mate with a drive output portion 74 that penetrates a drive engagement on instrument driver 75 at the distal end of robotic arm 76. The drive input portion 73 can include, for example, a receptacle, pulley, or spool. When physically connected, latched, and / or coupled, the mating drive input portion 73 of instrument base 72 can share a rotational axis with drive output portion 74 in instrument driver 75 to enable transmission of torque from drive output portion 74 to drive input portion 73. In some embodiments, drive output portion 74 can include splines designed to mate with a receptacle on drive input portion 73.

[0105] The elongate shaft 71 is designed to be delivered through either an anatomical opening or lumen, such as in an endoscope, or a minimally invasive incision, such as in laparoscopy. The elongate shaft 71 can be either flexible (e.g., having characteristics similar to an endoscope) or rigid (e.g., having characteristics similar to a laparoscope), or can include a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongate shaft extends to an end effector formed from a knuckle having at least one degree of freedom and can be actuated based on a force from a tendon as the drive input portion rotates in response to torque received from drive output portion 74 of instrument driver 75 and can be connected to a surgical tool or medical instrument, such as a gripper or forceps. When designed for endoscopy, the distal end of the flexible elongate shaft can include a steerable or controllable bending portion that can articulate and bend based on torque received from drive output portion 74 of instrument driver 75.

[0106] The torque from the instrument driver 75 is transmitted downstream of the elongate shaft 71 using a tendon along the shaft 71. These individual tendons, such as pull wires, can be individually secured to the respective drive input portions 73 within the instrument handle 72. From the handle 72, the tendon travels through one or more pull lumens along the elongate shaft 71 and is secured to the distal portion of the elongate shaft 71 or to a list at the distal portion of the elongate shaft. During a surgical procedure, such as a laparoscopic procedure, an endoscopic procedure, or a hybrid procedure, these tendons can be coupled to a distally mounted end effector such as a list, a grasping tool, or forceps. Under such an arrangement, the torque applied to the drive input portion 73 transmits tension to the tendon, thereby causing the end effector to be actuated in some manner. In some embodiments, during the surgical procedure, the tendon can rotate the joint about an axis, thereby moving the end effector in one direction or another. Alternatively, the tendon can be connected to one or more jaws of a grasping tool at the distal end of the elongate shaft 71, and the grasping tool closes in response to the tension from the tendon.

[0107] In an endoscopic examination, a tendon can be coupled to a flexure portion or an articulation portion positioned along (e.g., at a distal end of) an elongate shaft 71 via an adhesive, a control ring, or other mechanical fixation. When fixedly attached to the distal end of the flexure portion, torque exerted on a drive input portion 73 is transmitted to the tendon, causing a more flexible flexure portion (which may be referred to as an articulation portion or an articulation region) to flex or articulate. Along the non-flexure portion, it may be beneficial to spiral or coil individual pull lumens that direct individual tendons along (or inside) the wall of the endoscopic shaft to balance the radial forces resulting from the tension in the pull wires. The angle of the spiral and / or the spacing between these can be varied or designed for a specific purpose. As the spiral becomes narrower, it exhibits inferior shaft compression under load forces, while decreasing the number of spirals results in superior shaft compression under load forces but also indicates a limit to flexion. At the other end of the spectrum, directing the pull lumens parallel to the longitudinal axis of the elongate shaft 71 can enable controlled articulation in a desired flexure portion or articulation portion.

[0108] In an endoscopic examination, the elongate shaft 71 houses several components that assist in robotic procedures. The shaft can be configured with a working channel for deploying a surgical tool (or medical instrument), irrigation, and / or suction to the surgical area at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers that transmit signals to / from an optical assembly at the distal tip, and the optical assembly may include an optical camera. The shaft 71 may also house an optical fiber for conveying light from a proximally positioned light source, such as a light-emitting diode, to the distal end of the shaft.

[0109] At the distal end of the instrument 70, the distal tip may include an opening of a working channel for delivering tools to the surgical site for diagnosis and / or treatment, irrigation, and aspiration. Further, the distal tip may include a port for a camera, such as a fiber optic scope or a digital camera, for capturing an image of the internal anatomical space. In this regard, the distal tip may also include a port for a light source for illuminating the anatomical space when using the camera.

[0110] In the example of FIG. 16, the drive shaft axis, and thus the drive input axis, is orthogonal to the axis of the elongate shaft. However, this arrangement complicates the rolling ability of the elongate shaft 71. As a result of rolling the elongate shaft 71 along its axis while keeping the drive input portion 73 stationary, as the tendon exits the drive input portion 73 and enters the pull lumen within the elongate shaft 71, it results in an undesirable entanglement of the tendon. Such resulting tendon entanglement may interfere with any control algorithm intended to predict the movement of the flexible elongate shaft during an endoscopic procedure.

[0111] FIG. 17 illustrates an alternative design of an instrument driver and an instrument where the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, the circular instrument driver 80 includes four drive units in a state where the drive output portions 81 are aligned in parallel at the end of the robot arm 82. The drive units and their respective drive output portions 81 are accommodated in a rotating assembly 83 of the instrument driver 80 driven by one of the drive units within the assembly 83. In response to the torque provided by the rotary drive unit, the rotating assembly 83 rotates along a circular bearing connecting the rotating assembly 83 to the non-rotating portion 84 of the instrument driver. Power and control signals can be communicated from the non-rotating portion 84 of the instrument driver 80 to the rotating assembly 83 through electrical contacts and can be maintained through rotation by a brush-type slip ring connection (not shown). In other embodiments, the rotating assembly 83 is integrated with the non-rotatable portion 84 and can thus respond to a separate drive unit that is not parallel to the other drive units. The rotating mechanism 83 enables the instrument driver 80 to rotate the drive units and their respective drive output portions 81 as a single unit about the instrument driver axis 85.

[0112] Similar to the previously disclosed embodiments, the instrument 86 can include an elongated shaft portion 88 and an instrument base 87 (shown with a transparent outer skin for illustrative purposes) that includes a plurality of drive input portions 89 (such as receptacles, pulleys, and spools) configured to receive the drive output portions 81 within the instrument driver 80. Different from the previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87 in a state where its axis is substantially parallel to the axis of the drive input portions 89 rather than being orthogonal as seen in the design of FIG. 16.

[0113] When coupled to the rotating assembly 83 of the instrument driver 80, the medical instrument 86, which includes an instrument base 87 and an instrument shaft 88, rotates with the rotating assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is positioned at the center of the instrument base 87, the instrument shaft 88 will be coaxial with the instrument driver axis 85 when attached. Thus, rotation of the rotating assembly 83 causes the instrument shaft 88 to rotate about its longitudinal axis. Further, since the instrument base 87 rotates with the instrument shaft 88, any tendon connected to the drive input portion 89 on the instrument base 87 will not become entangled during rotation. Thus, the parallelism of the axes of the drive output portion 81, the drive input portion 89, and the instrument shaft 88 enables shaft rotation without entangling any control tendons.

[0114] FIG. 18 illustrates an instrument having an instrument base insertion architecture according to some embodiments. The instrument 150 can be coupled to any of the instrument drivers discussed above. The instrument 150 includes an elongate shaft 152, an end effector 162 connected to the elongate shaft 152, and a handle 170 coupled to the elongate shaft 152. The elongate shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongate shaft 152 includes one or more channels or grooves 158 along its outer surface. The grooves 158 are configured to receive one or more wires or cables 180. Thus, one or more cables 180 extend along the outer surface of the elongate shaft 152. In other embodiments, the cables 180 can also pass through the elongate shaft 152. Actuation of the end effector 162 is effected by operation of one or more cables 180 (e.g., via an instrument driver).

[0115] The instrument handle 170, which may also be referred to as an instrument base, can include an attachment joint 172 having one or more mechanical input portions 174, such as receptacles, pulleys, or spools, generally designed to mate with one or more torque couplers on the attachment surface of the instrument driver.

[0116] In some embodiments, the instrument 150 includes a series of pulleys or cables that allow the elongate shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that accommodates the insertion of the instrument, thereby effecting the insertion of the instrument 150 with minimal reliance on a robotic arm. In other embodiments, the robotic arm may primarily bear the burden of instrument insertion.

[0117] E. Controller. Any of the robotic systems described herein can include an input device or controller for operating an instrument attached to a robotic arm. In some embodiments, the instrument and the controller can be coupled (e.g., communicatively, electronically, wirelessly, and / or mechanically) such that operation of the controller causes a corresponding operation of the instrument, e.g., via master-slave control.

[0118] FIG. 19 is a perspective view of one embodiment of a controller 182. In this embodiment, the controller 182 includes a hybrid controller that can have both impedance control and admittance control. In other embodiments, the controller 182 can utilize only impedance control or passive control. In other embodiments, the controller 182 can utilize only admittance control. By being a hybrid controller, the controller 182 can advantageously reduce perceptual inertia during use.

[0119] In the illustrated embodiment, the controller 182 is configured to enable the operation of two medical instruments and includes two handles 184. Each handle 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0120] As shown in FIG. 19, each positioning platform 188 includes a SCARA arm (selective compliance assembly robot arm) 198 coupled to a column 194 by a linear joint 196. The linear joint 196 is configured to translate along the column 194 (e.g., along a rail 197) to enable each of the handles 184 to translate in the z direction and to provide a first degree of freedom. The SCARA arm 198 is configured to enable movement of the handle 184 in the x-y plane and to provide two additional degrees of freedom.

[0121] In some embodiments, one or more load cells are positioned within the controller. For example, in some embodiments, a load cell (not shown) is positioned on each body of the gimbal 186. By providing the load cell, a portion of the controller 182 can operate under admittance control, thereby advantageously reducing the perceived inertia of the controller during use. In some embodiments, the positioning platform 188 is configured for admittance control while the gimbal 186 is configured for impedance control. In other embodiments, the gimbal 186 is configured for admittance control while the positioning platform 188 is configured for impedance control. Thus, in some embodiments, the translational or positional degrees of freedom of the positioning platform 188 can depend on admittance control while the rotational degrees of freedom of the gimbal 186 depend on impedance control.

[0122] F. Navigation and Control. Conventional endoscopies can involve the use of fluoroscopy (such as that which can be delivered through a C-arm) and other forms of radiation-based imaging modalities to provide intra-cavity guidance to the operator, the physician. In contrast, the robotic systems contemplated by the present disclosure can provide non-radiation-based navigation means and localization means to reduce the physician's exposure to radiation and to reduce the amount of equipment in the operating room. As used herein, the term "localization" can refer to determining and / or monitoring the position of an object within a reference coordinate system. Techniques such as pre-operative mapping, computer vision, real-time EM tracking, and robotic command data can be used individually or in combination to achieve a radiation-free surgical environment. In other cases where radiation-based imaging modalities are still used, pre-operative mapping, computer vision, real-time EM tracking, and robotic command data can be used individually or in combination to improve information obtainable only through the radiation-based imaging modality.

[0123] FIG. 20 is a block diagram illustrating a localization system 90 that estimates the position of one or more elements of a robotic system, such as the position of an instrument, according to an exemplary embodiment. The localization system 90 can be a set of one or more computer devices configured to execute one or more instructions. The computer device can be embodied by one processor (or a plurality of processors) and computer-readable memory in one or more of the components discussed above. By way of example, the computer device can be in the tower 30 shown in FIG. 1, the cart shown in FIGS. 1-4, the bed shown in FIGS. 5-14, etc., but is not limited thereto.

[0124] As shown in FIG. 20, the location system 90 may include a location module 95 that processes input data 91-94 to generate location data 96 for the distal tip of a medical instrument. The location data 96 may be data or logic representing the location and / or orientation of the distal end of the instrument relative to a reference system. The reference system can be a reference system relative to the patient's anatomical structure or to a known object such as an EM field generator (see the following discussion regarding EM field generators).

[0125] Here, the various input data 91-94 will be described in more detail. Preoperative mapping can be achieved through the use of the collection of low-dose CT scans. The preoperative CT scans are reconstructed, for example, into a three-dimensional image that is visualized as a "slice" of a cross-sectional view of the patient's internal anatomical structure. When analyzed as a whole, an image-based model can be generated that targets anatomical cavities, anatomical spaces, and anatomical structures of the patient, such as the patient's pulmonary network. Techniques such as centerline shapes can be determined from the CT images and approximated to create a three-dimensional volume of the patient's anatomical structure, referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline shapes is discussed in U.S. Patent Application No. 14 / 523,760, the entire content of which is incorporated herein by reference. A network phase model can also be derived from CT images and is particularly suitable for bronchoscopy.

[0126] In some embodiments, the instrument may be equipped with a camera to provide visual data 92. The localization module 95 may process the visual data to enable one or more vision-based position tracking. For example, preoperative model data may be used in conjunction with visual data 92 to enable computer vision-based tracking of a medical instrument (e.g., an endoscope, or an instrument advancing through the working channel of an endoscope). For example, using preoperative model data 91, the robotic system may generate a library of predicted endoscope images from the model based on the expected movement path of the endoscope, with each image linked to a position within the model. During the procedure, this library may be referenced by the robotic system to assist in localization by comparing real-time images captured by a camera (e.g., a camera at the distal end of the endoscope) to those in the image library.

[0127] Other computer vision-based tracking techniques use feature tracking to determine the movement of the camera and, by extension, the movement of the endoscope. Some features of the localization module 95 may identify circular geometric shapes in the preoperative model data 91 that correspond to anatomical lumens, and track changes in these geometric shapes to determine which anatomical lumen has been selected and the relative rotational and / or translational movement of the camera. The use of phase maps may further improve vision-based algorithms or vision-based approaches.

[0128] Optical flow, another computer vision-based technique, may analyze the displacement and translation of image pixels within a video sequence in the visual data 92 to infer the movement of the camera. Examples of optical flow techniques may include motion detection, object segmentation calculation, luminance, motion-compensated coding, stereo parallax measurement, etc. By comparing multiple frames over multiple iterations, the movement and position of the camera (and by association, the endoscope) may be determined.

[0129] The position - specific module 95 can generate the real - time position of the endoscope within a global coordinate system that can be aligned with the patient's anatomical structure represented by the preoperative model using real - time EM tracking. In EM tracking, an EM sensor (or tracker) composed of one or more sensor coils embedded in a medical instrument (e.g., an endoscope tool) at one or more positions and orientations measures the variations in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be disposed near the patient to generate a low - intensity magnetic field that can be detected by the embedded sensors. The magnetic field induces a small current in the sensor coils of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "registered" intraoperatively to the patient's anatomical structure (e.g., the preoperative model) to determine the geometric transformation that aligns the position in the preoperative model of the patient's anatomical structure with a single position in the coordinate system. Once registered, an EM tracker embedded at one or more positions of the medical instrument (e.g., the distal tip of the endoscope) can provide a real - time display of the progress of the medical instrument through the patient's anatomical structure.

[0130] The robot commands and kinematic data 94 can also be used by the position - specific module 95 to provide position - specific data 96 for the robotic system. During preoperative calibration, the device pitch and yaw resulting from the joint movement commands can be determined. During the procedure, these calibration measurements can be used in combination with known insertion - depth information to estimate the position of the instrument. Alternatively, these calculations can be analyzed in combination with EM, vision, and / or phase modeling to estimate the position of the medical instrument within the network.

[0131] As shown in FIG. 20, some other input data can be used by the localization module 95. For example, although not shown in FIG. 20, an instrument utilizing shape-sensing fibers can provide shape data that the localization module 95 can use to determine the position and shape of the instrument.

[0132] The localization module 95 can use the input data 91-94 in combination (multiple possible). In some cases, such a combination may use a probabilistic approach where the localization module 95 assigns a confidence weight to the positions determined from each of the input data 91-94. Thus, if the EM data is not reliable (such as in the case of EM interference), it may reduce the reliability of the position determined by the EM data 93, and the localization module 95 may rely more strongly on the visual data 92 and / or the robot commands and kinematic data 94.

[0133] As discussed above, the robotic systems discussed in this specification can be designed to incorporate one or a combination of two or more of the above techniques. The computer-based control system of a robotic system based on a tower, bed, and / or cart can store computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, etc. The computer program instructions, when executed, cause the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and localization data such as the position of the instrument in the global coordinate system, an anatomical map, etc.

[0134] 2. Dynamic flexible scope drive device and method of using the same. This application discloses a robotic medical system that includes a robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope. The system can provide a display of the position or orientation of the flexible scope relative to a reference position.

[0135] In some embodiments, the robotic medical system can provide a display via a user interface or a display device of the robotic medical system. For example, in some embodiments, the robotic medical system provides a visual display of the position or orientation of a flexible scope (e.g., at the distal end of the scope).

[0136] In some embodiments, the robotic medical system is configured to operate the flexible scope in a particular mode of a plurality of modes (e.g., the robotic medical system may operate the flexible scope in a first mode of the plurality of modes at a first time and in a second mode different from the first mode of the plurality of modes at a second time different from the first time).

[0137] In some embodiments, the robotic medical system is configured to determine the position and / or orientation of the flexible scope. In some embodiments, the robotic medical system determines that the position or orientation of the scope has changed and updates an indicator to indicate the changed position or orientation of the scope. In some embodiments, the robotic medical system updates the indicator in real time (e.g., while the scope is moving) to indicate the changed position or orientation of the scope.

[0138] A. Robotic system. Figure 21 illustrates an exemplary robotic medical system 200 according to some embodiments. In some embodiments, the robotic medical system 200 is a robotic surgical system. In the example of FIG. 21, the robotic medical system 200 includes a patient support platform 202 (e.g., a patient platform, table, bed, etc.). Two ends along the length of the patient support platform 202 are respectively referred to as the "head" and the "foot". Two sides of the patient support platform 202 are respectively referred to as "left" and "right". The patient support platform 202 includes a support 204 (e.g., a rigid frame) for the patient support platform 202.

[0139] The robotic medical system 200 also includes a base 206 for supporting the robotic medical system 200. The base 206 includes wheels 208 that enable the robotic medical system 200 to be easily movable or repositionable in a physical environment. In some embodiments, the wheels 208 are omitted from the robotic medical system 200 or retractable, and the base 206 can be placed directly on the ground or floor. In some embodiments, the wheels 208 are replaced by feet.

[0140] The robotic medical system 200 includes one or more robotic arms 210. The robotic arms 210 can be configured to perform robotic medical procedures as described above with reference to FIGS. 1-20. FIG. 21 shows five robotic arms 210, but it should be understood that the robotic medical system 200 can include any number of robotic arms, less than five, or six or more.

[0141] The robotic medical system 200 also includes one or more bars 220 (e.g., adjustable arm supports or adjustable bars) that support the robotic arms 210. Each of the robotic arms 210 is supported and movably coupled on the bar 220 by respective base joints of the robotic arms. In some embodiments, as described in FIG. 12, the bar 220 can provide several degrees of freedom including elevation, lateral translation, tilt, etc. In some embodiments, each of the robotic arms 210 and / or the adjustable arm support 220 is also referred to as a respective kinematic chain.

[0142] FIG. 21 shows three robotic arms 210 supported by a bar 220 within the view of the figure. The remaining two robotic arms are supported by another bar located along the other length of the patient support platform 202.

[0143] In some embodiments, the adjustable arm support 220 can be configured to provide a base position for one or more of the robotic arms 210 for robotic medical procedures. The robotic arms 210 can be positioned relative to the patient support platform 202 by translating the robotic arms 210 along the length of the underlying bar 220 and / or by adjusting the position and / or orientation of the robotic arms 210 via one or more joints and / or links (see, e.g., FIG. 23). In some embodiments, the bar posture may be changed via manual operation, teleoperation, and / or powered assisted movement.

[0144] In some embodiments, the adjustable arm support 220 can translate along the length of the patient support platform 202. In some embodiments, translation of the bar 220 along the length of the patient support platform 202 causes one or more of the robotic arms 210 supported by the bar 220 to translate simultaneously with or relative to the bar. In some embodiments, the bar 220 can translate while keeping one or more of the robotic arms stationary relative to the base 206 of the robotic medical system 200.

[0145] In the example of FIG. 21, the adjustable arm support 220 is positioned along the length of the patient support platform 202. In some embodiments, the adjustable arm support 220 can extend over a partial or full length of the patient support platform 202 and / or over a partial or full width of the patient support platform 202.

[0146] According to some embodiments, during a robotic medical procedure, one or more of the robotic arms 210 can also be configured to hold an instrument 212 (e.g., an endoscope and / or any other instrument that can be used during surgery (e.g., a sensor, a lighting device, a cutting device, etc.), such as a robotically controlled medical instrument or tool), and / or can be coupled to one or more accessories that include one or more cannulas.

[0147] Figure 22 is another view of the exemplary robotic medical system 200 of FIG. 21 according to some embodiments. In this example, the robotic medical system 200 includes six robotic arms 210-1, 210-2, 210-3, 210-4, 210-5, and 210-6. The patient platform 202 is supported by a column 214 that extends between the base 206 and the patient platform 202. In some embodiments, the patient platform 202 includes a tilt mechanism 216. The tilt mechanism 216 can be positioned between the column 214 and the patient platform 202 to enable the patient platform 202 to pivot, rotate, or tilt relative to the column 214. The tilt mechanism 216 can be configured to allow lateral and / or longitudinal tilting of the patient platform 202. In some embodiments, the tilt mechanism 216 allows simultaneous lateral and longitudinal tilting of the patient platform 202.

[0148] FIG. 22 shows the patient platform 202 in a non-tilted state or non-tilted position. In some embodiments, the non-tilted state or non-tilted position is the default position of the patient platform 202. In some embodiments, the default position of the patient platform 202 is a substantially horizontal position as shown in FIG. 22. As illustrated, in the non-tilted state, the patient platform 202 can be positioned horizontally or parallel to the surface (e.g., the floor or the ground) that supports the robotic medical system 200. In some embodiments, the term "non-tilted" refers to a state where the angle between the default position and the current position is less than a threshold angle (e.g., less than 5 degrees, or less than the angle at which a patient would shift on the patient platform). In some embodiments, the term "non-tilted" refers to a state where the patient platform is substantially perpendicular to the direction of gravity regardless of the angle formed by the surface that supports the robotic medical system with respect to gravity.

[0149] Continuing to refer to FIG. 22, in the illustrated example of the robotic medical system 200, the patient platform 202 includes a support 204. In some embodiments, the support 204 includes a rigid support structure or frame and can support one or more surfaces, pads, or cushions 222. The upper surface of the patient platform 202 can include a support surface 224. During a medical procedure, the patient can be placed on the support surface 224.

[0150] FIG. 22 shows the robotic arm 210 and the adjustable arm support 220 in an exemplary deployed configuration where the robotic arm 210 reaches above the patient platform 202. In some embodiments, due to the configuration of the robotic medical system 200 that allows for the accommodation of different components below the patient platform 202, the robotic arm 210 and the arm support 220 can occupy the space below the patient platform 202. Thus, in some embodiments, the tilt mechanism 216 has a thin profile and / or a low volume to increase the space available for lower storage.

[0151] FIG. 22 also illustrates an exemplary x, y, and z coordinate system that can be used to explain certain features of the embodiments disclosed herein. It is understood that this coordinate system is provided for illustrative and explanatory purposes only and that other coordinate systems may be used. In the example illustrated, the x direction or x-axis extends laterally across patient platform 202 when patient platform 202 is in the non-inclined state. In some configurations, the x direction extends across patient platform 202 from one lateral side (e.g., the right side) to the other lateral side (e.g., the left side) of patient platform 202 when patient platform 202 is in the non-inclined state. The y direction or y-axis extends longitudinally along patient platform 202 when patient platform 202 is in the non-inclined state. That is, the y direction extends along patient platform 202 from one longitudinal end (e.g., the head end) to the other longitudinal end (e.g., the foot end) of patient platform 202 when patient platform 202 is in the non-inclined state. In the non-inclined state, patient platform 202 may be in the x-y plane that is parallel to the floor or ground, or may be parallel to the x-y plane. In the example illustrated, the z direction or z-axis extends vertically along column 214. In some embodiments, tilt mechanism 216 is configured to tilt patient platform 202 laterally by rotating patient platform 202 about a lateral tilt axis that is parallel to the y-axis. Tilt mechanism 216 can be further configured to tilt patient platform 202 longitudinally by rotating patient platform 202 about a longitudinal tilt axis that is parallel to the x-axis.

[0152] In some embodiments, the robotic medical system 200 includes a tower 230 (e.g., a tower viewer) or a physician console 240 (or both), as illustrated in FIG. 23. The tower 230 may correspond to the tower 30 described above and may provide support for control, electronics, fluidics, optics, sensors, and / or power for the patient support platform 202 and the physician console 240. In some embodiments, the tower 230 includes a display device 232. The display device 232 can include a user interface for displaying a surgical field acquired by one or more cameras 606 of the robotic medical system and / or one or more notifications to an operator of the robotic medical system 200. In some embodiments, the physician console 240 can include a display device 242 having a user interface that a physician operator uses to operate the patient support platform 202. For example, the display device 242 may include a user interface for displaying a surgical field acquired by one or more cameras 606 of the robotic medical system and / or one or more notifications to an operator of the robotic medical system 200. The physician console 240 can provide both robotic control and pre-operative and real-time information of a medical procedure to the physician operator. In some embodiments, the physician console 240 includes one or more input devices (e.g., buttons, switches, touch-sensitive surfaces, gimbals, etc.), such as a foot pedal 244.

[0153] B. Robotic Arm. FIGS. 24A-24C are different views of an exemplary robotic arm 210 according to some embodiments.

[0154] FIG. 24A illustrates that the robotic arm 210 includes a plurality of links 302 (e.g., a link mechanism). The links 302 (e.g., 302-1 to 302-4) are connected by one or more joints 304 (e.g., 304-1 to 304-5). Each of the joints 304 includes one or more degrees of freedom (DoF).

[0155] In FIG. 24A, the joints 304 include a first joint 304-1 (e.g., a base joint or A0 joint) located at or near the base 306 of the robotic arm 210. In some embodiments, the base joint 304-1 includes a linear joint that enables the robotic arm 210 to translate along the bar 220 (e.g., along the y-axis). The joints 304 also include a second joint 304-2. In some embodiments, the second joint 304-2 rotates relative to the base joint 304-1. The joints 304 also include a third joint 304-3 connected to one end of the link 302-2. In some embodiments, the joint 304-3 includes multiple DoF and facilitates both the inclination and rotation of the link 302-2 with respect to the joint 304-3.

[0156] FIG. 24A also shows a fourth joint 304-4 connected to the other end of the link 302-2. In some embodiments, the joint 304-4 includes an elbow joint that connects the link 302-2 and the link 302-3. The joints 304 further include a pair of joints 304-5 (e.g., a wrist roll joint) and 304-6 (e.g., a wrist pitch joint) located at the distal portion of the robotic arm 210.

[0157] The proximal end of the robotic arm 210 may be connected to the base 306, and the distal end of the robotic arm 210 may be connected to an advanced device manipulator (ADM) 308 (e.g., a tool driver, an instrument driver, or a robotic end effector, etc.). The ADM 308 may be configured to control the positioning and operation of a medical instrument (e.g., a tool, a scope, etc.).

[0158] The robotic arm 210 can also include a cannula sensor 310 for detecting the presence of the cannula or the proximity of the cannula to the robotic arm 210. In some embodiments, the robotic arm 210 is placed in a docking state (e.g., a docking position) when the cannula sensor 310 detects the presence of the cannula (e.g., via one or more processors of the robotic medical system 200). In some embodiments, when the robotic arm 210 is in the docking position, the robotic arm 210 can perform a null space motion to maintain the position and / or orientation of the cannula, as will be discussed in more detail below. Conversely, when the cannula is not detected by the cannula sensor 310, the robotic arm 210 is placed in an undocking state (e.g., an undocking position).

[0159] In some embodiments, as illustrated in FIG. 24A, the robotic arm 210 includes an input or button 312 (e.g., a donut-shaped button, or other types of controls, etc.) that can be used to put the robotic arm 210 into admittance mode (e.g., by pressing button 312). The admittance mode is also referred to as an admittance scheme or admittance control. In the admittance mode, the robotic system 210 measures the force and / or torque (e.g., applied to the robotic arm 210) and outputs the corresponding velocity and / or position. In some embodiments, the robotic arm 210 can be manually operated by a user in the admittance mode (e.g., during a setup procedure, or during a procedure, etc.). In some cases, by using admittance control, the operator does not need to overcome all of the inertia in the robotic medical system 200 to move the robotic arm 210. For example, under admittance control, when the operator applies a force to the arm, the robotic medical system 200 measures the force and can assist the operator in moving the robotic arm 210 by driving one or more motors associated with the robotic arm 210, thereby resulting in the desired velocity and / or position of the robotic arm 210.

[0160] In some embodiments, the link 302 can be detachably coupled to the medical tool 212 (e.g., to facilitate attachment and detachment of the medical tool 212 to and from the robotic arm 210). The joint 304 provides the robotic arm 210 with multiple degrees of freedom (DoF) that facilitate control of the medical tool 212 via the ADM 308. In one embodiment, as shown in FIG. 22, including multiple robotic arms, each robotic arm holds its own respective medical tool and can pivot the medical tool around a center of remote motion.

[0161] FIG. 24B is a front view of the robotic arm 210. FIG. 24C is a perspective view of the robotic arm 210. In some embodiments, the robotic arm 210 includes a second input or button 314 (e.g., a push button), different from the button 312 of FIG. 24A, for putting the robotic arm 210 into impedance mode (e.g., by pressing the button 314 once or continuously pressing it). In this example, the button 314 is located between joints 304-5 and 304-6. The impedance mode is also referred to as an impedance scheme or impedance control. In impedance mode, the robotic medical system 200 measures displacements (e.g., changes in position and velocity) and outputs forces and / or torques to facilitate manual movement of the robotic arm. In some embodiments, the robotic arm 210 can be manually operated by a user in impedance mode (e.g., during a setup procedure). In some embodiments, under impedance mode, movement of some operators of the robotic arm 210 can cause movement of one or more joints and / or links of the entire robotic arm 210.

[0162] In some embodiments, for admittance control, a force sensor or load cell measures the force the operator is applying to the robotic arm 210 and can move the robotic arm 210 to feel light. Under admittance control, the admittance control can feel lighter than impedance control because the motors in the controller can help accelerate the mass, hiding the perceived inertia of the robotic arm 210. In contrast, in impedance control, the user bears most, if not all, of the mass acceleration according to some embodiments.

[0163] In some situations, it may be inconvenient for the operator to reach for buttons 312 and / or 314 to activate a manual operation mode (e.g., admittance mode and / or impedance mode) depending on the position of the robotic arm 210 relative to the operator. Thus, in these situations, it may be convenient for the operator to trigger the manual operation mode by means other than buttons.

[0164] In some embodiments, the robotic arm 210 includes a single button (e.g., button 312 or 314) that can be used (e.g., by using different presses such as a long press, a short press, a continuous press, etc.) to put the robotic arm 210 into an admittance mode and / or an impedance mode. In some embodiments, the robotic arm 210 can be put into an impedance mode by the user pressing on an arm link mechanism (e.g., link 302) and / or a joint (e.g., joint 304) and overcoming a force threshold. In some embodiments, both the admittance mode and the impedance mode are common in that they enable the user to command movement by gripping the robotic arm 210 and interfacing directly with it.

[0165] In some embodiments, the robotic arm 210 includes an input control for activating an arm following mode. For example, in some embodiments, the robotic arm 210 can include a designated touch point located on a link 302 or joint 304 of the robotic arm (e.g., the outer shell of link 302 or button 316). User interaction with the designated touch point (e.g., user touch, contact, etc.) activates the arm following mode. In some embodiments, the robotic arm 210 includes a plurality of touch points. User interaction with any one of the touch points (e.g., one or two or more) activates the arm following mode.

[0166] During a medical procedure, it may be desirable to maintain the remote center of motion (RCM) of the ADM308 of the robotic arm 210 and / or the tool 212 coupled thereto in a static pose (e.g., position and / or orientation). The RCM may refer to a point within a space where the movement of a cannula or other access port into which the medical tool 212 is inserted is restricted. In some embodiments, the medical tool 212 includes an end effector that is inserted through an incision or natural orifice of a patient while maintaining the RCM. In some embodiments, the medical tool 212 includes an end effector that is in a retracted state during a setup process of the robotic medical system.

[0167] In some situations, the robotic medical system 200 can be configured to move one or more links 302 of the robotic arm 210 within a “null space” while the ADM308 and / or the RCM of the robotic arm 210 are maintained in their respective poses (e.g., position and / or orientation) to avoid collisions with nearby objects (e.g., other robotic arms). The null space can be considered a set of joint states in which the robotic arm 210 can move without causing movement of the ADM308 and / or the RCM, thereby maintaining the position and / or orientation of the medical tool 212 (e.g., within a patient). In some embodiments, the robotic arm 210 can have multiple positions and / or configurations available for each pose of the ADM308.

[0168] In order for the robotic arm 210 to move the instrument to a desired orientation within the space, in certain embodiments, the robotic arm 210 can have at least six DoFs, i.e., three DoFs for translation (e.g., X position, Y position, and Z position) and three DoFs for rotation (e.g., yaw, pitch, and roll). In some embodiments, each joint 304 can provide a single DoF to the robotic arm 210, and thus the robotic arm 210 can have at least six joints to achieve the degrees of freedom of movement for positioning the ADM 308 in any orientation within the space. To further maintain the ADM 308 and / or the remote center or movement of the robotic arm 210 in a desired orientation, the robotic arm 210 can further have at least one additional "redundant joint". Thus, in certain embodiments, the system may include a robotic arm 210 having at least seven joints 304, providing at least seven DoFs to the robotic arm 210. In some embodiments, the robotic arm 210 can include a subset of joints 304 each having two or more degrees of freedom, thereby achieving additional DoFs for null space motion. However, depending on the embodiment, the robotic arm 210 can have a greater or lesser number of DoFs.

[0169] Further, as described with respect to FIG. 12, the bar 220 (e.g., an adjustable arm support) can provide several degrees of freedom including elevation, lateral translation, tilt, etc. Thus, depending on the embodiment, the robotic medical system can have an even greater number of robotically controlled degrees of freedom beyond the degrees of freedom in the robotic arm 210 to provide null space movement and collision avoidance. In each of these embodiments, the end effector of one or more robotic arms (and any tools or instruments coupled thereto), and the remote center along the axis of the tool, can advantageously maintain an orientation and / or position within the patient.

[0170] The robotic arm 210 having at least one redundant DoF has at least one more DoF than the minimum number of DoFs for performing a given task. For example, the robotic arm 210 can have at least seven DoFs, and in some embodiments, one of the joints 304 of the robotic arm 210 can be considered a redundant joint. One or more redundant joints can enable the robotic arm 210 to move within the null space to maintain the posture of the ADM 308 and the position of the RCM and to avoid collisions (if any) with other robotic arms or objects.

[0171] In some embodiments, the robotic medical system 200 can be configured to perform collision avoidance, for example, by utilizing the movement of one or more redundant joints in the null space to avoid collisions (if any) between adjacent robotic arms 210. For example, when a robotic arm 210 collides with or approaches another robotic arm 210 (e.g., within its defined distance), one or more processors of the robotic medical system 200 can be configured to detect the collision or impending collision (e.g., via kinematics). Thus, the robotic medical system 200 can control one or both of the robotic arms 210 to adjust their respective joints within the null space to avoid the collision or impending collision. In one embodiment including at least a pair of robotic arms, the base and the end effector of one of the robotic arms can remain in their postures, while the links or joints between them move within the null space to avoid collisions with adjacent robotic arms.

[0172] C. Exemplary surgical setting. FIG. 25 is a perspective view of a robotic medical system 200 including four robotic arms 210-1, 210-2, 210-3, and 210-4 according to some embodiments. Each of the robotic arms 210-1, 210-2, 210-3, and 210-4 is coupled to a respective surgical tool 602 (e.g., 602-1 to 602-4 that may correspond to instrument 212) via a respective ADM308 (e.g., tool driver) such as ADM308-1 to 308-4. The surgical tool 602 may be inserted into the patient through respective ports 608 located on the patient. As used herein, a port (e.g., location of the port, entry port, entry point, port region, port area, or port position, etc.) refers to a location on the patient's body through which a medical tool / instrument (e.g., held by a robotic arm) may be inserted and movement may be restricted. In some embodiments, the port corresponds to an incision point (or incision region) made through the patient's skin to facilitate a medical operation or medical procedure. In some embodiments, the port corresponds to a natural opening such as the patient's mouth (e.g., for a bronchoscopy procedure). In some embodiments, the port corresponds to a medical device with an opening that is disposed at the incision point or natural opening to enable access to the surgical space through the opening. The patient figure is excluded from FIG. 25 to enhance the visibility of the robotic arms 210 and the surgical tool 602.

[0173] In FIG. 25, the robotic arm 210-2 is coupled to the camera 606. In this example, the camera 606 is coupled to the robotic arm 210 via (e.g., at the distal end of) the medical instrument 602-2 (e.g., an endoscope). In some embodiments, the camera 606 is part of the medical instrument 602-2. In some embodiments, the camera 606 can be a stand-alone device (e.g., not part of a surgical instrument) coupled to the robotic arm (e.g., the camera 606 is distinct and separate from the medical instrument). In some embodiments, the camera 606 defines an axis 604 (e.g., an optical axis) that specifies the orientation of the camera 606. The camera 606 (or scope) provides an image of the surgical site to facilitate control of the surgical tool for performing robotic medical procedures. For example, a robotically controllable endoscope of a robotic system can include a camera positioned at its distal tip. The user can view the image from the endoscope camera in a viewer to facilitate control of the endoscope and / or other components of the robotic medical system. As another example, the robotic system may include one or more cameras that are laparoscopically or endoscopically inserted into the patient. The user can view the image from the inserted camera to facilitate control of one or more additional robotically controlled medical instruments, such as one or more additional medical instruments inserted laparoscopically.

[0174] In some embodiments, the robotic medical system 200 includes a coordinate system (e.g., a robotic coordinate system, a coordinate frame, a system frame, etc., which may be a Cartesian coordinate system or a non-Cartesian coordinate system), and the respective positions of the patient support platform 202, the robotic arm 210, the adjustable arm support 220, and / or the instrument 212 are represented as coordinates (e.g., x-coordinates, y-coordinates, and z-coordinates) on the coordinate system. For example, the robotic medical system 200 (e.g., one or more processors 380 of the robotic medical system 200) may be configured to identify the position and orientation of the patient support platform 202, the robotic arm 210, the adjustable arm support 220, and / or the instrument 212 based on the coordinates in the coordinate system.

[0175] Figures 26A and 26B illustrate the camera field of view in a three - dimensional space according to some embodiments. In some embodiments, the robotic medical system 200 determines the field of view based on the focal length of the camera 606 (e.g., the field of view may correspond to the space within a certain distance from the center of the field of view of the camera 606). In some embodiments, the robotic medical system 200 uses data obtained by one or more sensors 388 of the robotic medical system 200, such as position (and orientation) data corresponding to the tip of an endoscope (e.g., a medical instrument) to which the camera 606 is attached, or position data corresponding to the camera 606, to determine the field of view (e.g., the safety region may be determined to be located at a specific distance from the camera 606). In some embodiments, the robotic medical system 200 stores information specifying the field of view of the camera 606 (e.g., in memory) and uses the stored information to determine the field of view of the camera 606 based on the position and orientation of the camera 606. For example, the robotic medical system 200 may store information specifying a predefined height (e.g., several centimeters) of a three - dimensional volume corresponding to the field of view, having the shape of a truncated square pyramid, as well as predefined base angles, tilt angles, and / or edge angles. In another example, the robotic medical system 200 may store information specifying a predefined radius or diameter and a predefined distance (e.g., several centimeters) for a spherical volume corresponding to the field of view, located at a predefined distance from the camera 606 and having the predefined radius or diameter. In some embodiments, the robotic medical system 200 determines the field of view according to the image(s) acquired using the camera 606.

[0176] In FIGS. 26A and 26B, a camera field of view 702 (such as as seen by a camera coupled to another robotic arm) is represented by a three-dimensional space (such as a pyramid). Those skilled in the art will understand that the three-dimensional space of the camera field of view can be represented by other shapes such as a cone, a cylinder, a sphere, a tetrahedron, etc. In the embodiment of FIG. 26A, the robotic arm 210-8 is coupled to a surgical tool 704 (such as a medical instrument) via an ADM308-8. The surgical tool 704 intersects (such as overlaps, is within, etc.) the three-dimensional space of the camera field of view 702. In contrast, FIG. 26B shows a robotic arm 210-9 that holds (such as via an ADM308-9) a surgical tool 708 that does not intersect the three-dimensional space of the camera field of view 702 (such as due to the position and / or orientation of the tool 708).

[0177] D. Flexible Scope FIGS. 27A - 27D are different views of a flexible scope 720 (such as a flexible laparoscope, a flexible endoscope, etc.) according to some embodiments. In some embodiments, the flexible scope is coupled to a robotic arm 210 of the robotic medical system 200.

[0178] FIG. 27A illustrates that the flexible scope 720 includes a distal end 724 (such as a distal portion) and an intermediate portion 722 (such as a portion positioned from the distal end 724 towards the proximal end 740). In some embodiments, the flexible scope 720 is coupled to a camera at the distal end 724. In some embodiments, the flexible scope 720 includes a plurality of links 726 (such as a linkage mechanism). In some embodiments, the plurality of links 726 are connected by one or more joints 728. For example, in FIG. 27A, the flexible scope 720 includes links 726-1 to 726-3 that are connected by joints 728-1 and 728-2. In some embodiments, the plurality of links 726 include four or more links.

[0179] In some embodiments, the flexible scope 720 includes a rigid (e.g., non-flexible) portion and a flexible portion. The rigid portion has a fixed (e.g., non-flexible, non-variable) shape. The flexible portion has a shape that can be manipulated (e.g., varied, changed, adjusted, conformed, etc.).

[0180] In some embodiments, the rigid portion is located at or toward the proximal end of the flexible scope (e.g., at the intermediate portion 722), and the flexible portion is located at the distal end of the flexible scope (e.g., the distal end 724). In some embodiments, the rigid portion is located at the distal end of the flexible scope (e.g., the distal end 724), and the flexible portion is located at or toward the proximal end of the flexible scope (e.g., at the intermediate portion 722). In some embodiments, the flexible scope 720 includes two or more rigid portions (e.g., links 726-1 and 726-3) coupled by a flexible portion (e.g., link 726-2). In some embodiments, the rigid portion and the flexible portion are connected by a link (e.g., link 726-2) and / or a joint (e.g., joint 728-1 or 728-2), and the flexible portion can move (e.g., tilt, rotate, displace, etc.) relative to the rigid portion.

[0181] In some embodiments, the flexible scope 720 introduces additional degrees of freedom compared to a rigid scope. For example, in addition to the insertion, extraction, pan, and rotation operations available on a rigid scope, at least a portion (e.g., the distal portion or the intermediate portion) of the flexible scope 720 can articulate (e.g., tilt, rotate, pivot, bend, perform a rotational movement, etc.) relative to another portion of the flexible scope 720, thereby enabling the user to visualize a target anatomical structure from a perspective that is not achievable with a rigid scope. In some embodiments, the flexible scope 720 has a smaller volume than a rigid scope and thus frees up space within the surgeon's workspace.

[0182] In some embodiments, the robotic system 200 is configured to operate the flexible scope 720 in a plurality of modes (e.g., drive modes), such as a joint motion mode (e.g., joint motion drive mode), a trajectory mode (e.g., trajectory drive mode), and an automatic insertion and retraction mode (e.g., automatic insertion and retraction drive mode), as further described below (e.g., via the processor 380). For example, in some embodiments, the robotic system 200 operates the flexible scope 720 in the joint motion mode at a first time, in the trajectory mode at a second time different from the first time, and in the automatic insertion and retraction mode at a third time different from the first and second times. In some embodiments, the plurality of modes includes two of the aforementioned modes. In some embodiments, the plurality of modes includes four or more modes.

[0183] FIG. 27B illustrates the operation of the flexible scope 720 in the joint motion drive mode according to some embodiments. In some embodiments, in the joint motion drive mode, the distal portion 724 of the flexible scope 720 can be moved (730) (e.g., by the processor 380) upward, downward, leftward, or rightward (e.g., joint motion, perform joint motion movement, tilt, rotate, pivot, perform rotational movement, etc.). For example, the processor 380 may provide an electrical signal (e.g., to one or more actuators such as a motor or to an actuation controller) to cause movement of the distal portion 724 of the flexible scope 720. FIG. 27B illustrates three postures or states of the flexible scope 720 when the distal portion 724 is moved or rotated (730) from an initial position or orientation (“a”) (e.g., aligned with the intermediate portion 722) (e.g., facing north) to a second position or second orientation (“b”) (e.g., facing northwest), or a third position or third orientation (“c”) (e.g., facing northeast). In some embodiments, as illustrated in FIG. 27B, the intermediate portion 722 of the flexible scope 720 remains stationary in the joint motion drive mode.

[0184] FIG. 27C illustrates the operation of the flexible scope 720 in the orbital drive mode according to some embodiments. In some embodiments, in the orbital drive mode, the flexible scope 720 (e.g., at the distal portion 724) can be moved (732) (e.g., traced) around a target point 734 (e.g., an orbital point) in space (e.g., by the processor 380). For example, while the flexible scope 720 is operating in the orbital drive mode, when the flexible scope is moved (e.g., laterally), the distal end of the flexible scope 720 rotates or tilts to face towards the target point 734. In FIG. 27C, three exemplary positions or postures of the flexible scope 720 as the distal end 724 of the flexible scope 720 rotates around the target point 734 are shown at (i), (ii), and (iii).

[0185] In some embodiments, the target point 734 is a fixed point in space. In some embodiments, the target point 734 is predefined (e.g., selected) by the user. In some embodiments, the target point 734 is predefined by the robotic system. In some embodiments, the target point 734 is automatically selected by the robotic medical system 200 (e.g., independently of user input). For example, the target point 734 may be selected as the location of a particular internal organ or anatomical structure of the patient.

[0186] FIG. 27D illustrates the operation of the flexible scope 720 in the automatic insertion and retraction drive modes, according to some embodiments. In some embodiments, in the automatic insertion and retraction modes, the flexible scope 720 can be automatically inserted (e.g., moved in direction 736) from its current position to the surgical site (e.g., without applying manual force to the flexible scope 720), (e.g., by processor 380). In some embodiments, as illustrated in FIG. 27D, the distal portion 724 of the flexible scope flexes (e.g., along the sheath) as the flexible scope is inserted. In the automatic insertion and retraction modes, the flexible scope 720 can also be automatically retracted (e.g., moved in direction 738) from the surgical site to the port or cannula of the robotic arm to which the flexible scope 720 is coupled (e.g., without applying manual force to the flexible scope 720), (e.g., by processor 380). In some embodiments, the distal portion 724 of the flexible scope 720 straightens as it is retracted.

[0187] A flexible scope is beneficial as it enables a user (e.g., a surgeon) to obtain different views of target anatomical structures that may not be possible with a rigid scope. However, operating (e.g., driving) a flexible scope in various drive modes can be difficult. For example, in an articulation drive mode, it can be difficult for a user to identify the current shape or configuration of the flexible scope, or the degree or direction of articulation, or the direction in which the scope should be moved to return from a bent state to a straight state. In an orbital drive mode, it can be difficult for a user to determine the distance between the flexible scope and a target point around which the flexible scope is orbiting, or the position of the flexible scope within the orbit, or the shape of the flexible scope. In an automatic insertion and retraction drive mode, it can be difficult for a user to understand the current state or position (along the insertion path) of the flexible scope when it is being inserted or retracted, or whether the flexible scope is in a bent state or has been straightened. Accordingly, there is a need for a robotic medical system that can facilitate a user's visualization of how a flexible scope is positioned or oriented.

[0188] E. Exemplary user interface. According to some embodiments, a robotic system may include a user interface configured to display a dynamic visual representation of different drive modes of the flexible scope described above. In some embodiments, the user interface is configured to display a dynamic visual representation of the flexible scope in accordance with a determination that a robotic arm is coupled to the flexible scope (e.g., regardless of the drive mode).

[0189] Figures 28A - 28D illustrate an exemplary user interface that may be displayed on a display device when a flexible scope is operating in an articulation drive mode, according to some embodiments.

[0190] FIG. 28A shows a user interface 802 displayed on a display device 800 according to some embodiments. In some embodiments, as shown in FIG. 28A, the user interface 802 includes an image or representation of the field of view of a camera (e.g., a view of the surgical site such as the treatment site) or a scope (e.g., the flexible scope 720). FIG. 28A also shows that the user interface 802 includes images or representations of medical instruments 500 and 550 located within the field of view of the camera or scope.

[0191] In some embodiments, as illustrated in FIG. 28A, the user interface 802 includes indicators 804 (e.g., visual indicators) that identify medical instruments (e.g., "Tool A", "Tool B", "Tool C", "Tool D") at the surgical site. In some embodiments, the indicator 804 includes a number associated with the robotic arm coupled to each medical instrument. For example, the indicator 804-2 includes the number "2" indicating that Tool B is coupled to the second robotic arm of the robotic medical system.

[0192] In some embodiments, as illustrated in FIG. 28A, the user interface 802 includes indicators 806 (e.g., visual indicators, scope tabs, etc.) for indicating the position and / or orientation of a flexible scope (e.g., the flexible scope 720) or a portion thereof (e.g., the distal end 724 of the flexible scope 720) relative to a reference position. The reference position can be a neutral position (e.g., a position corresponding to a scope that is completely straight, a position determined based on the insertion port, etc.). In some embodiments, the indicator 806 has an initial size similar to that of the indicator 804 and then expands (e.g., to the size illustrated in FIG. 28A) when the camera coupled to the flexible scope is active or becomes active (e.g., according to a determination by the robotic medical system).

[0193] The inset of FIG. 28A shows that the indicator 806 includes graphical elements such as a grid 809 including a reference axis 807-1 (e.g., a horizontal axis), a reference axis 807-2 (e.g., a vertical axis), and a marker 808 (e.g., a dot, a point, etc.). The marker 808 indicates the direction and / or degree of articulation movement (e.g., movement, flexion, pivoting, etc.) of the flexible scope. In the example of FIG. 28A, the indicator 806 indicates that the distal end of the flexible scope is articulating (e.g., moving, pivoting, flexing, etc.) upward and leftward relative to its reference position by showing a marker 808 positioned above and to the left of the origin (e.g., the intersection of the reference axes 807-1 and 807-2) formed by the reference axes 807-1 and 807-2.

[0194] In some embodiments, the indicator 806 includes a number associated with a robotic arm coupled to the flexible scope. For example, FIG. 28A shows that the indicator 806 includes the number "3" indicating that the flexible scope is coupled to the third robotic arm of the robotic medical system.

[0195] In some embodiments, as illustrated in FIG. 28A, the user interface 802 includes an indicator 810 (e.g., along the periphery of the user interface 802 or a portion thereof) for indicating the position (or orientation) of the medical instrument with respect to the field of view. For example, in FIG. 28A, the indicator 810-1 is located along the left edge of the user interface 802 and indicates that the medical instrument 500 is positioned on the left side of the field of view. In some embodiments, as shown in FIG. 28A, the indicator 810-1 includes information identifying the medical instrument 500 (e.g., the number "1" indicating the first robotic arm, etc., a number associated with the robotic arm coupled to the medical instrument 550).

[0196] In some embodiments, the indicator 810 (or any other indicator for any medical tool) is displayed when the corresponding medical tool is located outside the field of view. In some embodiments, the indicator 810 (or any other indicator for any medical tool) is displayed regardless of whether the corresponding medical tool is located within or outside the field of view.

[0197] FIG. 28B shows that in some embodiments, the indicator 806 includes a representation 812 (e.g., a real-time representation or a three-dimensional representation or rendering) corresponding to the shape of the flexible scope to further assist the user in understanding the shape of the scope. In some embodiments, the representation 812 includes a rendering of the link 814. In some embodiments, the number of links 814 (e.g., 814-1 to 814-3) within the representation 812 corresponds to the number of links (e.g., links 726-1 to 726-3) of the flexible scope.

[0198] The transition between FIGS. 28B and 28C shows a change in the articulation movement of the distal end of the flexible scope from the upward and leftward positions in FIG. 28B to the upward and rightward positions in FIG. 28C. In some embodiments, in response to detecting a change in the articulation movement of the flexible scope, the user interface 802 updates one or more graphical elements of the indicator 806 (e.g., via the processor 380) to reflect the changed position and / or shape of the scope. For example, in FIG. 28C, the user interface 802 displays an updated graphical element (e.g., a marker) 811 positioned above and to the right of the origin formed by the reference axes 807-1 and 807-2 to indicate that the distal end of the scope has moved (articulated) to the upward and rightward positions. FIG. 28C also shows a user interface 802 display having an updated representation 813 that reflects the current shape of the flexible scope. As illustrated in FIGS. 28B and 28C, the indicator 806 may operate or function as a dynamic indicator that changes in real time as the scope moves.

[0199] In some embodiments, the user interface 802 displays a representation of the field of view of the flexible scope camera that is updated in response to movement of the flexible scope. For example, in FIG. 28C, as the flexible scope articulates upward and to the right relative to its reference position, the user interface 802 displays an updated representation 820 of the field of view of the camera that is translated to the left (relative to the rightward movement of the distal end of the flexible scope) compared to the representation 818 of the field of view of the camera in FIG. 28B.

[0200] FIG. 28D illustrates a user interface 802 that includes one or more indicators 822 (e.g., bars, articulation bars, etc.) according to some embodiments. In this example, the one or more indicators 822 include a horizontal bar 822-1 and a vertical bar 822-2. In some embodiments, the horizontal bar 822-1 includes a graphical element (e.g., an arrow) 824-1 that represents the leftward articulation direction relative to a reference position 826-1 (e.g., the center of the horizontal bar), and a graphical element (e.g., an arrow) 824-2 that represents the rightward articulation direction relative to the reference position 826-1. In some embodiments, the vertical bar 822-2 includes a graphical element (e.g., an arrow) 824-3 that represents the upward articulation direction relative to a reference position 826-2 (e.g., the center of the vertical bar), and a graphical element (e.g., an arrow) 824-4 that represents the downward articulation direction relative to the reference position 826-2.

[0201] In some embodiments, the indicator(s) 822 includes graphical element 828-1 that represents the direction of movement of the distal end of the flexible scope along the horizontal direction. In some embodiments, the indicator 822 includes graphical element 828-2 that represents the direction of movement of the distal end of the flexible scope along the vertical direction. In the example of FIG. 28D, the distal end of the flexible scope is in the right and upward directions relative to its reference position. Thus, in the user interface 802 of FIG. 28D, the graphical element 828-1 is positioned to the right of the reference position 826-1, and the graphical element 828-2 is positioned above the reference position 826-2. In some embodiments, the distal end of the flexible scope moves in the right and upward directions (e.g., articulates, bends, curves, etc.) relative to its reference position, and the user interface 802 indicates the direction of movement using the graphical element 828-1 and the graphical element 828-2.

[0202] In some embodiments, each of the graphical elements 828-1 and 828-2 has (or is characterized by) a respective length that represents the degree of articulation of the flexible scope. In some embodiments, the lengths of the graphical elements 828-1 and 828-2 are dynamically (e.g., in real time) updated as the scope is moved. For example, the graphical element 828-1 has a length 830-1 that increases (e.g., in real time) as the scope articulates in the right direction away from the reference position and decreases as the scope articulates in the left direction toward the reference position. Element 828-2 has a length 830-2 that increases (e.g., in real time) as the scope articulates in the upward direction away from the reference position and decreases as the scope articulates in the downward direction toward the reference position.

[0203] FIGS. 29A-29D illustrate exemplary user interfaces that are displayed on a display device when a flexible scope is operating in an orbital drive mode, according to some embodiments.

[0204] Figures 29A and 29B show a user interface 802 that includes an indicator 832. In some embodiments, the indicator 832 includes a graphical element 834 (e.g., a point, sphere, cross, "X" symbol, white circle, black circle, etc.) that represents an orbital point (e.g., a target point such as target point 734) along which the flexible scope is tracing an orbit.

[0205] During the surgery, the surgeon can drive an input device on the surgeon console 240 to control the insertion depth of the flexible scope into the surgical site. In some embodiments, the graphical element 834 has (e.g., is characterized by) a dimension (e.g., length, width, height, diameter, etc.) that can be updated (e.g., changed) according to a change in the insertion depth of the flexible scope. For example, the transition from Figure 29A to Figure 29B indicates that the flexible scope is moving closer to the surgical site. In some embodiments, as shown by the transition from Figure 29A to Figure 29B, the diameter of the graphical element 834 displayed on the user interface 802 increases (e.g., expands) as the flexible scope approaches the surgical site (or moves toward a target organ such as an organ positioned at the center of the camera's field of view). In some embodiments, the smaller the diameter of the graphical element 834 in Figure 29A, the greater the distance between the scope and the surgical site, and the larger the diameter of the graphical element 834 in Figure 29B, the smaller the distance between the scope and the surgical site.

[0206] In some embodiments, the user interface 802 displays a representation of the camera's field of view that changes according to the insertion depth of the flexible scope 720 into the surgical site. For example, in Figure 29B, the user interface 802 displays a representation 842 of the camera's field of view that has a higher magnification compared to the magnification of the representation 840 of the field of view illustrated in Figure 29A.

[0207] In some embodiments, as illustrated in FIGS. 29A and 29B, the indicator 832 includes a graphical element 836 (e.g., a line, a dashed line, etc.) representing a line of sight from an orbital point (or a target point) to a port corresponding to the flexible scope (e.g., the port into which the flexible scope is inserted). In some embodiments, the indicator 832 includes a graphical element 838 for indicating a direction to the scope port. In some embodiments, when the scope port, the camera, and the orbital point are aligned (e.g., the port is behind the camera and the orbital point is in front of the camera along a common line of sight), the graphical element 836 is omitted. For example, the graphical element 836 may converge or merge with the graphical element 834 (e.g., together with the graphical element 838), such that the graphical element 836 is no longer displayed within the user interface 802 when the scope port, the camera, and the orbital point are aligned.

[0208] FIG. 29C illustrates a user interface 802 with an orbital sphere according to some embodiments. In this example, the user interface 802 displays an indicator 844 that includes a graphical element 846 (e.g., a sphere, a 3D sphere, a spherical body, etc.) representing an orbital sphere indicating the range of orbital motion (e.g., the full range) of the flexible scope. In some embodiments, the indicator 844 includes a graphical element 848 (e.g., a first circle, a white circle, etc.) representing the orbital point around which the flexible scope is orbiting. In some embodiments, the indicator 844 includes a graphical element 850 (e.g., a marker, a second circle, a black circle, etc.) representing the position of the flexible scope within the orbital sphere (e.g., in real time). In the example of FIG. 29C, the distal end of the scope is in an upper and left position relative to the orbital point. In some embodiments, when the scope is aligned with the orbital point (e.g., as illustrated in FIG. 27C as configuration (ii)), the graphical element 850 is no longer displayed. For example, the graphical element 850 may converge or merge with the graphical element 836 such that the graphical element 848 is no longer displayed within the user interface 802 when the scope port, the camera, and the orbital point are aligned.

[0209] In some embodiments, the graphical element 846 has a size (e.g., diameter, length, dimension, etc.) that correlates with the distance between the scope and the target point. In some embodiments, the size of the graphical element 846 is dynamic and changes according to (e.g., in real time) the distance between the scope and the target point. For example, the size (e.g., diameter) of the graphical element 846 increases as the scope is inserted and moved towards the surgical site (e.g., when the distance between the scope and the target point decreases), and decreases as the scope is retracted from the surgical site (e.g., when the distance between the scope and the target point increases). In some embodiments, in accordance with the change in the size of the graphical element 846, the user interface 802 also displays the graphical elements 848 and 850 at updated positions on the graphical element 846. The overlay of the graphical elements 846, 848, and 850 across the surgical field provides the surgeon with insights regarding the position and configuration of the scope, and how the scope can be straightened (e.g., by adjusting the scope from the position corresponding to representation 850 to the position corresponding to representation 848).

[0210] FIG. 29D illustrates a user interface 802 with a 3D axis indicator 854, according to some embodiments. The indicator 854 includes graphical elements 856 that represent axes (e.g., x-axis 856-1, y-axis 856-2, and z-axis 856-3). The orientation or rotation of the axes 856 represents the orientation or position of the flexible scope relative to the port. In some embodiments, the user interface displays the indicator 854 with the graphical elements 856 in accordance with a determination that the scope and the trajectory point are not aligned (e.g., as illustrated in FIGS. 27C as configurations or postures (i) and (iii)). In some embodiments, when the scope is aligned with the trajectory point (e.g., when the scope is a straight scope), the graphical element 856-3 is either not displayed or becomes a point.

[0211] Figures 30A-30C illustrate exemplary user interfaces displayed on a display device when a flexible scope operates in an automatic insertion and retraction mode, according to some embodiments.

[0212] Figure 30A illustrates the display of an indicator 860 in a user interface 802. In some embodiments, the indicator 860 is coupled to a robotic arm and includes a graphical element 861 corresponding to the shape (e.g., contour) of the flexible scope used in the present procedure. In this example, the flexible scope is configured as a 90° articulating flexible scope. The indicator 860 also includes a graphical element 862 (e.g., shading, rendering, etc.) that is overlaid on the graphical element 861 and represents the real-time position or state of the scope. In the example of Figure 30A, the scope is fully inserted, and thus the graphical element 862 completely fills (e.g., completely overlays) the graphical element 861.

[0213] Figure 30B illustrates the display of the indicator 860 within the user interface 802 when the scope is partially retracted. In Figure 30B, the graphical element 862 partially fills the graphical element 861. The shaded portion of the scope (e.g., collectively represented by the graphical elements 861 and 862) represents the current shape and position of the scope in real time and is dynamically updated (e.g., as the scope continues to retract or is re-inserted).

[0214] In some embodiments, the user interface 802 displays a text element 864, such as "Insert" or "Retract," to indicate whether the scope is being inserted or retracted.

[0215] FIG. 30C illustrates a progress bar according to some embodiments. In this example, the user interface 802 displays an indicator 866 (e.g., a bar) that includes a graphical element 868 (e.g., a bar, a shaded portion, a shadow, a rendering, etc.) that indicates the degree of insertion and / or retraction of the scope. In some embodiments, the indicator 866 includes a graphical element 870 (e.g., a text element) corresponding to the fully inserted (e.g., maximum insertion) position of the scope. In some embodiments, the indicator 866 includes a graphical element 872 (e.g., an icon, a text element, etc.) that indicates whether the flexible scope is articulating. For example, in some embodiments, the graphical element 872 is visually emphasized (e.g., lit up, highlighted, etc.) when or while the flexible scope is articulating. In the example of FIG. 30C, the flexible scope is partially inserted and in an articulated state. In this example, the graphical element 868 indicates the degree of insertion, and the visual emphasis of the graphical element 872 indicates that the scope is articulating.

[0216] FIGS. 30D-30F show examples of the indicator 866 indicating different states of the scope.

[0217] FIG. 30D illustrates an indicator 866 corresponding to the fully retracted linear state of a flexible scope according to some embodiments. In some embodiments, the indicator 866 includes a graphical element 874 (e.g., a text element) corresponding to the fully retracted position of the scope. In some embodiments, the indicator 866 includes a graphical element 870 (e.g., a text element) corresponding to the fully inserted position of the scope as described with respect to FIG. 30C. In FIG. 30D, the indicator 866 also includes the graphical element 872. In some embodiments, the graphical element 872 is visually de-emphasized as shown in FIG. 30D, or omitted (or not displayed), indicating that the scope is not articulating.

[0218] Figure 30E illustrates an indicator 866 corresponding to a partially inserted straight (e.g., non-articulated) state of a flexible scope according to some embodiments. In Figure 30E, the graphical element 868 indicates the degree of insertion (e.g., partial insertion). Similar to the indicator 866 shown in Figure 30D, the graphical element 872 may be visually de-emphasized as shown in Figure 30E, or alternatively, omitted (or not displayed) to indicate that the scope is not articulating.

[0219] Figure 30F illustrates an indicator 866 corresponding to a fully inserted and articulated (e.g., partially articulated, fully articulated, etc.) state of a flexible scope according to some embodiments. In Figure 30F, the graphical element 870 indicates the degree of insertion of the scope. For example, as shown in Figure 30F, the graphical element 870 is visually emphasized to indicate that the scope is fully inserted. In addition, the graphical element 872 indicates whether the scope is articulating. For example, as shown in Figure 30F, the graphical element 872 is visually emphasized to indicate that the scope is articulating.

[0220] Figure 31A illustrates different configurations of a flexible scope, such as a scope articulated at 0°, 45°, and 90°, according to some embodiments. In some embodiments, the shape or configuration of the scope (e.g., 0° scope, 45° scope, or 90° scope) at the time when retraction or insertion is initiated is displayed within the user interface 802 (e.g., as shown in Figures 30A - 30B) while the progress of the insertion or retraction is indicated by the graphical element 862.

[0221] Figures 31B - 31F illustrate exemplary user interface elements displayed on a display device to indicate the status of a flexible scope according to some embodiments.

[0222] FIG. 31B illustrates indicator 880 according to some embodiments. In some embodiments, indicator 880 is shown in a graphical projection (e.g., a non-primary graphical projection). In some embodiments, indicator 880 is shown in a perspective view, an axonometric view, such as shown in FIG. 31B, a dimetric view, or a trimetric view, an oblique view, or a secondary projection view. In some embodiments, indicator 880 includes or is displayed with one or more reference planes (e.g., x-z plane 882 and y-z plane 884) or graphical representations thereof. In some embodiments, one or more reference planes are translucent (e.g., semi-transparent). This facilitates the simultaneous display of the scope position with respect to one or more reference planes. FIG. 31B shows indicator 880 in different states (e.g., 880-1 to 880-4). For example, indicator 880 in state 880-1 represents that the scope bends upward (e.g., in the direction of the y-axis), indicator 880 in state 880-2 represents that the scope bends to the right (e.g., in the direction of the x-axis), indicator 880 in state 880-3 represents that the scope bends downward (e.g., opposite to the direction of the y-axis), and indicator 880 in state 880-4 represents that the scope bends to the left (e.g., opposite to the direction of the x-axis). In some embodiments, indicator 880 is placed in different states representing the bending of the scope in a direction that is non-parallel and non-perpendicular to the reference plane.

[0223] Figure 31C illustrates an indicator 886 according to some embodiments. The indicator 886 shown in Figure 31C is similar to the indicator 880 shown in Figure 31B. However, in Figure 31C, the indicator 886 is shown together with a reference user interface element 888. In some embodiments, the reference user interface element 888 is illustrated as a two-dimensional-like representation. In some embodiments, the reference user interface element 888 has a round shape (e.g., a circle, an ellipse, an oval, etc.). In some embodiments, the reference user interface element 888 is illustrated as a three-dimensional-like representation. In some embodiments, the reference user interface element 888 is a graphical representation of a sphere, a dome, or a half-dome. In some embodiments, the reference user interface element 888 includes one or more axis indicators (e.g., an x-axis indicator 892 and a y-axis indicator 894). In some embodiments, the reference user interface element 888 (and one or more axis indicators) is translucent (e.g., semi-transmissive). This facilitates the simultaneous display of the scope position with respect to the reference user interface element 888 and one or more axis indicators. Figure 31C shows representations of the indicator 886 in multiple states (e.g., the indicator 886 representing a scope that bends upward is shown as a solid line, and the indicator 886 representing a scope that bends leftward, rightward, and downward is shown as a dashed line), but in some embodiments, only the indicator 886 in a single state is displayed (e.g., to represent the state of a scope that bends upward, the indicator 886 represented by a solid line is shown without a dashed line). In some embodiments, a portion of the reference user interface element 888 is visually distinguished (e.g., emphasized) to indicate the direction or orientation of the scope.For example, a portion of a reference user interface element 888 (e.g., a semi-dome), such as one of four pre-defined quadrants, is visually distinguished (e.g., in terms of transparency, color, etc., visually distinguished from the remaining ones of the four pre-defined quadrants of the semi-dome) to emphasize the orientation or state of the flexible scope (or the orientation of the state in which the flexible scope is moving).

[0224] FIG. 31D illustrates an indicator 896 according to some embodiments. The indicator 896 shown in FIG. 31D is similar to the indicator 880 shown in FIG. 31B, except that the indicator 896 indicates the insertion and / or advancement of the scope in addition to the flexion of the scope. For example, the indicator 896 in states 896-1, 896-2, and 896-3 indicates different degrees of insertion of the scope. The indicator 896 in state 896-4 indicates the flexion of the scope, and by representing the scope in a flexed form, indicates that the scope is in a flexed state (or that the scope has started to flex). In some embodiments, one or more reference user interface elements (e.g., the reference plane shown in FIG. 31B or the reference user interface element 888 shown in FIG. 31C) are displayed on the indicator 896. In some embodiments, one or more reference user interface elements are displayed according to a determination that the scope is flexed (e.g., one or more reference user interface elements are displayed only after the scope is flexed and not before flexion). In some embodiments, one or more reference user interface elements cease to be displayed according to a determination that the scope is straight (e.g., one or more reference user interface elements are removed when the scope returns from a flexed state to a straight state). In some embodiments, one or more reference user interface elements are always displayed. In some embodiments, one or more reference user interface elements are not shown.

[0225] In some embodiments, indicator 880 or 896 is displayed within a picture-in-picture window such as windows 3102, 3104, or 3106 shown in FIG. 31E. In some embodiments, the picture-in-picture window is placed at a corner of user interface 802. This reduces operator (e.g., surgeon) distraction.

[0226] In some embodiments, the picture-in-picture window shown in FIG. 31E is moved away from user interface 802 so that the picture-in-picture window does not overlap with user interface 802 (e.g., the picture-in-picture windows such as windows 3102, 3104, and 3106 are moved to a location below user interface 802), as shown in FIG. 31F. In some embodiments, user interface 802 is scaled (e.g., reduced) and / or moved (e.g., the center of user interface 802 is moved upward) to provide space on display device 800 for the display of a window that is away from user interface 802, as shown in FIG. 31F.

[0227] In some embodiments, the visual representations shown in FIGS. 28A-20D, FIGS. 29A-29D, FIGS. 30A-30F, and FIGS. 31A-31F, or one or more portions thereof, are color-coded. In some embodiments, at least a portion (e.g., the straight portion) of the indicator 896 may be color-coded to indicate the insertion ratio. For example, at least the straight portion of the indicator 896 has a first color, such as green, while the scope is inserted from 0% to 80%, a second color, such as yellow, while the scope is inserted from more than 80% to 99%, and a third color, such as red, while the scope is in a 100% inserted state. In some embodiments, at least a portion of the indicator 896 may be color-coded to indicate the degree of bending of the flexible end of the scope (e.g., at least a portion of the indicator 896 has a specific color, such as red, to indicate that the scope cannot be bent further or that the scope is in a state of maximum bending). In some embodiments, the insertion degree of the scope or the bending degree of the scope is indicated by color without displaying the entire indicator 896. For example, a color indicator without the shape of the scope is used to indicate the insertion degree of the scope or the bending degree of the scope. In some embodiments, the insertion degree of the scope or the bending degree of the scope is indicated by an animation associated with the indicator 896. For example, the indicator 896 may be displayed to vibrate for at least a predefined time to indicate the insertion degree of the scope or the bending degree of the scope (e.g., that the scope has reached a fully inserted or fully bent state).

[0228] In some embodiments, the visual representations illustrated in FIGS. 28A-20D, FIGS. 29A-29D, FIGS. 30A-30F, and FIGS. 31A-31F are provided with one or more audible signals (e.g., dedicated sounds, e.g., a first sound indicating that the scope is fully inserted and a second sound different from the first sound indicating that the scope is fully bent).

[0229] In some embodiments, the visual representations illustrated in FIGS. 28A-28D, FIGS. 29A-29D, FIGS. 30A-30F, and FIGS. 31A-31F are displayed when a surgeon activates the "camera clutch" mode. In some embodiments, the surgeon activates the "camera clutch" mode by pressing a camera pedal (e.g., foot pedal 244) of the surgeon console 240, or by activating a button on the console touch screen display, or by a bedside arm touch point.

[0230] In some embodiments, the visual representations illustrated in FIGS. 28A-28D, FIGS. 29A-29D, FIGS. 30A-30C, and FIGS. 31A-31F are displayed according to a determination (e.g., by processor 380) that the robotic medical system is operating (regardless of whether, for example, a flexible scope is operating in a particular drive mode).

[0231] As described above, FIGS. 28A-28D, FIGS. 29A-29D, and FIGS. 30A-30C illustrate exemplary user interfaces. One of ordinary skill in the art will understand that one or more processors (e.g., a central processing unit, a graphics processing unit, an acceleration processing unit, an application specific integrated circuit, etc.) may be configured to generate electrical signals (e.g., video signals) such that one or more display devices can display the user interfaces illustrated in FIGS. 28A-28D, FIGS. 29A-29D, and FIGS. 30A-30C.

[0232] F. Exemplary process for displaying visual indicators.

[0233] Figures 32A-32F illustrate a flowchart of a method 900 implemented by one or more processors (e.g., processor 380) of a robotic system (e.g., a robotic medical system 200 as illustrated in FIGS. 21 and 22, a surgical robotic system, or a robotic surgery platform, etc.). The surgical robotic system includes a memory storing instructions executable by one or more processors.

[0234] The robotic system includes a first robotic arm (e.g., a robotic manipulator) (e.g., robotic arm 210-1 of FIG. 22) coupled to a flexible scope (e.g., a flexible scope 720 such as a flexible laparoscope or a flexible endoscope).

[0235] The robotic system includes a viewer (e.g., a display device 232 located on a tower 230, or a display device 242 included with a physician console 240) for displaying a view of a surgical site derived from the flexible scope (e.g., the flexible scope is coupled to a camera at a distal end of the scope) (the flexible scope introduces new degrees of freedom and enables a surgeon to control the camera in more ways than with a rigid scope).

[0236] The robotic system includes one or more processors (e.g., processor 380) and a memory (e.g., memory 382) storing instructions executable by the one or more processors.

[0237] The robotic system operates the flexible scope in a particular mode (e.g., a drive mode) (e.g., each of a plurality of (drive) modes) among a plurality of modes (902). In some embodiments, the plurality of modes includes an articulation mode, a trajectory mode, and / or an automatic insertion and retraction mode. In some embodiments, the plurality of modes also includes one or more additional modes.

[0238] According to the determination (904) that the flexible scope is operating in a specific mode among a plurality of modes, the robot system provides an electrical signal (e.g., a video signal such as an analog video signal or a digital video signal) for presenting a first visual indicator corresponding to each mode (e.g., the indicators 806 in FIGS. 28A, 28B, and 28C, the indicator 822 in FIG. 28D, the indicators 832 in FIGS. 29A and 29B, the indicator 844 in FIG. 29C), the indicator 854 in FIG. 29D, the indicators 860 in FIGS. 30A and 30B, or the indicator 866 in FIG. 30C) on the viewer. The first visual indicator indicates at least one of the position (e.g., the first position) or orientation (e.g., the first orientation) of the flexible scope with respect to the reference position (e.g., the position and / or orientation of the distal end of the flexible scope).

[0239] In some embodiments, the flexible scope is inserted (906) into the patient through a first port (e.g., port 608), and the reference position is determined based on (e.g., corresponding to) the position of the first port. For example, in some embodiments, the flexible scope is aligned with the line of sight from the cannula to the port when inserted through the port into the surgical site.

[0240] In some embodiments, while the robot system is operating the flexible scope in a first mode (e.g., the joint drive mode) among a plurality of modes, the robot system causes (e.g., via one or more processors) the distal portion of the flexible scope (e.g., the distal end 724 of the flexible scope 720, the link 726 located at the distal portion of the flexible scope, the joint 728 of the distal portion of the flexible scope, etc.) to move from the first position or the first orientation to a second position different from the first position or a second orientation different from the first orientation (e.g., joint movement, tilt, rotation, pivoting movement, rotational movement, etc.) (908).

[0241] For example, movement of the distal portion of the flexible scope is achieved by positioning the flexible scope in a second position (different from the first position) or a second orientation (different from the first orientation), and the first visual indicator is updated to indicate the second position or second orientation of the flexible scope (or its distal portion).

[0242] In some embodiments, movement of the distal end of the flexible scope includes movement in an upward, downward, leftward, and / or rightward direction.

[0243] In some embodiments, the flexible scope includes a proximal end (e.g., intermediate portion 722) that remains stationary while the distal portion is moved from a first position or first orientation to a second position or second orientation.

[0244] In some embodiments, the robotic system updates (910) the first visual indicator to indicate the second position or second orientation of the distal portion of the flexible scope relative to a reference position (e.g., relative to the neutral position of the flexible scope) (e.g., when the flexible scope is perfectly aligned (straight) with the line of sight from the cannula to the port). For example, in FIG. 28B, indicator 806 is updated to indicate the second position or second orientation of the distal portion of the flexible scope.

[0245] In some embodiments, the first visual indicator includes one or more reference axes and one or more graphical elements (e.g., markers) positioned relative to the one or more reference axes to specify the direction and / or extent of movement (912). For example, indicator 806 includes reference axis 807 and graphical element (e.g., marker) 808 positioned relative to reference axis 807 to specify the direction and / or extent of movement.

[0246] In some embodiments, the robotic system provides (914) an electrical signal (e.g., automatically, in real time) for presenting an updated graphical element (e.g., graphical element 811) indicative of a change in the position and / or orientation of the flexible scope associated with the movement, in accordance with a determination that the distal portion of the flexible scope has moved from a first position and / or orientation to a second position and / or orientation (e.g., automatically, in real time).

[0247] In some embodiments, the robotic system determines (916) the shape of the flexible scope (e.g., in real time). For example, the robotic system may determine the shape of the flexible scope from control signals provided to the flexible scope. The robotic system provides (918) an image (e.g., a rendered image such as a 3D rendered image) (e.g., representation 812) corresponding to (e.g., indicative of, specifying, or including) the determined shape.

[0248] In some embodiments, the image includes (920) two or more links (e.g., link 814) corresponding to two or more links (e.g., link 726) of the flexible scope.

[0249] In some embodiments, the robotic system provides (922) an electrical signal for presenting an updated image (e.g., representation 813) indicative of the changed shape, in accordance with a (e.g., real-time) determination that the shape of the flexible scope has changed from a first shape to a second shape (e.g., due to a change in the direction and / or degree of articulation).

[0250] In some embodiments, the first visual indicator includes (924) a plurality of graphical elements (e.g., two bars (the white portions of the bars), rectangles, ellipses, etc.). The plurality of graphical elements includes a first graphical element (e.g., graphical element 828-1) (e.g., a bar, rectangle, line, ellipse, etc.) and a second graphical element (e.g., graphical element 828-2) that is different from the first graphical element (e.g., a bar, rectangle, line, ellipse, etc.). The first graphical element represents a first component (e.g., direction and / or degree) of the movement of the distal end of the flexible scope along a first direction (e.g., left or right). The second graphical element represents a second component (e.g., direction and / or degree) of the movement of the distal end of the flexible scope along a second direction (e.g., upward or downward). The first direction is different from the second direction (e.g., the second direction is perpendicular to the first direction, and the first direction and the second direction are not parallel to each other).

[0251] In some embodiments, the robotic system provides (926) an electrical signal for updating the position of the first graphical element in accordance with a determination (e.g., in real time) that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude. In some embodiments, the robotic system provides (928) an electrical signal for updating the position of the second graphical element in accordance with a determination (e.g., in real time) that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude.

[0252] In some embodiments, the robotic system provides (930) an electrical signal to update the length (e.g., length 830-1) of a first graphical element (e.g., graphical element 828-1) according to a determination (e.g., in real time) that movement of the flexible scope in a first direction (e.g., left or right) has changed from a first magnitude to a second magnitude. In some embodiments, the robotic system provides (932) an electrical signal to update the length (e.g., length 830-2) of a second graphical element (e.g., graphical element 828-2) according to a determination (e.g., in real time) that movement of the flexible scope in a second direction (e.g., up or down) has changed from a third magnitude to a fourth magnitude.

[0253] In some embodiments, the view of the surgical site is derived from a camera coupled to the flexible scope (e.g., at the distal portion of the scope). The robotic system expands (934) a first visual indicator (e.g., the scope tab) according to a determination that the camera is active. For example, as illustrated in FIG. 28A, in some embodiments, indicator 806 may have an initial size similar to indicator 804. The size of indicator 806 expands when the camera coupled to flexible scope 720 is active.

[0254] In some embodiments, the robotic system causes movement of the flexible scope about a target point (e.g., target point 734) in space while operating the flexible scope in a second mode (e.g., the orbital drive mode) of a plurality of modes (e.g., orbital points) (e.g., the orbital drive mode).

[0255] In some embodiments, the target point is predefined (e.g., selected) by the surgeon. In some embodiments, the target point is predefined by the robotic system. In some embodiments, the target point is randomly selected by the robotic system. In some embodiments, the target point corresponds to the center of the camera's field of view when the trajectory drive mode is activated. In some embodiments, the target point is located off-center from the center of the camera's field of view.

[0256] In some embodiments, the first visual indicator includes (938) a first graphical element (e.g., graphical element 834) (e.g., black circle, white circle, sphere, cross) representing the target point.

[0257] In some embodiments, the robotic system provides (940) an electrical signal for overlaying the first graphical element on the displayed field of view (e.g., overlaying the first point on the displayed field of view of the surgical site). For example, in FIG. 29A, graphical element 834 is overlaid on the representation 840 of the field of view.

[0258] In some embodiments, the first visual indicator includes (942) a second graphical element (e.g., graphical element 836) (e.g., straight line, solid line, dashed line) representing the (e.g., direct) line of sight from the target point to the port of the flexible scope.

[0259] In some embodiments, the first visual indicator includes a text element (e.g., element 838) (e.g., "scope port") indicating the direction of the scope port.

[0260] In some embodiments, when the port, camera, and target point are aligned (meaning the port is immediately behind the camera), the second graphical element converges / fuses with the first graphical element.

[0261] In some embodiments, a first graphical element (e.g., graphical element 834) has a first size (e.g., diameter, length, width, area, etc.) (e.g., is characterized by the first size). The robotic system adjusts (e.g., changes, modifies, etc.) the size of the first graphical element from the first size to a second size (e.g., in real time) (944) in accordance with a change in the insertion depth of the flexible scope into the surgical site (e.g., graphical element 834 in FIGS. 29A and 29B).

[0262] In some embodiments, the first size is predefined by one or more processors of the robotic system (e.g., processor 380).

[0263] In some embodiments, the first size indicates the distance from the flexible scope to the target organ.

[0264] In some embodiments, the first size indicates the insertion distance of the scope (e.g., how far the scope is inserted into the body). For example, a larger size indicates that the scope is inserted further into the body, and a smaller size indicates that the scope is closer to the surface.

[0265] In some embodiments, the size of the first graphical element increases in accordance with a determination that the insertion depth has increased (e.g., the flexible scope is inserted further into the surgical site). In some embodiments, the size of the first graphical element decreases in accordance with a determination that the insertion depth has decreased (e.g., the flexible scope is moving further away from the surgical site).

[0266] In some embodiments, a first visual indicator (e.g., indicator 844) includes a second graphical element (e.g., graphical element 846) (e.g., spherical, spherical body, etc.) that represents the range of the orbital movement of the flexible scope (946).

[0267] In some embodiments, the first visual indicator (e.g., indicator 844) includes (948) a third graphical element (e.g., graphical element 850) (e.g., a marker such as a black circle, "x", etc.) that represents the (e.g., real-time, current) position of the flexible scope.

[0268] In some embodiments, the robotic system provides (950) an electrical signal for overlaying the first graphical element, the second graphical element, and the third graphical element (e.g., graphical elements 846, 848, and 850 of FIG. 29C) on the displayed field of view.

[0269] In some embodiments, the first visual indicator (e.g., indicator 854) includes one or more axes (e.g., three axes such as axis 856) that represent the insertion direction (or rotation) of the flexible scope with respect to one or more axes (e.g., axis 856 of FIG. 29D). In some embodiments, the one or more axes are positioned adjacent to the upper edge, lower edge, left edge, or right edge of the user interface 802. In some embodiments, the one or more axes are positioned at the corners of the user interface 802.

[0270] In some embodiments, the robotic system automatically inserts (e.g., from the cannula of the first robotic arm to the surgical site, from the port to the surgical site, etc.) and / or (automatically) retracts (e.g., from the surgical site to the port, from the surgical site to the cannula of the first robotic arm, etc.) the flexible scope (e.g., without applying manual force to the flexible scope) when the robotic system is operating the flexible scope in a third mode (e.g., the automatic insertion and retraction mode) of a plurality of modes (952).

[0271] In some embodiments, the robotic system determines (954) the shape of the flexible scope (e.g., in real time) during activation of the third mode (or during automatic insertion or automatic retraction). For example, the robotic system may determine the shape of the flexible scope based on a control signal provided to the flexible scope. Alternatively, the robotic system may determine the shape of the flexible scope based on signals indicating the position or state of each joint of the flexible scope. The robotic system provides (956) an image (e.g., a rendering image such as a 3D rendering image) corresponding to the determined shape (e.g., the graphical element 861 in FIGS. 30A and 30B).

[0272] In some embodiments, the robotic system updates (958) the image (e.g., automatically, in real time) while the flexible scope is being automatically inserted or retracted (e.g., the graphical element 862 in FIG. 30B).

[0273] In some embodiments, the first visual indicator (e.g., indicator 866) includes (960) graphical elements (e.g., graphical element 868) (e.g., bars, shading, etc.) indicating the degree of insertion (e.g., fully inserted, partially inserted, etc.) and / or the degree of retraction (e.g., fully retracted, partially retracted, etc.). In some embodiments, the first visual indicator (e.g., indicator 866) includes graphical elements (e.g., graphical element 872) indicating the articulation movement (e.g., straight, articulated) of the flexible scope.

[0274] In some embodiments, the robotic system includes (962) a second robotic arm (e.g., a robotic manipulator) (e.g., robotic arm 210-2) coupled to a surgical tool (e.g., the medical instrument 212 in FIGS. 21 and 24A, the surgical tool 602 in FIG. 25, and the surgical tools 704 and 708 in FIGS. 26A and 26B).

[0275] In some embodiments, the robotic system provides (964) an electrical signal for presenting on a viewer a second visual indicator (e.g., indicator 804) corresponding to a surgical tool. In some embodiments, the second visual indicator identifies a medical instrument (e.g., surgical) at the surgical site. In some embodiments, the second visual indicator 804 includes an association between the robotic arm and each medical instrument to which it is coupled. For example, in FIG. 28A, indicator 804-2 includes the number "2" indicating that tool B is coupled to the second robotic arm of the robotic medical system.

[0276] In some embodiments, the first visual indicator includes (966) information (e.g., a robotic arm number) identifying the first robotic arm coupled to the flexible scope. For example, in FIGS. 28A, 28B, and 28C, indicator 806 includes the number "3" identifying the robotic arm (e.g., robotic arm 3) coupled to the flexible scope.

[0277] In some embodiments, the first visual indicator is presented (968) simultaneously with the display of the surgical site view on the viewer (e.g., the indicators in FIGS. 28A-28D, FIGS. 29A-29D, and FIGS. 30A-30C).

[0278] In some embodiments, the first visual indicator is presented (970) around (e.g., along the periphery of) the display of the surgical site view (e.g., the indicators in FIGS. 28A-28D, FIG. 29D, and FIG. 30C).

[0279] In some embodiments, the first visual indicator is presented (972) on top of the display of the surgical site view (e.g., the indicators in FIGS. 28A-28D, FIGS. 29A-29D, and FIGS. 30A-30C).

[0280] In some embodiments, the viewer is part of a surgeon console (e.g., the display device 242 included in the physician console 240) (974).

[0281] In some embodiments, the surgeon console includes an input device (e.g., an input device such as a user interface, button, switch, touch-sensing surface, or gimbal of the display device 242, a foot pedal 244, etc.) (976). The robotic system receives an input on the input device (978). The robotic system determines that the flexible scope is operating in a particular mode according to the received input (980).

[0282] In some embodiments, the input device includes a foot pedal (e.g., the foot pedal 244) (982).

[0283] In some embodiments, the first visual indicator includes a graphical representation of the flexible scope in a non-primary graphical projection.

[0284] In some embodiments, the robotic system displays one or more reference user interface elements according to a determination that the flexible scope is bent.

[0285] In some embodiments, the robotic system stops displaying one or more reference user interface elements according to a determination that the flexible scope is straight.

[0286] 3. Implementation System and Terms. FIG. 33 is a schematic diagram illustrating electronic components of a medical robotic system (e.g., a surgical robotic system) according to some embodiments.

[0287] A robotic medical system (e.g., a surgical robotic system) includes one or more processors 380 that communicate with a computer-readable storage medium 382 (e.g., a computer memory device such as random access memory, read-only memory, static random access memory, and non-volatile memory, as well as other storage devices such as hard drives, optical disks, magnetic tape recording, or any combination thereof) that stores instructions for performing any method described herein (e.g., the operations described with respect to FIGS. 25, 26A, 26B, 27A-27D, 28A-28D, 29A-29D, 30A-30C, and 32A-32F). The one or more processors 380 also communicate (via a system bus or any suitable electrical circuit) with an input / output controller 384. The input / output controller 384 receives sensor data from one or more sensors 388-1, 388-2, etc. and relays the sensor data to the one or more processors 380. The input / output controller 384 also receives instructions and / or data from the one or more processors 380 and relays the instructions and / or data to one or more actuators such as first motors 387-1 and 387-2. In some embodiments, the input / output controller 384 is coupled to one or more actuator controllers 386 and provides instructions and / or data to at least a subset of the one or more actuator controllers 386, which in turn provide control signals to the selected actuators. In some embodiments, the one or more actuator controllers 386 are integrated with the input / output controller 384, and the input / output controller 384 provides control signals directly to the one or more actuators 387 (without an intervening actuator controller). FIG. 30 shows one actuator controller 386 (e.g., indicating that there is one actuator controller for the entire medical robotic system, although in some embodiments additional actuator controllers may be used (e.g., one actuator controller per actuator, etc.).In some embodiments, one or more processors 380 communicate with one or more displays 381 for displaying information, as described herein.

[0288] It should be noted that, as used herein, the terms "coupled," "coupling," "is coupled to," or other variations of the word coupling may indicate either an indirect connection or a direct connection. For example, if a first component is "coupled to" a second component, the first component may be either indirectly connected to the second component through another component or directly connected to the second component.

[0289] The function for a tool to determine whether it is within or outside the surgical field provided by a camera or a scope and to render one or more indicators representing the position or orientation of one or more of the medical tools described herein may be stored as one or more instructions on a processor-readable medium or a computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or a processor. By way of example, and not limitation, such a medium can include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, compact disc read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be noted that the computer-readable medium can be tangible and non-transitory. As used herein, the term "code" can refer to software, instructions, code, or data that is executable by a computing device or a processor.

[0290] The methods disclosed herein include one or more steps or acts for achieving the described methods. The method steps and / or acts can be replaced with each other without departing from the scope of the claims. In other words, the order and / or use of the specific steps and / or acts can be modified without departing from the scope of the claims, provided that no specific order of steps or acts is required for the proper operation of the described methods.

[0291] As used herein, the term "plurality" means two or more. For example, a plurality of components means two or more components. The term "determine" encompasses a wide variety of acts, and thus, "determine" can include calculating, computing, processing, deriving, investigating, examining (e.g., examining a table, database, or another data structure), verifying, etc. Also, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Further, "determine" can include solving, selecting, choosing, establishing, etc.

[0292] The phrase "based on" does not mean "based only on" unless explicitly specified otherwise. In other words, the phrase "based on" describes both "based only on" and "based at least on".

[0293] As used herein, the term "exemplary" means "serving as an example, instance, or illustration", and does not necessarily imply any preference or superiority of the example over any other configuration or implementation.

[0294] As used herein, the term "and / or" encompasses any combination of the listed elements. For example, "A, B, and / or C" includes the following sets of elements: only A, only B, only C, A and B without C, A and C without B, B and C without A, and the combination of all three elements A, B, and C.

[0295] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the invention. For example, those skilled in the art will understand that many corresponding alternative and equivalent structural details, such as equivalent ways of fastening, attaching, coupling, or engaging tool components, equivalent mechanisms for producing certain operating movements, and equivalent mechanisms for delivering electrical energy, can be employed. Accordingly, the invention is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0296] 〔Embodiment〕 (1) A robotic system, a first robotic arm coupled to a flexible scope, a viewer for displaying a view of a surgical site derived from the flexible scope, one or more processors, a memory configured to store instructions executed by the one or more processors, the stored instructions including instructions for operating the flexible scope in a particular mode of a plurality of modes, and providing an electrical signal for presenting a first visual indicator corresponding to each mode on the viewer according to a determination that the flexible scope is operating in the particular mode of the plurality of modes, the first visual indicator indicating at least one of a position or orientation of the flexible scope relative to a reference position, a memory. A robotic system comprising (2) The robotic system according to Embodiment 1, wherein the flexible scope is inserted into a patient through a first port, and the reference position is determined based on the position of the first port. (3) The robotic system of embodiment 1 or 2, wherein the robotic system causes movement of a distal portion of the flexible scope from a first position or a first orientation to a second position different from the first position or a second orientation different from the first orientation while the robotic system is operating the flexible scope in a first mode of the plurality of modes. (4) The robot system of embodiment 3, wherein the stored instructions include instructions for updating the first visual indicator to indicate the second position or the second orientation of the distal portion of the flexible scope relative to the reference position. (5) The robot system of embodiment 4, wherein the first visual indicator includes one or more reference axes and a graphical element positioned relative to the one or more reference axes to identify at least one of the direction or extent of the movement.

[0297] (6) The stored instructions also include The robot system of embodiment 4 or 5, further comprising instructions for providing an electrical signal to present an updated graphical element indicating at least one change in the position or orientation of the flexible scope associated with a determination that the distal portion of the flexible scope has moved to at least one of (i) the first position to the second position, or (ii) from the first orientation to the second orientation. (7) The stored instructions also include determining a shape of the flexible scope; A robot system described in any of embodiments 3 to 6, comprising instructions for: and providing an image corresponding to the determined shape. (8) The stored instructions also include The robotic system of embodiment 7, further comprising instructions for providing an electrical signal to present an updated image showing the changed shape in accordance with a determination that the shape of the flexible scope has changed from a first shape to a second shape. (9) The image includes two or more links corresponding to two or more links of the flexible scope, and the robot system according to Embodiment 7 or 8. (10) The first visual indicator includes a plurality of graphical elements, and the plurality of graphical elements include a first graphical element and a second graphical element different from the first graphical element. The first graphical element represents a first component of the movement of the distal end of the flexible scope along a first direction. The second graphical element represents a second component of the movement of the distal end of the flexible scope along a second direction, and the first direction is different from the second direction. The robot system according to any one of Embodiments 3 to 9.

[0298] (11) The stored instructions also Provide an electrical signal for updating the position of the first graphical element according to a determination that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude. Provide an electrical signal for updating the position of the second graphical element according to a determination that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude. The robot system according to Embodiment 10 includes instructions for this. (12) The stored instructions also Provide an electrical signal for updating the length of the first graphical element according to a determination that the movement of the flexible scope in the first direction has changed from a first magnitude to a second magnitude. Provide an electrical signal for updating the length of the second graphical element according to a determination that the movement of the flexible scope in the second direction has changed from a third magnitude to a fourth magnitude. The robot system according to Embodiment 10 or 11 includes instructions for this. (13) The view of the surgical site is derived from a camera coupled to the flexible scope, The robot system according to any one of embodiments 1 to 12, wherein the stored instructions also include instructions for expanding the first visual indicator according to a determination that the camera is active. (14) The robot system according to any one of embodiments 1 to 13, which causes the flexible scope to move around a target point in space while the robot system is operating the flexible scope in a second mode among the plurality of modes. (15) The robot system according to embodiment 14, wherein the first visual indicator includes a first graphical element representing the target point.

[0299] (16) The robot system according to embodiment 15, wherein the stored instructions also include instructions for providing an electrical signal for overlaying the first graphical element on the view. (17) The robot system according to embodiment 15 or 16, wherein the first visual indicator includes a second graphical element representing a line of sight from the target point to the port of the flexible scope. (18) The first graphical element has a first size, The stored instructions also The robot system according to any one of embodiments 15 to 17, wherein the stored instructions include instructions for adjusting the size of the first graphical element from the first size to a second size according to a change in the insertion depth of the flexible scope into the surgical site. (19) The robot system according to any one of embodiments 15 to 18, wherein the first visual indicator includes a second graphical element representing a range of orbital movement of the flexible scope. (20) The robot system according to embodiment 19, wherein the first visual indicator includes a third graphical element representing the position of the flexible scope.

[0300] (21) The robot system according to embodiment 20, wherein the stored instructions also include instructions for providing an electrical signal for overlaying the first graphical element, the second graphical element, and the third graphical element on the visual field. (22) The first visual indicator comprises one or two or more axes and a graphical element representing the insertion direction of the flexible scope with respect to the one or two or more axes, the robot system according to any one of embodiments 1 to 21. (23) The robot system according to any one of embodiments 1 to 22, wherein when the robot system operates the flexible scope in a third mode among the plurality of modes, the flexible scope is automatically inserted and / or retracted. (24) The stored instructions also include instructions for determining the shape of the flexible scope during activation of the third mode and providing an image corresponding to the determined shape, the robot system according to embodiment 23. (25) The stored instructions also include instructions for updating the image while the flexible scope is automatically inserted or retracted, the robot system according to embodiment 24.

[0301] (26) The first visual indicator includes a graphical element indicating at least one of the degree of insertion, the degree of retraction, or the articular movement of the flexible scope, the robot system according to any one of embodiments 23 to 25. (27) The robot system according to any one of embodiments 1 to 26, further comprising a second robotic arm coupled to a surgical tool. (28) The stored instructions also include instructions for providing an electrical signal for presenting a second visual indicator corresponding to the surgical tool on the viewer, the robot system according to embodiment 27. (29) The first visual indicator includes information identifying the first robotic arm coupled to the flexible scope, the robotic system according to any one of Embodiments 1 to 28. (30) The first visual indicator is displayed on the viewer simultaneously with the display of the visual field of the surgical site, the robotic system according to any one of Embodiments 1 to 29.

[0302] (31) The first visual indicator is displayed around the display of the visual field of the surgical site, the robotic system according to Embodiment 30. (32) The first visual indicator is displayed on the display of the visual field of the surgical site, the robotic system according to Embodiment 30 or 31. (33) The viewer is part of a surgeon console, the robotic system according to any one of Embodiments 1 to 32. (34) The surgeon console includes an input device, The stored instructions are, receiving an input on the input device, determining, according to the received input, that the flexible scope is operating in the specific mode, the robotic system according to Embodiment 33 including instructions for this. (35) The input device includes a foot pedal, the robotic system according to Embodiment 34.

[0303] (36) The first visual indicator includes a graphical representation of the flexible scope in a graphical projection, the robotic system according to any one of Embodiments 1 to 35. (37) The stored instructions include instructions for displaying one or more reference user interface elements according to a determination that the flexible scope is bent, the robotic system according to Embodiment 36. (38) The robot system according to embodiment 37, wherein the stored instructions include instructions for stopping displaying the one or more reference user interface elements according to a determination that the flexible scope is straight. (39) The robot system according to any one of embodiments 36 to 38, wherein the stored instructions include instructions for displaying one or more reference user interface elements according to a determination of the degree of insertion of the flexible scope. (40) A robot system, a first robotic arm coupled to a flexible scope, a viewer for displaying a view of a surgical site derived from the flexible scope, one or more processors, a memory configured to store instructions executed by the one or more processors, the stored instructions including operating the flexible scope, providing an electrical signal for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of a position or orientation of the flexible scope relative to a reference position, and a memory including instructions for providing.

[0304] (41) An electronic device communicating with a robot system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the electronic device including one or more processors, a memory configured to store instructions executed by the one or more processors, the stored instructions including operating the flexible scope in a specific mode among a plurality of modes, In accordance with a determination that the flexible scope is operating in the particular mode of the plurality of modes, providing an electrical signal for presenting, on the viewer, a first visual indicator corresponding to each of the modes, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position, and a memory comprising instructions for doing so. An electronic device. (42) An electronic device that communicates with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the electronic device comprising: One or more processors; A memory configured to store instructions executed by the one or more processors, the stored instructions comprising: Operating the flexible scope; Providing an electrical signal for presenting, on the viewer, a first visual indicator, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position, and a memory comprising instructions for doing so. An electronic device. (43) A computer-readable storage medium storing instructions executed by one or more processors that communicate with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the stored instructions comprising: Operating the flexible scope in a particular mode of a plurality of modes; In accordance with a determination that the flexible scope is operating in the particular mode of the plurality of modes, providing an electrical signal for presenting, on the viewer, a first visual indicator corresponding to each of the modes, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position, and a computer-readable storage medium comprising instructions for doing so. (44) A computer-readable storage medium storing instructions executed by one or more processors that communicate with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the stored instructions being to operate the flexible scope, to provide an electrical signal for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position, the computer-readable storage medium including instructions for.

Claims

**Claim 1** A robotic system comprising: a first robotic arm coupled to a flexible scope; a viewer for displaying a view of a surgical site derived from the flexible scope; one or more processors; a memory configured to store instructions executed by the one or more processors, the stored instructions including instructions for: operating the flexible scope in a particular mode of a plurality of modes; and providing an electrical signal for presenting, on the viewer, a first visual indicator corresponding to each mode in accordance with a determination that the flexible scope is operating in the particular mode of the plurality of modes, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position. **Claim 2** The robotic system of claim 1, wherein the flexible scope is inserted into a patient through a first port, and the reference position is determined based on the position of the first port. **Claim 3** The robotic system of claim 1 or 2, wherein the robotic system causes movement of a distal portion of the flexible scope from a first position or a first orientation to a second position different from the first position or a second orientation different from the first orientation while operating the flexible scope in a first mode of the plurality of modes. **Claim 4** The robotic system of claim 3, wherein the stored instructions include instructions for updating the first visual indicator to indicate the second position or the second orientation of the distal portion of the flexible scope relative to the reference position. **Claim 5** The robotic system of claim 4, wherein the first visual indicator includes one or more reference axes and graphical elements positioned relative to the one or more reference axes to identify at least one of a direction or an extent of the movement. **Claim 6** The stored instructions also include: The robot system according to claim 4, comprising instructions for providing an electrical signal for presenting an updated graphical element indicating at least one change in the position or orientation of the flexible scope associated with the movement, according to a determination that the distal portion of the flexible scope has moved (i) from the first position to the second position, or (ii) from the first orientation to the second orientation, or at least one of both.

7. The stored instructions also include instructions for determining the shape of the flexible scope and providing an image corresponding to the determined shape, for the robot system according to claim 3.

8. The stored instructions also include instructions for providing an electrical signal for presenting an updated image indicating the changed shape, according to a determination that the shape of the flexible scope has changed from a first shape to a second shape, for the robot system according to claim 7.

9. The image includes two or more links corresponding to two or more links of the flexible scope, for the robot system according to claim 7.

10. The first visual indicator includes a plurality of graphical elements, the plurality of graphical elements including a first graphical element and a second graphical element different from the first graphical element, the first graphical element representing a first component of the movement of the distal end of the flexible scope along a first direction, the second graphical element representing a second component of the movement of the distal end of the flexible scope along a second direction, the first direction being different from the second direction, for the robot system according to claim 3.

11. The stored instructions also include instructions for providing an electrical signal for updating the position of the first graphical element, according to a determination that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude, and providing an electrical signal for updating the position of the second graphical element, according to a determination that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude, for the robot system according to claim 10.

12. The stored instructions also providing an electrical signal for updating the length of the first graphical element according to a determination that the movement of the flexible scope in the first direction has changed from a first magnitude to a second magnitude; The robot system according to claim 10, comprising instructions for providing an electrical signal for updating the length of the second graphical element according to a determination that the movement of the flexible scope in the second direction has changed from a third magnitude to a fourth magnitude. **Claim 13** The visual field of the surgical site is derived from a camera coupled to the flexible scope. The robot system according to claim 1, wherein the stored instructions further include instructions for expanding the first visual indicator according to a determination that the camera is active. **Claim 14** The robot system according to claim 1, wherein the robot system causes the flexible scope to move about a target point in space while operating the flexible scope in a second mode of the plurality of modes. **Claim 15** The robot system according to claim 14, wherein the first visual indicator includes a first graphical element representing the target point. **Claim 16** The robot system according to claim 15, wherein the stored instructions further include instructions for providing an electrical signal for overlaying the first graphical element on the visual field. **Claim 17** The robot system according to claim 15, wherein the first visual indicator includes a second graphical element representing a line of sight from the target point to the port of the flexible scope. **Claim 18** The first graphical element has a first size. The stored instructions also The robot system according to claim 15, wherein the stored instructions further include instructions for adjusting the size of the first graphical element from the first size to a second size according to a change in the insertion depth of the flexible scope into the surgical site. **Claim 19** The robot system according to claim 15, wherein the first visual indicator includes a second graphical element representing a range of orbital movement of the flexible scope. **Claim 20** The robot system according to claim 19, wherein the first visual indicator includes a third graphical element representing the position of the flexible scope. **Claim 21** The robot system according to claim 20, wherein the stored instructions further include instructions for providing an electrical signal for overlaying the first graphical element, the second graphical element, and the third graphical element on the visual field.

22. The first visual indicator comprises one or more axes and a graphical element representing the insertion direction of the flexible scope with respect to the one or more axes, the robot system according to claim 1.

23. The robot system according to claim 1, wherein the robot system automatically inserts and / or retracts the flexible scope when operating the flexible scope in a third mode among the plurality of modes.

24. The stored instructions also include instructions for determining the shape of the flexible scope during activation of the third mode and providing an image corresponding to the determined shape, the robot system according to claim 23.

25. The robot system according to claim 24, wherein the stored instructions further include instructions for updating the image while the flexible scope is being automatically inserted or retracted.

26. The first visual indicator includes a graphical element indicating at least one of the degree of insertion, the degree of retraction, or the articulation movement of the flexible scope, the robot system according to claim 23.

27. The robot system according to claim 1, further comprising a second robotic arm coupled to the surgical tool.

28. The robot system according to claim 27, wherein the stored instructions further include instructions for providing an electrical signal for presenting a second visual indicator corresponding to the surgical tool on the viewer.

29. The first visual indicator includes information identifying the first robotic arm coupled to the flexible scope, the robot system according to claim 1.

30. The first visual indicator is displayed on the viewer simultaneously with the display of the visual field of the surgical site, the robot system according to claim 1.

31. The first visual indicator is displayed around the display of the visual field of the surgical site, the robot system according to claim 30.

32. The robot system according to claim 30, wherein the first visual indicator is displayed on the display of the visual field of the surgical site.

33. The robot system according to claim 1, wherein the viewer is part of a surgeon console.

34. The surgeon console includes an input device, The stored instructions are receiving an input on the input device and determining that the flexible scope is operating in the specific mode according to the received input. The robot system according to claim 33 includes instructions for this.

35. The robot system according to claim 34, wherein the input device includes a foot pedal.

36. The robot system according to claim 1, wherein the first visual indicator includes a graphical representation of the flexible scope in a graphical projection.

37. The stored instructions include instructions for displaying one or more reference user interface elements according to a determination that the flexible scope is bent. The robot system according to claim 36.

38. The stored instructions include instructions for aborting the display of the one or more reference user interface elements according to a determination that the flexible scope is straight. The robot system according to claim 37.

39. The stored instructions include instructions for displaying one or more reference user interface elements according to a determination of the degree of insertion of the flexible scope. The robot system according to claim 36.

40. A robot system, a first robotic arm coupled to a flexible scope, a viewer for displaying a visual field of a surgical site derived from the flexible scope, one or more processors, a memory configured to store instructions executed by the one or more processors, wherein the stored instructions are operating the flexible scope and providing an electrical signal for presenting a first visual indicator on the viewer, wherein the first visual indicator indicates at least one of a position or orientation of the flexible scope relative to a reference position. A memory including instructions for this. A robot system comprising.

41. An electronic device that communicates with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the electronic device comprising: one or more processors; a memory configured to store instructions executed by the one or more processors, the stored instructions comprising: operating the flexible scope in a particular mode of a plurality of modes; providing an electrical signal for presenting a first visual indicator corresponding to each mode on the viewer according to a determination that the flexible scope is operating in the particular mode of the plurality of modes, the first visual indicator indicating at least one of a position or orientation of the flexible scope relative to a reference position. A memory comprising instructions for providing. An electronic device comprising: **Claim 42** An electronic device that communicates with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the electronic device comprising: one or more processors; a memory configured to store instructions executed by the one or more processors, the stored instructions comprising: operating the flexible scope; providing an electrical signal for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of a position or orientation of the flexible scope relative to a reference position. A memory comprising instructions for providing. An electronic device comprising: **Claim 43** A computer-readable storage medium storing instructions executed by one or more processors that communicate with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the stored instructions comprising: operating the flexible scope in a particular mode of a plurality of modes; A computer-readable storage medium including instructions for providing an electrical signal for presenting, on a viewer, a first visual indicator corresponding to each of the modes, in accordance with a determination that the flexible scope is operating in the particular mode of the plurality of modes, the first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position. Claim 44 A computer-readable storage medium storing instructions executable by one or more processors communicating with a robotic system having a first robotic arm coupled to a flexible scope and a viewer for displaying a view of a surgical site derived from the flexible scope, the stored instructions including operating the flexible scope; providing an electrical signal for presenting, on the viewer, a first visual indicator indicating at least one of a position or an orientation of the flexible scope relative to a reference position.

Images