Dynamic flexible endoscope driver and method of use thereof

Abstract

A surgical robotic system may include a first robotic arm coupled to a flexible sight glass and a viewer for displaying a field of view of a surgical site derived from the flexible sight glass. The surgical robotic system may be configured to operate the flexible endoscope in a particular mode of a plurality of modes. In accordance with a determination that the flexible endoscope is operating in the particular mode, the surgical robotic system may provide an electrical signal for presenting a first visual indicator on the viewer corresponding to the respective mode.

Description

Technical Field

[0001] The systems and methods disclosed herein relate to devices and methods for indicating the position or orientation of a surgical tool, and more particularly to a surgical robotic system for indicating the position or orientation of a flexible surgical tool. Background Art

[0002] The robotically enabled medical system is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy and non-invasive procedures such as endoscopy (eg, bronchoscopy, ureteroscopy, gastroscopy, etc.).

[0003] Such robotic medical systems may include a robotic arm configured to control the movement of a surgical tool during a given medical procedure. To achieve a desired pose of the surgical tool, the robotic arm may be placed in a specific pose during teleoperation. Some robotic-enabled medical systems may include an arm support (e.g., a rod) that is connected to a corresponding base of the robotic arm and supports the robotic arm. Summary of the invention

[0004] One or more instruments may be coupled to one or more robotic arms of a robotic medical system (e.g., a surgical robotic system) for a medical procedure. For example, an instrument may be coupled to a robotic arm as a starting instrument for performing a procedure or as a replacement instrument in the middle of a procedure.

[0005] In a robotic system that includes multiple robotic arms, at least one robotic arm may be connected to a camera or scope that provides a surgical field of view. The scope is an important tool for laparoscopic or endoscopic surgery. The control methods of a straight and rigid robotic scope may be limited. For example, the scope can be inserted, removed, translated left and right, and rotated. However, in a robotic system that includes multiple robotic arms, a rigid scope may pose challenges. For example, the control methods of a rigid scope may be limited, which may lead to increased collisions between robotic arms. In addition, the size of a rigid scope may be large, which may crowd the surgeon's workspace or further limit the area where surgical instruments and ports may be located on the patient.

[0006] In order to alleviate these challenges, the robot arm can be connected to a flexible scope (e.g., a flexible laparoscope, a flexible endoscope, etc.) including a camera. The flexible scope introduces additional degrees of freedom, thereby allowing the surgeon to control the camera connected to the flexible scope in more ways than the way available when using a rigid scope. For example, in addition to the insertion, removal, translation and rotation operations that can be achieved using a rigid scope, the user can also move and rotate at least a portion (e.g., distal portion) of the flexible scope (e.g., articulation, pivoting, flexing, bending, etc.). Therefore, the user can view the target anatomical structure from a perspective (e.g., angle) that cannot be achieved using a rigid scope. In some cases, the flexible scope may have a volume that is less than the volume of the rigid scope, thereby making room in the surgeon's workspace. However, operating a flexible scope may bring new challenges. For example, the user may not be able to easily determine the shape of the flexible scope, or the direction and / or degree of movement or rotation of the scope, or how the scope recovers to a straight configuration from a curved configuration.

[0007] Therefore, there is a need for a robotic medical system that can facilitate the user to visualize the position or orientation of a flexible scope in conjunction with the surgical field of view.

[0008] As disclosed herein, a robotic medical system (e.g., a surgical robotic system) may include a robotic arm coupled to a flexible endoscope. The robotic medical system also includes a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. For example, the field of view of the surgical site may correspond to the field of view of a camera coupled to the flexible endoscope (or may be determined based on the field of view of the camera). A user interface displayed on the viewer may be configured to include an indicator indicating a position and / or orientation of the flexible endoscope relative to a reference position.

[0009] As disclosed herein, a flexible scope can be inserted into a patient through a port, and a reference position can be determined based on (eg, corresponding to) a position of the port.

[0010] As disclosed herein, the robotic medical system is configured to operate a flexible endoscope in multiple modes, such as an articulation mode, an orbital mode, and an automatic insertion and / or retraction mode (also referred to as an automatic insertion-retraction mode).

[0011] As disclosed herein, the robotic medical system can be aware of a camera field of view (e.g., by storing information indicating the camera field of view). In some embodiments, the field of view is determined based on an image provided by a camera. In some embodiments, the field of view is determined based on the position and orientation of the camera (e.g., a predefined volume of space in front of the camera).

[0012] As disclosed herein, the robotic medical system is configured to allow movement of a robotic arm. In some embodiments, the robotic medical system may cause mechanical movement of the robotic arm. In some embodiments, the robotic medical system may allow manual movement of the robotic arm.

[0013] As disclosed herein, the robotic medical system is configured to determine the position and / or orientation of the flexible scope. In some embodiments, the robotic medical system determines that the position or orientation of the scope has changed, and updates an indicator to indicate the changed position or orientation of the scope. In some embodiments, the robotic medical system updates the indicator in real time (e.g., as the scope moves) to indicate the changed position and orientation of the scope.

[0014] Therefore, the systems and / or methods disclosed herein advantageously improve the operation of a robotic medical system during a surgical procedure. For example, the use of a flexible scope enables a user to view a target anatomical structure from a perspective that may not be possible using a rigid scope. The user interface displayed on the viewer can be configured to include an indicator of the position or orientation of the flexible scope, which enhances the user experience because the user can quickly determine the position and / or orientation of the flexible scope. The ability to determine the position or orientation of the scope can also improve the safety of the surgical procedure.

[0015] The systems, methods and devices of the disclosure each have several innovative aspects, no single one of which is solely responsible for the desirable attributes disclosed herein.

[0016] According to some embodiments of the present disclosure, a robotic system includes a first robotic arm coupled to a flexible endoscope. The robotic system also includes a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The robotic system also includes one or more processors, and a memory storing instructions for execution by the one or more processors. The stored instructions include instructions for the following operations: operating the flexible endoscope in a specific mode among multiple modes. The stored instructions include instructions for the following operations: based on determining that the flexible endoscope is operating in a specific mode among the multiple modes, providing an electrical signal for presenting a first visual indicator corresponding to the corresponding mode on the viewer. The first visual indicator indicates at least one of the following: the position or orientation of the flexible endoscope relative to a reference position.

[0017] In some embodiments, the flexible scope is inserted into the patient through a first port, and the reference position is determined based on the position of the first port.

[0018] In some embodiments, when the robotic system is operating the flexible endoscope in a first mode of the multiple modes, the robotic system causes the distal portion of the flexible endoscope to move from a first position or a first orientation to a second position different from the first position or a second orientation different from the first orientation.

[0019] In some embodiments, the stored instructions include instructions for updating the first visual indicator to indicate the second position or the second orientation of the distal portion of the flexible scope relative to the reference position.

[0020] In some embodiments, the first visual indicator includes one or more reference axes and a graphical element positioned relative to the one or more reference axes to identify at least one of a direction or extent of movement.

[0021] In some embodiments, the stored instructions also include instructions for: based on determining that the distal portion of the flexible endoscope has performed at least one of the following operations: (i) moving from the first position to the second position or (ii) moving from the first orientation to the second orientation, providing an electrical signal for presenting an updated graphical element that indicates the change in the position or orientation of the flexible endoscope associated with the movement.

[0022] In some embodiments, the stored instructions further include instructions for determining a shape of the flexible scope and providing an image corresponding to the determined shape.

[0023] In some embodiments, the stored instructions further include instructions for: based on determining that the shape of the flexible scope has changed from a first shape to a second shape, providing an electrical signal for presenting an updated image indicative of the changed shape.

[0024] In some embodiments, the image includes two or more links corresponding to two or more links of the flexible scope.

[0025] In some embodiments, the first visual indicator includes a plurality of graphical elements. The plurality of graphical elements include a first graphical element and a second graphical element different from the first graphical element. The first graphical element represents a first component of movement of the distal end of the flexible scope along a first direction. The second graphical element represents a second component of movement of the distal end of the flexible scope along a second direction, wherein the first direction is different from the second direction.

[0026] In some embodiments, the stored instructions include instructions for the following operations: based on determining that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude, providing an electrical signal for updating the position of the first graphical element. The stored instructions also include instructions for the following operations: based on determining that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude, providing an electrical signal for updating the position of the second graphical element.

[0027] In some embodiments, the stored instructions include instructions for the following operations: based on determining that the movement of the flexible scope in the first direction has changed from a first magnitude to a second magnitude, providing an electrical signal for updating the length of the first graphical element. The stored instructions also include instructions for the following operations: based on determining that the movement of the flexible scope in the second direction has changed from a third magnitude to a fourth magnitude, providing an electrical signal for updating the length of the second graphical element.

[0028] In some embodiments, the field of view of the surgical site is obtained from a camera coupled to the flexible scope.The stored instructions also include instructions for expanding the first visual indicator based on determining that the camera is active.

[0029] In some embodiments, when the robotic system is operating the flexible scope in a second mode of the plurality of modes, the robotic system causes movement of the flexible scope around a target point in space.

[0030] In some embodiments, the first visual indicator includes a first graphical element representing the target point.

[0031] In some embodiments, the stored instructions also include instructions for providing an electrical signal for superimposing the first graphical element on a displayed field of view.

[0032] In some embodiments, the first visual indicator includes a second graphical element representing a line of sight from the target point to a port of the flexible scope.

[0033] In some embodiments, the first graphical element has a first size. The stored instructions also include instructions for adjusting the size of the first graphical element from the first size to a second size based on a change in the depth of insertion of the flexible scope into the surgical site.

[0034] In some embodiments, the first visual indicator includes a second graphical element representing the range of orbital motion of the flexible scope.

[0035] In some embodiments, the first visual indicator includes a third graphical element representing the position of the flexible scope.

[0036] In some embodiments, the stored instructions also include instructions for providing electrical signals for superimposing the first graphical element, the second graphical element, and the third graphical element on a displayed field of view.

[0037] In some embodiments, the first visual indicator includes one or more axes and a graphical element indicating a direction of insertion of the flexible scope relative to the one or more axes.

[0038] In some embodiments, when the robotic system is operating the flexible scope in a third mode of the plurality of modes, the robotic system automatically inserts and / or retracts the flexible scope.

[0039] In some embodiments, the stored instructions further include instructions for: determining a shape of the flexible scope during activation of the third mode; and providing an image corresponding to the determined shape.

[0040] In some embodiments, the stored instructions further include instructions for updating the image when the flexible scope is automatically inserted or retracted.

[0041] In some embodiments, the first visual indicator includes a graphical element indicating at least one of: an extent of insertion, an extent of retraction, or articulation of the flexible scope.

[0042] In some embodiments, the robotic system includes a second robotic arm coupled to the surgical tool.

[0043] In some embodiments, the stored instructions include instructions for providing an electrical signal for presenting a second visual indicator corresponding to the surgical tool on the viewer.

[0044] In some embodiments, the first visual indicator includes information identifying the first robotic arm coupled to the flexible scope.

[0045] In some embodiments, the first visual indicator is displayed simultaneously with the display of the field of view of the surgical site on the viewer.

[0046] In some embodiments, the first visual indicator is displayed around the display of the field of view of the surgical site.

[0047] In some embodiments, the first visual indicator is displayed on the display of the field of view of the surgical site.

[0048] In some embodiments, the viewer is part of the surgeon's console.

[0049] In some embodiments, the surgeon console includes an input device. The stored instructions include instructions for receiving input on the input device and determining that the flexible scope is operating in the particular mode based on the received input.

[0050] In some embodiments, the input device comprises a foot switch.

[0051] According to some embodiments of the present disclosure, a robotic system includes a first robotic arm coupled to a flexible endoscope. The robotic system includes a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The robotic system also includes one or more processors, and a memory storing instructions for execution by the one or more processors. The stored instructions include instructions for the following operations: operating the flexible endoscope, and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of the following: a position or orientation of the flexible endoscope relative to a reference position.

[0052] According to some embodiments, an electronic device in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The electronic device includes one or more processors and a memory. The memory stores instructions for execution by the one or more processors. The stored instructions include instructions for the following operations: operating the flexible endoscope in a specific mode among multiple modes. The stored instructions also include instructions for the following operations: based on determining that the flexible endoscope is operating in a specific mode among the multiple modes, providing an electrical signal for presenting a first visual indicator corresponding to the corresponding mode on the viewer. The first visual indicator indicates at least one of the following: the position or orientation of the flexible endoscope relative to a reference position.

[0053] According to some embodiments, an electronic device in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The electronic device includes one or more processors and a memory. The memory stores instructions for execution by the one or more processors. The stored instructions include instructions for the following operations: operating the flexible endoscope; and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of the following: a position or orientation of the flexible endoscope relative to a reference position.

[0054] According to some embodiments, a computer-readable storage medium stores instructions for execution by one or more processors in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The stored instructions include instructions for the following operations: operating the flexible endoscope in a specific mode among a plurality of modes. The stored instructions include instructions for the following operations: based on determining that the flexible endoscope is operating in a specific mode among the plurality of modes, providing an electrical signal for presenting a first visual indicator corresponding to the corresponding mode on the viewer. The first visual indicator indicates at least one of the following: a position or orientation of the flexible endoscope relative to a reference position.

[0055] According to some embodiments, a computer-readable storage medium stores instructions for execution by one or more processors in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The stored instructions include instructions for: operating the flexible endoscope in a specific mode among a plurality of modes. The stored instructions include instructions for: operating the flexible endoscope; and providing an electrical signal for presenting a first visual indicator on the viewer. The first visual indicator indicates at least one of: a position or orientation of the flexible endoscope relative to a reference position.

[0056] It should be noted that the various embodiments described above may be combined with any other embodiments described herein. The features and advantages described in the specification are not all inclusive, and in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, the specification, and the claims. In addition, it should be noted that the language used in the specification is primarily selected for readability and instructional purposes, and may not be selected to describe or limit the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.

[0058] Figure 1 Embodiments of a cart-based robotic system arranged for use in diagnostic and / or therapeutic bronchoscopy procedures are illustrated.

[0059] Figure 2 Depicted Figure 1 Other aspects of the robotic system.

[0060] Figure 3 An example of a device arranged for ureteroscopy is shown. Figure 1 Implementation scheme of a robotic system.

[0061] Figure 4 Illustrated is arranged for use in a vascular procedure Figure 1 Implementation scheme of a robotic system.

[0062] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is illustrated.

[0063] Figure 6 Provided Figure 5 Alternative view of the robotic system.

[0064] Figure 7 An example system configured to stow a robotic arm is illustrated.

[0065] Figure 8 An embodiment of a table-based robotic system configured for use in a ureteroscopy procedure is illustrated.

[0066] Fig. 9 An embodiment of a table-based robotic system configured for laparoscopic procedures is illustrated.

[0067] Fig.10 Illustrated Figures 5 to 9 Embodiment of a stage-based robotic system with pitch or tilt adjustment.

[0068] Fig.11 Provided Figures 5 to 10 Detailed illustration of the interface between the stage and the column of a stage-based robotic system.

[0069] Fig.12 An alternative embodiment of a table-based robotic system is illustrated.

[0070] Fig.13 Illustrated Fig.12 End view of a stage-based robotic system.

[0071] Fig.14 An end view of a table-based robotic system with a robotic arm attached thereto is illustrated.

[0072] Fig.15 An exemplary instrument driver is illustrated.

[0073] Fig.16 An exemplary medical instrument having a pair of instrument drivers is illustrated.

[0074] Fig.17 An alternative design of an instrument driver and instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0075] Fig.18 An instrument having an instrument-based insertion architecture is illustrated.

[0076] Fig.19 An exemplary controller is illustrated.

[0077] Fig. 20 Depicts a block diagram illustrating an example embodiment of estimating Figures 1 to 10 The position of one or more elements of the robotic system (such as Figures 16 to 18 A positioning system that determines the position of the device.

[0078] Fig.21 An exemplary robotic system according to some embodiments is illustrated.

[0079] Fig. 22 Another view of an exemplary robotic system is illustrated in accordance with some embodiments.

[0080] Fig.23 Components of a robotic medical system according to some embodiments are illustrated.

[0081] Fig.24A , FIG. 24B to FIG. 24C Different views of an exemplary robotic arm are illustrated in accordance with some embodiments.

[0082] Fig.25 A perspective view of a robotic medical system is illustrated, according to some embodiments.

[0083] Fig.26A and Fig.26B Illustrated is a camera field of view in three-dimensional space according to some embodiments.

[0084] Fig.27A , Fig.27B , Fig.27C and Fig.27D A flexible scope according to some embodiments is illustrated.

[0085] Fig.28A , Fig.28B , Fig.28C and Fig.28D An exemplary user interface displayed on a display device when the flexible scope is operating in an articulation drive mode according to some embodiments is illustrated.

[0086] Fig.29A , Fig.29B , Fig.29C and Fig.29D An exemplary user interface displayed on a display device when the flexible scope is operating in a track drive mode according to some embodiments is illustrated.

[0087] Fig. 30A , Fig. 30B , Fig. 30C , Fig.30D , Fig.30E and Fig.30F An exemplary user interface displayed on a display device when a flexible scope is operating in an automatic insertion and retraction mode according to some embodiments is illustrated.

[0088] Fig.31A Different configurations of flexible scopes according to some embodiments are illustrated.

[0089] FIG. 31B to FIG. 31F Exemplary user interface elements displayed on a display device to indicate a status of a flexible scope according to some embodiments are illustrated.

[0090] Fig.32A , Fig.32B , Fig.32C , Fig.32D , Fig.32E and Fig.32F A flow chart illustrating a method performed by one or more processors of a robotic system according to some embodiments.

[0091] Fig.33 is a schematic diagram illustrating electronic components of a robotic medical system according to some embodiments. DETAILED DESCRIPTION

[0092] 1. Overview .

[0093] Various aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. In the endoscopy procedure, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0094] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm motions and positions. Additionally, the system may provide the physician with the ability to perform procedures with improved ease of use such that one or more of the system's instruments may be controlled by a single user.

[0095] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other embodiments of the disclosed concepts are possible, and various advantages can be achieved using the disclosed embodiments. Titles are included herein for reference and to help locate the various sections. These titles are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout the specification.

[0096] A. Robotic System-Cart .

[0097] Robotically enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1 An embodiment of a cart-based, robotically enabled system 10 arranged for use in a diagnostic and / or therapeutic bronchoscopy procedure is illustrated. During a bronchoscopy procedure, the system 10 may include a cart 11 having one or more robotic arms 12 to deliver a medical instrument such as a steerable endoscope 13 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., the oral cavity of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned proximate to the patient's upper torso to provide access to the entry point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the access point. When performing a gastrointestinal (GI) procedure using a gastroscope (a specialized endoscope for GI procedures), the bronchoscope may also be utilized. Figure 1 The arrangement in . Figure 2 An example embodiment of a cart is depicted in greater detail.

[0098] Continue to refer Figure 1, once the cart 11 is correctly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or in a combination thereof. As shown, the steerable endoscope 13 may include at least two telescopic parts, such as an inner guide portion and an outer sheath portion, each of which is coupled to a separate instrument driver from a set of instrument drivers 28, each of which is coupled to the distal end of a separate robotic arm. This linear arrangement of the instrument driver 28 that facilitates coaxial alignment of the guide portion with the sheath portion produces a "virtual track" 29, which can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual tracks described herein are depicted using dashed lines in the accompanying drawings, and therefore the dashed lines do not depict any physical structure of the system. The translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical applications or physician preferences. For example, in bronchoscopy, the angle and position of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.

[0099] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.

[0100] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as, for example, a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove potential cancerous tissue. In some cases, diagnostic and therapeutic treatments can be delivered in separate procedures. In these cases, the endoscope 13 can also be used to deliver a benchmark to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0101] The system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that may be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 30 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location to be out of the way during a procedure.

[0102] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions may control the entire system or a subsystem thereof. For example, when executed by a processor of a computer system, the instructions may cause components of the robotic system to actuate associated brackets and arm mounts, actuate a robotic arm, and control a medical device. For example, in response to receiving a control signal, a motor in a joint of a robotic arm may position the arm into a particular posture.

[0103] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to the system that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, the irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.

[0104] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more mobile cart 11.

[0105] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0106] The tower 30 may also include a console 31, in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the system 10 is typically designed to provide both preoperative information and real-time information for robotic control and surgery, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the health or vital signs of the patient and the operation of the system, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, the console 30 is housed in a body separate from the tower 30.

[0107] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions may be coupled in separate wiring and connections. For example, while power may be provided to the cart via a single cable, support for control, optics, fluidics, and / or navigation may also be provided via separate cables.

[0108] Figure 2 Provided by Figure 1 Detailed illustration of an embodiment of a cart for a cart-based robotic-enabled system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2 The cradle 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The cradle 17 also includes a cradle interface 19 that allows the cradle 17 to translate vertically along the column 14.

[0109] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 contains a vertical translation interface to position and maintain the bracket at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet a variety of table heights, patient sizes, and physician preferences. Similarly, the separately configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0110] In some embodiments, the slot 20 may be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the internal cavity of the column 14 and the vertical translation interface when the carriage 17 is vertically translated. The slot cover can be deployed by a pair of spring reels positioned near the vertical top and bottom of the slot 20. The cover is coiled in the reel before deployment, and only extends and retracts from the coiled state of the cover as the carriage 17 vertically translates up and down. When the carriage 17 translates toward the reel, the spring loading of the reel provides the force to retract the cover into the reel, while also maintaining a tight seal when the carriage 17 translates away from the reel. The cover can be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure appropriate extension and retraction of the cover when the carriage 17 translates.

[0111] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).

[0112] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints and therefore provides seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different linkage mechanism positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0113] The cart base 15 balances the weight of the column 14, bracket 17, and arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart to be moved and / or fixed. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart to be easily moved around the room before a procedure. After reaching the appropriate position, the casters 25 can be secured using wheel locks to keep the cart 11 in place during the procedure.

[0114] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or a dual-purpose device, such as, for example, a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors and coordinate information from sensors and important patient statistics, such as respiration, heart rate and / or pulse. The console 16 may be positioned and tilted to allow the physician to approach the console from the side of the column 14 opposite to the bracket 17. From this position, the physician can observe the console 16, the robot arm 12 and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 for helping to manipulate and stabilize the cart 11.

[0115] Figure 3 An embodiment of a robotically enabled system 10 arranged for ureteroscopy is illustrated. In a ureteroscopy procedure, a cart 11 can be positioned to deliver a ureteroscope 32 (a procedure-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. In a ureteroscopy, it may be desirable to align the ureteroscope 32 directly with the patient's urethra to reduce friction and forces on sensitive anatomical structures in this area. As shown, the cart 11 can be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 can insert the ureteroscope 32 directly into the patient's lower abdomen through the urethra along a virtual track 33 from the foot of the table.

[0116] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.

[0117] Figure 4An embodiment of a robotically enabled system similarly arranged for vascular procedures is illustrated. In a vascular procedure, the system 10 may be configured so that the cart 11 can deliver a medical device 34 (such as a steerable catheter) to an access point in the femoral artery in the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's legs and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 may be guided and inserted by translating the instrument drive 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as the carotid and brachial arteries near the shoulder and wrist.

[0118] B. Robotic system - unit .

[0119] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment within the operating room by removing a cart, which allows for greater access to the patient. Figure 5 An embodiment of such a robotically enabled system arranged for use in a bronchoscopy procedure is illustrated. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.

[0120] Figure 6An alternative view of the system 36 without the patient and medical device for discussion purposes is provided. As shown, the column 37 may include one or more brackets 43 shown as annular in the system 36, on which one or more robot arms 39 may be based. The bracket 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robot arm 39 may be positioned to reach the patient. The bracket 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robot arm 39 to access multiple sides of the table 38, such as, for example, both sides of the patient. In an embodiment with multiple brackets, the brackets may be positioned individually on the column and may translate and / or rotate independently of the other brackets. Although the bracket 43 does not need to be around the column 37 or even circular, the annular shape as shown facilitates the rotation of the bracket 43 around the column 37 while maintaining structural balance. The rotation and translation of the bracket 43 allows the system to align medical devices such as endoscopes and laparoscopes to different entry points on the patient. In other embodiments (not shown), the system 36 may include a patient table or bed with an adjustable arm support in the form of a pole or track extending alongside the patient table or bed. One or more robotic arms 39 may be attached to the adjustable arm support (e.g., via a shoulder with an elbow joint) that may be vertically adjusted. By providing vertical adjustment, the robotic arms 39 are advantageously able to be compactly stowed beneath the patient table or bed and subsequently raised during a procedure.

[0121] The arm 39 may be mounted on the carriage via a set of arm mounts 45 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 so that when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned on the same side of the table 38 (e.g., Figure 6 As shown), on the opposite side of the platform 38 (as shown Fig. 9 shown) or on an adjacent side of stage 38 (not shown).

[0122] The column 37 provides structural support for the stage 38 and provides a path for the vertical translation of the carriage. Internally, the column 37 can be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage based on the lead screw. The column 37 can also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0123] The base 46 has Figure 2The cart base 15 in the illustrated cart 11 functions similarly, accommodating heavier components to balance the table / bed 38, column 37, bracket 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. Casters deployed from the bottom of the table base 46 may extend in opposite directions on both sides of the base 46 and retract when the system 36 needs to be moved.

[0124] continue Figure 6 , the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in the previously disclosed embodiments, the tower can provide various support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower can also be movable to be positioned away from the patient, thereby improving the physician's access and eliminating clutter in the operating room. In addition, placing the components in the tower allows more storage space in the table base for potential stowage of the robotic arm. The tower may also include a main controller or console that provides a user interface such as a keyboard and / or suspension for user input, and a display screen (or touch screen) for preoperative and intraoperative information such as real-time imaging, navigation and tracking information. In some embodiments, the tower may also include a holder for a gas tank to be used for insufflation.

[0125] In some embodiments, the table base can fold up and store the robotic arm when not in use. Figure 7 A system 47 for stowing a robotic arm in an embodiment of a table-based system is illustrated. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the arm 50 around a column 53, and closed to stow the carriage, the arm mount, and the arm to protect them when not in use. The base cover 52 can be sealed along the edges of its opening with a membrane 54 to prevent dust and fluids from entering when closed.

[0126] Figure 8An embodiment of a robotically enabled table-based system configured for a ureteroscopy procedure is illustrated. During ureteroscopy, the table 38 may include a rotating portion 55 for positioning the patient at an angle to the column 37 and the table base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) so as to position the bottom portion of the rotating portion 55 away from the column 37. For example, the pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the bracket 35 (not shown) about the column 37, the robotic arm 39 can insert the ureteroscope 56 directly into the patient's groin area along the virtual track 57 to reach the urethra. During ureteroscopy, stirrups 58 may also be fixed to the rotating portion 55 of the table 38 to support the position of the patient's legs during the procedure and allow full access to the patient's groin area.

[0127] In a laparoscopic procedure, minimally invasive instruments may be inserted into the patient's anatomical structure through one or more small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instrument includes an elongated rigid member, such as a shaft, for accessing the anatomical structure within the patient. After the patient's abdominal cavity is inflated, the instrument may be guided to perform a surgical or medical task, such as grasping, cutting, ablating, suturing, etc. In some embodiments, the instrument may include a scope, such as a laparoscope. Fig. 9 Embodiments of a robotically enabled table-based system configured for laparoscopic procedures are illustrated. Fig. 9 As shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 so that the arm mounts 45 can be used to position instruments 59 through minimal incisions on both sides of the patient to reach his / her abdominal cavity.

[0128] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. Fig.10 Embodiments of a robotically enabled medical system with pitch or tilt adjustment are illustrated. Fig.10 As shown, system 36 can accommodate the tilt of table 38 to position one portion of the table at a greater distance from the floor than another portion. Additionally, arm mount 45 can be rotated to match the tilt so that arm 39 maintains the same planar relationship with table 38. To accommodate steeper angles, column 37 can also include telescoping portion 60 that allows vertical extension of column 37 to prevent table 38 from contacting the floor or colliding with base 46.

[0129] Fig.11A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-to-table interface, each axis being actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment in one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2. In some embodiments, a ball joint can be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0130] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position (i.e., positioning the patient's lower abdomen higher from the floor than the patient's lower abdomen) for lower abdominal surgery. The Trendelenburg position causes the patient's internal organs to glide toward his / her upper abdomen by gravity, thereby clearing the abdominal cavity to allow minimally invasive tools to enter and perform lower abdominal surgical or medical procedures, such as laparoscopic prostatectomy.

[0131] Fig.12 and Fig.13 An isometric view and an end view of an alternative embodiment of a table-based surgical robotic system 100 are illustrated. The surgical robotic system 100 includes a table 101 that can be configured to support one or more robotic arms (see, e.g., Fig.14 ) is provided with one or more adjustable arm supports 105 of the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports may be provided on opposite sides of the table 101. The adjustable arm support 105 may be configured so that it can be moved relative to the table 101 to adjust and / or change the orientation of the adjustable arm support 105 and / or any robotic arm mounted thereto relative to the table 101. For example, the adjustable arm support 105 may be adjusted in one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides high flexibility to the system 100, including the ability to easily stow the one or more adjustable arm supports 105 and any robotic arm attached thereto below the table 101. The adjustable arm support 105 may be raised from a stowed orientation to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from the stowed position to a position above the upper surface of the table 101 .

[0132] The adjustable arm support 105 may provide several degrees of freedom, including lift, lateral translation, tilt, etc. Fig.12 and Fig.13 In the illustrated embodiment, the arm support 105 is configured to have four degrees of freedom. Fig.12105 . The first degree of freedom allows the adjustable arm support 105 to be adjusted in the z-direction ("Z lift"). For example, the adjustable arm support 105 may include a bracket 109 that is configured to move upward or downward along or relative to the column 102 of the support table 101. The second degree of freedom allows the adjustable arm support 105 to be tilted. For example, the adjustable arm support 105 may include a swivel joint that allows the adjustable arm support 105 to be aligned with the bed in a Trendelenburg position. The third degree of freedom allows the adjustable arm support 105 to "pivot up," which can be used to adjust the distance between one side of the table 101 and the adjustable arm support 105. The fourth degree of freedom allows the adjustable arm support 105 to translate along the longitudinal length of the table.

[0133] Fig.12 and Fig.13 The surgical robot system 100 in the embodiment of the present invention may include a table supported by a column 102 mounted to a base 103. The base 103 and the column 102 support the table 101 relative to the support surface. The floor axis 131 and the support axis 133 are located at Fig.13 Shown in.

[0134] An adjustable arm support 105 can be mounted to column 102. In other embodiments, arm support 105 can be mounted to table 101 or base 103. Adjustable arm support 105 can include bracket 109, rod or track connector 111, and rod or track 107. In some embodiments, one or more robotic arms mounted to track 107 can translate and move relative to each other.

[0135] The bracket 109 may be attached to the column 102 via a first joint 113 that allows the bracket 109 to move relative to the column 102 (e.g., such as up and down along a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117 that provides a third degree of freedom ("upward pivot") to the adjustable arm support 105. Additional joints 119 may be provided (in Fig.13 ), the additional joint mechanically constrains the third joint 117 to maintain the orientation of the track 107 when the track connector 111 rotates about the third axis 127. The adjustable arm support 105 may include a fourth joint 121, which may provide the adjustable arm support 105 with a fourth degree of freedom (translation) along a fourth axis 129.

[0136] Fig.14An end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101 is illustrated. A first robotic arm 142A is attached to a rod or track 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the track 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the track 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0137] In some embodiments, one or more of the robotic arms 142A, 142B include an arm with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B may include eight degrees of freedom, including an insertion axis (including 1 degree of freedom for insertion), a wrist (including 3 degrees of freedom for wrist pitch, yaw, and roll), an elbow (including 1 degree of freedom for elbow pitch), a shoulder (including 2 degrees of freedom for shoulder pitch and yaw), and a base 144A, 144B (including 1 degree of freedom for translation). In some embodiments, the insertion degrees of freedom may be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0138] C. Device Drivers and Interfaces .

[0139] The end effector of the system's robotic arm includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device that may not contain any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.

[0140] Fig.15An example instrument driver is illustrated. The instrument driver 62, positioned at the distal end of the robotic arm, includes one or more drive units 63 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., Fig.15 In operation, the control circuit 68 will receive the control signal, transmit the motor signal to the motor 66, compare the resulting motor speed measured by the encoder 67 to the desired speed, and modulate the motor signal to generate the desired torque.

[0141] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, an exemplary sterile adapter may include a series of rotational inputs and rotational outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover composed of a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, medical devices can dock with patients in areas that require sterilization (i.e., sterile areas).

[0142] D.Medical devices .

[0143] Fig.16An example medical device with a paired instrument driver is illustrated. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its intended design for manual interaction by a physician, may typically include a rotatable drive input 73 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 extending through a distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 may share an axis of rotation with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include splines that are designed to mate with a receptacle on the drive input 73.

[0144] The elongated shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 71 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from a joined wrist formed by a connecting fork having at least one degree of freedom and a surgical tool or medical instrument (such as, for example, a grasper or scissors), which can be actuated based on the force from the tendon when the drive input rotates in response to the torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft can include a manipulable or controllable curved segment that can be articulated and bent based on the torque received from the drive output 74 of the instrument driver 75.

[0145] The torque from the instrument driver 75 is transmitted to the elongated shaft 71 down the shaft 71 using tendons. These individual tendons (such as pull wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are guided downward along one or more pull cavities of the elongated shaft 71 and anchored at the distal portion of the elongated shaft 71, or in the wrist at the distal portion of the elongated shaft. During surgical procedures such as laparoscopic, endoscopic, or hybrid procedures, these tendons can be connected to a distally mounted end effector, such as a wrist, a grasper, or a pair of scissors. In such an arrangement, the torque applied to the drive input 73 transmits tension to the tendons, causing the end effector to actuate in some manner. In some embodiments, during a surgical procedure, the tendons can cause the joint to rotate around an axis, causing the end effector to move in one direction or another. Alternatively, the tendons can be connected to one or more jaws of a grasper at the distal end of the elongated shaft 71, wherein the tension from the tendons causes the grasper to close.

[0146] In endoscopy, the tendon may be coupled to a bending or articulation segment located along the elongated shaft 71 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the bending segment, torque applied to the drive input 73 will be transmitted downward along the tendon, causing the softer bending segment (sometimes referred to as the segment or region capable of articulation) to bend or articulate. Along the non-bending segment, it may be advantageous to spiral or swirl a separate traction lumen that guides a separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the traction wire. For specific purposes, the angle of the spiral and / or the spacing therebetween may be varied or designed, wherein a tighter spiral presents less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but also presents limited bending. In another case, the traction lumen may be guided parallel to the longitudinal axis of the elongated shaft 71 to allow controlled articulation in the desired bending or articulation segment.

[0147] In endoscopy, the elongated shaft 71 houses a plurality of components to assist in robotic procedures. The shaft may include a working channel for deploying surgical tools (or medical devices), irrigation components, and / or suction components to the operating area at the distal end of the shaft 71. The shaft 71 may also house wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, which may include an optical camera. The shaft 71 may also house an optical fiber to carry light from a proximally located light source (such as a light emitting diode) to the distal end of the shaft.

[0148] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, irrigation, and suction to the surgical site. The distal tip may also include a port for a camera (such as a fiberscope or a digital camera) to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source for illuminating the anatomical space when the camera is used.

[0149] exist Fig.16 In the example of , the axis of the drive shaft, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Rolling the elongated shaft 71 along its axis while keeping the drive input 73 stationary can cause undesirable tangling of the tendons as they extend out of the drive input 73 and into the traction cavity within the elongated shaft 71. The resulting tangling of such tendons can disrupt any control algorithm designed to predict the movement of the flexible elongated shaft during an endoscopic procedure.

[0150] Fig.17 An alternative design of an instrument driver and an instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units, wherein the drive outputs 81 of these drive units are aligned in parallel at the end of a robot arm 82. The drive unit and its corresponding drive output 81 are contained in a rotating assembly 83 of the instrument driver 80, which is driven by a drive unit in the drive unit in the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along the circular bearing that connects the rotating assembly 83 to the non-rotating part 84 of the instrument driver. Electric power and control signals can be conveyed to the rotating assembly 83 from the non-rotating part 84 of the instrument driver 80 by electrical contact, and the power and control signals can be maintained by the rotation of the brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can be responsive to a separate drive unit integrated into the non-rotating part 84, and is therefore not parallel to other drive units. The rotating mechanism 83 allows the instrument driver 80 to rotate the drive unit and its corresponding drive output 81 around the instrument driver axis 85 as a single unit.

[0151] Similar to previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown as having a transparent outer surface for discussion purposes) that includes a plurality of drive inputs 89 (such as sockets, pulleys, and spools) configured to receive drive outputs 81 in the instrument driver 80. Unlike previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 89, rather than being substantially parallel to the axis of the drive inputs 89. Fig.16 as orthogonal in the design.

[0152] When coupled to the rotation assembly 83 of the instrument driver 80, the medical device 86, including the instrument base 87 and the instrument shaft 88, rotates in combination with the rotation assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is located at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Therefore, the rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. In addition, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not tangle during rotation. Therefore, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows the shaft to rotate without tangling any control tendons.

[0153] Fig.18 An instrument with an instrument-based insertion architecture according to some embodiments is illustrated. The instrument 150 can be coupled to any of the instrument drivers discussed above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 coupled to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The groove 158 is configured to receive one or more wires or cables 180 passing through the groove. Therefore, one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cable 180 may also extend through the elongated shaft 152. The manipulation of the one or more cables 180 (e.g., via the instrument driver) causes the actuation of the end effector 162.

[0154] The instrument handle 170 (also referred to as the instrument base) may generally include an attachment interface 172 having one or more mechanical input portions 174, such as a socket, pulley, or spool, which are designed to reciprocately engage with one or more torque connectors on the attachment surface of the instrument driver.

[0155] In some embodiments, the instrument 150 includes a series of pulleys or cables that enable the elongated shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that accommodates instrument insertion, thereby minimizing reliance on a robotic arm to provide insertion of the instrument 150. In other embodiments, the robotic arm may be largely responsible for instrument insertion.

[0156] E. Controller .

[0157] Any of the robotic systems described herein may include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller may be coupled to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) such that manipulation of the controller causes corresponding manipulation of the instrument, e.g., via master-slave control.

[0158] Fig.19 1 is a perspective view of an embodiment of the controller 182. In this embodiment, the controller 182 comprises a hybrid controller that can have both impedance and admittance control. In other embodiments, the controller 182 can utilize only impedance or passive control. In other embodiments, the controller 182 can utilize only admittance control. By being a hybrid controller, the controller 182 can advantageously have lower perceived inertia when in use.

[0159] In the illustrated embodiment, the controller 182 is configured to allow manipulation of two medical devices and includes two handles 184. Each of the handles 184 is connected to a gimbal 186. Each gimbal 186 is connected to a positioning platform 188.

[0160] like Fig.19 As shown, each positioning platform 188 includes a SCARA arm (selective compliant assembly robot arm) 198 coupled to the column 194 via a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along the track 197) to allow each of the handles 184 to translate in the z direction, thereby providing a first degree of freedom. The SCARA arm 198 is configured to allow the handles 184 to move in the xy plane, thereby providing two additional degrees of freedom.

[0161] In some embodiments, one or more load sensors are positioned in the controller. For example, in some embodiments, a load sensor (not shown) is positioned in the body of each gimbal in the gimbal 186. By providing a load sensor, a portion of the controller 182 can be operated under admittance control, thereby advantageously reducing the perceived inertia of the controller when in use. In some embodiments, the positioning platform 188 is constructed for admittance control, and the gimbal 186 is constructed for impedance control. In other embodiments, the gimbal 186 is constructed for admittance control, and the positioning platform 188 is constructed for impedance control. Therefore, for some embodiments, the translational degree of freedom or the azimuth degree of freedom of the positioning platform 188 can rely on admittance control, while the rotational degree of freedom of the gimbal 186 relies on impedance control.

[0162] F. Navigation and Control .

[0163] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operator physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the position of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve information obtained only by radiation-based imaging modalities.

[0164] Fig. 20 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an example embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory in one or more of the components discussed above. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figures 1 to 4 The cart shown, Figures 5 to 14 The bed shown, etc.

[0165] like Fig. 20 As shown, the positioning system 90 may include a positioning module 95 that processes the input data 91-94 to generate position data 96 for the distal tip of the medical device. The position data 96 may be data or logic that represents the position and / or orientation of the distal end of the device relative to a reference system. The reference system may be a reference system relative to the patient's anatomy or a known object such as an EM field generator (see below for a discussion of EM field generators).

[0166] The various input data 91-94 are now described in more detail. Preoperative mapping can be accomplished using a collection of low-dose CT scans. The preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of cross-sectional views of the patient's internal anatomy. When analyzed as a whole, image-based models of anatomical cavities, spaces, and structures for the patient's anatomical structures (such as the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated from the CT images to form a three-dimensional volume of the patient's anatomical structure, which is referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suitable for bronchoscopy.

[0167] In some embodiments, the instrument can be equipped with a camera to provide visual data 92. The positioning module 95 can process the visual data to implement one or more vision-based position tracking. For example, preoperative model data can be used in conjunction with the visual data 92 to implement computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advanced through a working channel of an endoscope). For example, using the preoperative model data 91, the robotic system can generate a library of expected endoscopic images from the model based on the expected path of travel of the endoscope, each image connected to a location within the model. During surgery, the robotic system can reference the library to compare real-time images captured at a video camera (e.g., a camera at the distal end of an endoscope) with those in the image library to assist in positioning.

[0168] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera, and therefore the motion of the endoscope. Some features of the localization module 95 may identify circular geometric structures in the preoperative model data 91 that correspond to anatomical cavities and track changes in those geometric structures to determine which anatomical cavity is selected, as well as track relative rotational and / or translational motion of the camera. The use of topological maps may further enhance vision based algorithms or techniques.

[0169] Optical flow (another computer vision-based technique) can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, brightness, motion compensated encoding, stereo disparity measurements, etc. Through multiple iterations of multi-frame comparisons, the movement and position of the camera (and therefore the endoscope) can be determined.

[0170] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope in a global coordinate system that can be aligned to the patient's anatomical structure represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils in one or more positions and orientations embedded in a medical device (e.g., an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "aligned" to the patient's anatomical structure (e.g., a preoperative model) during surgery to determine a geometric transformation that aligns a single position in the coordinate system with a position in the preoperative model of the patient's anatomical structure. Once registered, an embedded EM tracker in one or more locations of a medical device (eg, the distal tip of an endoscope) can provide a real-time indication of the progress of the medical device through the patient's anatomy.

[0171] The robotic commands and kinematic data 94 may also be used by a positioning module 95 to provide orientation data 96 for the robotic system. The pitch and yaw of the device derived from the joint motion commands may be determined during preoperative calibration. In surgery, these calibration measurements may be used in conjunction with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations may be analyzed in conjunction with EM, vision, and / or topological modeling to estimate the position of the medical device within the network.

[0172] like Fig. 20 As shown, the positioning module 95 can use a variety of other input data. For example, although Fig. 20 Not shown, but an instrument utilizing shape sensing fibers may provide shape data that positioning module 95 may use to determine the position and shape of the instrument.

[0173] The localization module 95 may use the input data 91-94 in combination. In some cases, such a combination may use a probabilistic approach, where the localization module 95 assigns a confidence weight to a position determined from each of the input data 91-94. Thus, in situations where the EM data may be unreliable (such as where EM interference may be present), the confidence in the position determined by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematics data 94.

[0174] As discussed above, the robotic systems discussed herein may be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart may store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.) that, when executed, causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the position of an instrument in a global coordinate system, an anatomical map, etc.

[0175] 2. Dynamic flexible scope actuator and method of use thereof .

[0176] The present patent application discloses a robotic medical system including a robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope. The system can provide an indication of the position or orientation of the flexible endoscope relative to a reference position.

[0177] In some embodiments, the robotic medical system may provide an indication via a user interface or display device of the robotic medical system. For example, in some embodiments, the robotic medical system provides a visual indication of the position or orientation of the flexible scope (e.g., at the distal end of the scope).

[0178] In some embodiments, the robotic medical system is configured to operate the flexible endoscope in a particular mode among multiple modes (e.g., the robotic medical system may operate the flexible endoscope in a first mode among the multiple modes at a first time, and operate the flexible endoscope in a second mode among the multiple modes that is different from the first mode at a second time that is different from the first time).

[0179] In some embodiments, the robotic medical system is configured to determine the position and / or orientation of the flexible scope. In some embodiments, the robotic medical system determines that the position or orientation of the scope has changed, and updates the indicator to indicate the changed position or orientation of the scope. In some embodiments, the robotic medical system updates the indicator in real time (e.g., as the scope moves) to indicate the changed position and orientation of the scope.

[0180] A. Robotic system

[0181] Fig.21 An exemplary robotic medical system 200 according to some embodiments is illustrated. In some embodiments, the robotic medical system 200 is a robotic surgical system. Fig.21In the example of FIG. 1 , the robotic medical system 200 includes a patient support platform 202 (e.g., a patient platform, table, bed, etc.). The two ends along the length of the patient support platform 202 are referred to as the "head" and "legs", respectively. The two sides of the patient support platform 202 are referred to as the "left side" and the "right side", respectively. The patient support platform 202 includes a support member 204 (e.g., a rigid frame) for the patient support platform 202.

[0182] The robotic medical system 200 also includes a base 206 for supporting the robotic medical system 200. The base 206 includes wheels 208 that allow the robotic medical system 200 to be easily moved or repositioned in a physical environment. In some embodiments, the wheels 208 are omitted from the robotic medical system 200 or are retractable, and the base 206 can rest directly on the ground or floor. In some embodiments, the wheels 208 are replaced with feet.

[0183] The robotic medical system 200 includes one or more robotic arms 210. The robotic arms 210 may be configured to perform the tasks described above with reference to Figures 1 to 20 The robotic medical procedures described. Fig.21 Five robotic arms 210 are shown, but it should be understood that the robotic medical system 200 may include any number of robotic arms, including less than five, or six or more.

[0184] The robotic medical system 200 also includes one or more rods 220 (e.g., adjustable arm supports or adjustable rods) supporting the robotic arms 210. Each of the robotic arms 210 is supported on and movably coupled to the rod 220 via a corresponding base joint of the robotic arm. In some embodiments, and as Fig.12 As described in, the rod 220 can provide several degrees of freedom, including lift, lateral translation, tilt, etc. In some embodiments, each of the robotic arm 210 and / or the adjustable arm support 220 is also referred to as a corresponding kinematic chain.

[0185] Fig.21 Three robotic arms 210 are shown supported by a pole 220 located in the field of view of the figure. The remaining two robotic arms are supported by another pole located on another length of the patient support platform 202.

[0186] In some embodiments, the adjustable arm support 220 can be configured to provide a base position for one or more of the robotic arms 210 to perform a robotic medical procedure. The robotic arm 210 can be adjusted by translating the robotic arm 210 along the length of the underlying rod 220 of the robotic arm 210 and / or by adjusting the position and / or orientation of the robotic arm 210 via one or more joints and / or linkages (see, e.g., Fig.23), the robotic arm can be positioned relative to the patient support platform 202. In some embodiments, the pole position can be changed via manual manipulation, teleoperation, and / or power-assisted movement.

[0187] In some embodiments, the adjustable arm support 220 can translate along the length of the patient support platform 202. In some embodiments, translation of the rod 220 along the length of the patient support platform 202 causes one or more of the robotic arms 210 supported by the rod 220 to translate simultaneously with or relative to the rod. In some embodiments, the rod 220 can be translated while keeping one or more of the robotic arms stationary relative to the base 206 of the robotic medical system 200.

[0188] exist Fig.21 In the example of , the adjustable arm support 220 is positioned along the length of the patient support platform 202. In some embodiments, the adjustable arm support 220 can extend over part or all of the length of the patient support platform 202, and / or extend over part or all of the width of the patient support platform 202.

[0189] According to some embodiments, during a robotic medical procedure, one or more of the robotic arms 210 may also be configured to hold an instrument 212 (e.g., a robotically controlled medical instrument or tool, such as an endoscope and / or any other instrument that may be used during a surgical procedure (e.g., sensors, lighting instruments, cutting instruments, etc.)) and / or be coupled to one or more accessories, including one or more cannulas.

[0190] Fig. 22 Illustrated according to some embodiments Fig.21 200 . In this example, the robotic medical system 200 includes six robotic arms 210-1, 210-2, 210-3, 210-4, 210-5, and 210-6. The patient platform 202 is supported by a column 214 extending between the base 206 and the patient platform 202. In some embodiments, the patient platform 202 includes a tilt mechanism 216. The tilt mechanism 216 can be positioned between the column 214 and the patient platform 202 to allow the patient platform 202 to pivot, rotate, or tilt relative to the column 214. The tilt mechanism 216 can be configured to allow lateral tilting and / or longitudinal tilting of the patient platform 202. In some embodiments, the tilt mechanism 216 allows the patient platform 202 to be tilted laterally and longitudinally at the same time.

[0191] Fig. 22The patient platform 202 is shown in a non-tilted state or position. In some embodiments, the non-tilted state or position is the default position of the patient platform 202. In some embodiments, the default position of the patient platform 202 is as follows: Fig. 22 As illustrated, in the non-tilted state, the patient platform 202 may be positioned horizontally or parallel to a surface (e.g., the ground or floor) supporting the robotic medical system 200. In some embodiments, the term "non-tilted" refers to a state in which the angle between the default position and the current position is less than a threshold angle (e.g., less than 5 degrees, less than an angle that would cause the patient to shift on the patient platform, etc.). In some embodiments, the term "non-tilted" refers to a state in which the patient platform is substantially perpendicular to the direction of gravity, regardless of the angle formed by the surface supporting the robotic medical system relative to gravity.

[0192] Continue to refer Fig. 22 In the illustrated example of the robotic medical system 200, the patient platform 202 includes a support 204. In some embodiments, the support 204 includes a rigid support structure or frame and may support one or more surfaces, pads, or cushions 222. An upper surface of the patient platform 202 may include a support surface 224. During a medical procedure, a patient may be placed on the support surface 224.

[0193] Fig. 22 The robotic arm 210 and the adjustable arm support 220 are shown in an exemplary deployed configuration, wherein the robotic arm 210 reaches above the patient platform 202. In some embodiments, due to the configuration of the robotic medical system 200 that enables various components to be stowed below the patient platform 202, the robotic arm 210 and the arm support 220 can occupy space below the patient platform 202. Therefore, in some embodiments, the tilt mechanism 216 has a low profile and / or a small volume to increase the space available for storage below.

[0194] Fig. 22Also illustrated are example x, y, and z coordinate systems that can be used to describe certain features of the embodiments disclosed herein. It should be understood that the coordinate system is provided only for the purpose of example and explanation, and other coordinate systems may be used. In the illustrated example, when the patient platform 202 is in an untilted state, the x-direction or x-axis extends across the patient platform 202 in a lateral direction. In some configurations, when the patient platform 202 is in an untilted state, the x-direction extends from one lateral side (e.g., right side) across the patient platform 202 to another lateral side (e.g., left side). When the patient platform 202 is in an untilted state, the y-direction or y-axis extends along the patient platform 202 in a longitudinal direction. That is, when the patient platform 202 is in an untilted state, the y-direction extends from one longitudinal end (e.g., head end) to another longitudinal end (e.g., leg end) along the patient platform 202. In an untilted state, the patient platform 202 may be located in or parallel to the xy plane, which may be parallel to the floor or ground. In the illustrated example, the z-direction or z-axis extends in a vertical direction along the column 214. In some embodiments, the tilt mechanism 216 is configured to tilt the patient platform 202 laterally by rotating the patient platform 202 about a lateral tilt axis parallel to the y-axis. The tilt mechanism 216 can further be configured to tilt the patient platform 202 longitudinally by rotating the patient platform 202 about a longitudinal tilt axis parallel to the x-axis.

[0195] In some embodiments, the robotic medical system 200 includes a tower 230 (e.g., a tower viewer) or a physician console 240 (or both), such as Fig.23 As shown. The tower 230 may correspond to the tower 30 described above, and may provide support for controls, electronics, fluids, optics, sensors, and / or power for the patient support platform 202 and the physician console 240. In some embodiments, the tower 230 includes a display device 232. The display device 232 may include a user interface for displaying a surgical view obtained by one or more cameras 606 of the robotic medical system and / or one or more notifications to the operator of the robotic medical system 200. In some embodiments, the physician console 240 may include a display device 242 having a user interface for the physician operator to operate the patient support platform 202. For example, the display device 242 may include a user interface for displaying a surgical view obtained by one or more cameras 606 of the robotic medical system and / or one or more notifications to the operator of the robotic medical system 200. The physician console 240 may provide the physician operator with preoperative and real-time information of the robot control and medical procedures. In some embodiments, the physician console 240 includes one or more input devices (e.g., buttons, switches, touch-sensitive surfaces, universal brackets, etc.), such as a foot switch 244.

[0196] B.Robot Arm

[0197] FIG. 24A to FIG. 24C Various views of an exemplary robotic arm 210 are illustrated, according to some embodiments.

[0198] Fig.24A The illustrated robotic arm 210 includes a plurality of links 302 (eg, a linkage mechanism). The links 302 (eg, 302-1 to 302-4) are connected by one or more joints 304 (eg, 304-1 to 304-5). Each of the joints 304 includes one or more degrees of freedom (DoF).

[0199] exist Fig.24A , the joint 304 includes a first joint 304-1 (e.g., a base joint or an A0 joint) located at or near the base 306 of the robotic arm 210. In some embodiments, the base joint 304-1 includes a prismatic joint that allows the robotic arm 210 to translate along the rod 220 (e.g., along the y-axis). The joint 304 also includes a second joint 304-2. In some embodiments, the second joint 304-2 rotates relative to the base joint 304-1. The joint 304 also includes a third joint 304-3 connected to one end of the link 302-2. In some embodiments, the joint 304-3 includes multiple DoFs and facilitates tilting and rotation of the link 302-2 relative to the joint 304-3.

[0200] Fig.24A A fourth joint 304-4 is also shown connected to the other end of the link 302-2. In some embodiments, the joint 304-4 includes an elbow joint connecting the link 302-2 and the link 302-3. The joint 304 also includes a pair of joints 304-5 (e.g., wrist roll joint) and 304-6 (e.g., wrist pitch joint) located on the distal portion of the robot arm 210.

[0201] The proximal end of the robotic arm 210 may be connected to the base 306, and the distal end of the robotic arm 210 may be connected to an advanced device manipulator (ADM) 308 (e.g., a tool driver, an instrument driver, or a robotic end effector, etc.). The ADM 308 may be configured to control the positioning and manipulation of a medical device (e.g., a tool, an endoscope, etc.).

[0202] The robotic arm 210 may also include a cannula sensor 310 for detecting the presence of a cannula or the proximity of a cannula to the robotic arm 210. In some embodiments, when the cannula sensor 310 (e.g., via one or more processors of the robotic medical system 200) detects the presence of a cannula, the robotic arm 210 is placed in a docked state (e.g., a docked position). In some embodiments, when the robotic arm 210 is in the docked position, the robotic arm 210 may perform zero-space motion to maintain the position and / or orientation of the cannula, as discussed in further detail below. Conversely, when the cannula sensor 310 does not detect a cannula, the robotic arm 210 is placed in an undocked state (e.g., an undocked position).

[0203] In some embodiments, and as Fig.24A As illustrated, the robotic arm 210 includes an input or button 312 (e.g., a donut button, or other type of control, etc.) that can be used to place the robotic arm 210 in an admittance mode (e.g., by pressing the button 312). The admittance mode is also referred to as an admittance scheme or admittance control. In the admittance mode, the robotic system 210 measures the force and / or torque (e.g., applied to the robotic arm 210) and outputs a corresponding speed and / or position. In some embodiments, the robotic arm 210 can be manually manipulated by a user in the admittance mode (e.g., during the setting of a procedure, or between procedures, etc.). In some cases, by using admittance control, the operator does not need to overcome all inertia in the robotic medical system 200 to move the robotic arm 210. For example, under admittance control, when the operator applies a force on the arm, the robotic medical system 200 can measure the force and help the operator move the robotic arm 210 by driving one or more motors associated with the robotic arm 210, thereby obtaining the desired speed and / or position of the robotic arm 210.

[0204] In some embodiments, the link 302 can be removably coupled to the medical tool 212 (e.g., to facilitate easy installation and removal of the medical tool 212 from the robotic arm 210). The joint 304 provides the robotic arm 210 with multiple degrees of freedom (DoF) that facilitate control of the medical tool 212 via the ADM 308. Fig. 22 In the illustrated embodiment comprising multiple robotic arms, each robotic arm can hold its respective medical tool and pivot the medical tool about a remote center of motion.

[0205] Fig. 24B A front view of the robotic arm 210 is illustrated. Fig.24C A perspective view of a robotic arm 210 is illustrated. In some embodiments, the robotic arm 210 includes Fig.24A312 in the figure, a second input portion or button 314 (e.g., a push button) is provided, which is used to place the robotic arm 210 in impedance mode (e.g., by a single press or continuous pressing and holding of the button 314). In this example, the button 314 is located between the joint 304-5 and the joint 304-6. The impedance mode is also referred to as an impedance scheme or impedance control. In impedance mode, the robotic medical system 200 measures displacement (e.g., changes in position and velocity) and outputs force and / or torque to facilitate manual movement of the robotic arm. In some embodiments, the robotic arm 210 can be manually manipulated by a user in impedance mode (e.g., during a setup procedure). In some embodiments, in impedance mode, movement of one portion of the robotic arm 210 by the operator can cause movement of one or more joints and / or links of the entire robotic arm 210.

[0206] In some embodiments, for admittance control, a force sensor or load sensor can measure the force that the operator is applying to the robotic arm 210 and move the robotic arm 210 in a way that feels light. Admittance control may feel lighter than impedance control because under admittance control, the perceived inertia of the robotic arm 210 can be hidden because the motor in the controller can help accelerate the mass. In contrast, with impedance control, according to some embodiments, the user is responsible for most, if not all, of the acceleration of the mass.

[0207] In some cases, depending on the position of the robot arm 210 relative to the operator, it may be inconvenient to reach the button 312 and / or the button 314 to activate the manual steering mode (e.g., the admittance mode and / or the impedance mode). Therefore, in these cases, the operator may conveniently trigger the manual steering mode instead of triggering the manual steering mode via a button.

[0208] In some embodiments, the robotic arm 210 includes a single button (e.g., button 312 or 314) that can be used to place the robotic arm 210 in an admittance mode and / or an impedance mode (e.g., by using different presses, such as a long press, a short press, a press and hold, etc.). In some embodiments, the robotic arm 210 can be placed in an impedance mode by a user pushing an arm link (e.g., link 302) and / or a joint (e.g., joint 304) and overcoming a force threshold. In some embodiments, the admittance mode and the impedance mode are common because both modes allow a user to grab the robotic arm 210 and command movement by directly interacting with the robotic arm.

[0209] In some embodiments, the robotic arm 210 includes an input control for activating the arm following mode. For example, in some embodiments, the robotic arm 210 may include a designated contact point located on a link 302 or joint 304 of the robotic arm (e.g., a housing of the link 302 or a button 316). User interaction (e.g., user touch, contact, etc.) with the designated contact point activates the arm following mode. In some embodiments, the robotic arm 210 includes multiple contacts. User interaction with any (e.g., one or more) of the contacts activates the arm following mode.

[0210] During a medical procedure, it may be desirable for the ADM 308 of the robotic arm 210 and / or the remote center of motion (RCM) of the tool 212 coupled thereto to remain in a static pose (e.g., position and / or orientation). The RCM may refer to a point in space at which a cannula or other access port into which the medical tool 212 is inserted is restricted in motion. In some embodiments, the medical tool 212 includes an end effector that is inserted through an incision or natural orifice of a patient while maintaining the RCM. In some embodiments, the medical tool 212 includes an end effector that is in a retracted state during the setup process of the robotic medical system.

[0211] In some cases, the robotic medical system 200 can be configured to move one or more links 302 of the robotic arm 210 within a "null space" to avoid collisions with nearby objects (e.g., other robotic arms) while the ADM 308 of the robotic arm 210 and / or RCM remains in its corresponding pose (e.g., position and / or orientation). The null space can be considered a set of joint states that the robotic arm 210 can move without causing the ADM 308 and / or RCM to move, thereby maintaining the position and / or orientation of the medical tool 212 (e.g., within the patient). In some embodiments, the robotic arm 210 can have multiple positions and / or configurations that can be used for each pose of the ADM 308.

[0212] For a robotic arm 210 to move an instrument to a desired position in space, in some embodiments, the robotic arm 210 may have at least six DoFs—three DoFs for translation (e.g., X, Y, and Z positions) and three DoFs for rotation (e.g., yaw, pitch, and roll). In some embodiments, each joint 304 may provide a robotic arm 210 with a single DoF, and thus, the robotic arm 210 may have at least six joints to achieve the degrees of freedom of movement to position the ADM 308 in any position in space. To further maintain the ADM 308 and / or remote center or motion of the robotic arm 210 at a desired position, the robotic arm 210 may also have at least one additional “redundant joint.” Thus, in some embodiments, a system may include a robotic arm 210 with at least seven joints 304, which provides a robotic arm 210 with at least seven DoFs. In some embodiments, the robotic arm 210 may include a subset of joints 304, each joint having more than one degree of freedom, thereby achieving additional DoFs for zero-space motion. However, depending on the embodiment, the robotic arm 210 may have more or fewer DoFs.

[0213] In addition, as regards Fig.12 As described, the rod 220 (e.g., an adjustable arm support) can provide several degrees of freedom, including lifting, lateral translation, tilting, etc. Thus, depending on the embodiment, the robotic medical system can have more degrees of freedom for robotic control than just in the robotic arm 210 to provide zero-space movement and collision avoidance. In corresponding embodiments of these embodiments, the end effector of one or more robotic arms (and any tools or instruments coupled thereto) and / or the remote center along the axis of the tool can advantageously maintain a posture and / or position within the patient.

[0214] A robotic arm 210 having at least one redundant DoF has at least one more DoF than the minimum number of DoFs required to perform a given task. For example, according to some embodiments, the robotic arm 210 may have at least seven DoFs, wherein one of the joints 304 of the robotic arm 210 may be considered a redundant joint. The one or more redundant joints may allow the robotic arm 210 to move in null space to maintain the pose of the ADM 308 and the position of the RCM while avoiding collisions with other robotic arms or objects.

[0215] In some embodiments, the robotic medical system 200 can be configured to perform collision avoidance by utilizing movement of one or more redundant joints in null space to avoid collisions, for example, between adjacent robotic arms 210. For example, when a robotic arm 210 collides with or approaches (e.g., within a defined distance) another robotic arm 210, one or more processors of the robotic medical system 200 can be configured to detect a collision or an impending collision (e.g., via kinematics). Accordingly, the robotic medical system 200 can control one or both of the robotic arms 210 to adjust their respective joints in null space to avoid the collision or the impending collision. In one embodiment including at least one pair of robotic arms, the base of one of the robotic arms and its end effector can maintain its pose while the link or joint therebetween moves in null space to avoid collisions with adjacent robotic arms.

[0216] C. Exemplary Surgical Setup

[0217] Fig.25 A perspective view of a robotic medical system 200 including four robotic arms 210-1, 210-2, 210-3, and 210-4 is illustrated according to some embodiments. Each of the robotic arms 210-1, 210-2, 210-3, and 210-4 is connected to a corresponding surgical tool 602 (e.g., 602-1 to 602-4, which may correspond to the instrument 212) via a corresponding ADM 308 (e.g., tool driver) such as ADM 308-1 to 308-4. The surgical tool 602 can be inserted into the patient's body via a corresponding port 608 located on the patient. As used herein, a port (e.g., a port location, an access port, an access point, a port area, a port region, or a port orientation, etc.) refers to a location on the patient's body through which a medical tool / instrument (e.g., held by a robotic arm) can be inserted and constrained in motion. In some embodiments, the port corresponds to an incision point (or incision area) formed through the patient's skin to facilitate a medical surgery or procedure. In some embodiments, the port corresponds to a natural orifice, such as a patient's mouth (e.g., for bronchoscopy procedures). In some embodiments, the port corresponds to a medical device having an opening that is placed at the incision site or natural orifice to allow access to the surgical space through the opening. The view of the patient has been Fig.25 is excluded in order to enhance visibility of the robotic arm 210 and the surgical tool 602.

[0218] exist Fig.25, the robotic arm 210-2 is coupled to a camera 606. In this example, the camera 606 is coupled to the robotic arm 210-2 (e.g., at the distal end of the medical instrument 602-2) via a medical instrument 602-2 (e.g., an endoscope). In some embodiments, the camera 606 is part of the medical instrument 602-2. In some embodiments, the camera 606 can be a separate device (e.g., not part of the surgical instrument) coupled to the robotic arm (e.g., the camera 606 is different from and separate from the medical instrument). In some embodiments, the camera 606 defines an axis 604 (e.g., an optical axis) that identifies the orientation of the camera 606. The camera 606 (or endoscope) provides an image of the surgical site to facilitate control of surgical tools to perform robotic medical procedures. For example, a robotically controllable endoscope of a robotic system may include a camera positioned at its distal end. A user may view images from the camera of the endoscope in a viewer to facilitate control of the endoscope and / or other components of the robotic medical system. As another example, the robotic system may include one or more cameras inserted into a laparoscope or endoscope into a patient's body. A user may view images from the inserted camera to facilitate control of one or more additional robotically controlled medical instruments, such as one or more additional laparoscopically inserted medical instruments.

[0219] In some embodiments, the robotic medical system 200 includes a coordinate system (e.g., a robot coordinate system, a coordinate system, a system coordinate system, etc., which can be a Cartesian or non-Cartesian coordinate system), and the respective positions of the patient support platform 202, the robotic arm 210, the adjustable arm support 220, and / or the instrument 212 are represented as coordinates on the coordinate system (e.g., x, y, and z coordinates). For example, the robotic medical system 200 (e.g., one or more processors 380 of the robotic medical system 200) can be configured to identify the position and orientation of the patient support platform 202, the robotic arm 210, the adjustable arm support 220, and / or the instrument 212 based on the coordinates in the coordinate system.

[0220] Fig.26A and Fig.26BThe camera field of view in three-dimensional space according to some embodiments is illustrated. In some embodiments, the robotic medical system 200 determines the field of view based on the focal length of the camera 606 (e.g., the field of view may correspond to a space within a certain distance from the center of the field of view of the camera 606). In some embodiments, the robotic medical system 200 determines the field of view using data obtained by one or more sensors 388 of the robotic medical system 200, such as position (and orientation) data corresponding to the tip of an endoscope (e.g., a medical instrument) to which the camera 606 is coupled or position data corresponding to the camera 606 (e.g., a safe zone may be determined to be located at a specific distance from the camera 606). In some embodiments, the robotic medical system 200 stores (e.g., in a memory) information identifying the field of view of the camera 606, and uses the stored information to determine the field of view of the camera 606 based on the position and orientation of the camera 606. For example, the robotic medical system 200 may store information identifying a predefined height (e.g., several centimeters) and predefined base angles, bevel angles, and / or edge angles of a three-dimensional volume corresponding to the field of view, the three-dimensional volume having the shape of a truncated rectangular cone. As another example, the robotic medical system 200 may store information identifying a predefined radius or diameter and a predefined distance (e.g., a number of centimeters) of a spherical volume corresponding to the field of view, the spherical volume being located at a predefined distance from the camera 606 and having the predefined radius or diameter. In some embodiments, the robotic medical system 200 determines the field of view based on an image obtained using the camera 606.

[0221] exist Fig.26A and Fig.26B In , the camera field of view 702 (e.g., as seen by a camera coupled to another robot arm) is represented by a three-dimensional space (e.g., a cone). One of ordinary skill in the art will appreciate that the three-dimensional space of the camera field of view may be represented by other shapes, such as a cone, a cylinder, a sphere, a tetrahedron, etc. In Fig.26A In the example of , the robotic arm 210-8 is coupled to a surgical tool 704 (e.g., a medical instrument) via the ADM 308-8. The surgical tool 704 intersects (e.g., overlaps, is within, etc.) the three-dimensional space of the camera field of view 702. In contrast, Fig.26B The robotic arm 210 - 9 is shown holding a surgical tool 708 (eg, via the ADM 308 - 9 ) that does not intersect the three-dimensional space of the camera field of view 702 (eg, due to the position and / or orientation of the tool 708 ).

[0222] D. Flexible Speculum

[0223] FIG. 27A to FIG. 27DDifferent views of a flexible scope 720 (eg, a flexible laparoscope, a flexible endoscope, etc.) according to some embodiments are illustrated. In some embodiments, the flexible scope is coupled to the robotic arm 210 of the robotic medical system 200 .

[0224] Fig.27A The flexible scope 720 is illustrated as including a distal end 724 (e.g., a distal portion) and an intermediate portion 722 (e.g., a portion positioned from the distal end 724 toward the proximal end 740). In some embodiments, the flexible scope 720 is coupled to a camera at the distal end 724. In some embodiments, the flexible scope 720 includes a plurality of links 726 (e.g., a linkage). In some embodiments, the plurality of links 726 are connected by one or more joints 728. For example, in Fig.27A In the embodiment, the flexible scope 720 includes links 726-1 to 726-3 connected by joints 728-1 and 728-2. In some embodiments, the plurality of links 726 includes four or more links.

[0225] In some embodiments, the flexible scope 720 includes a rigid (e.g., non-flexible) portion and a flexible portion. The rigid portion has a fixed (e.g., non-flexible, unchangeable) shape. The flexible portion has a shape that can be manipulated (e.g., changed, altered, adjusted, modified, etc.).

[0226] In some embodiments, the rigid portion is located at or near the proximal end of the flexible scope (e.g., middle portion 722), and the flexible portion is located at the distal end of the flexible scope (e.g., distal end 724). In some embodiments, the rigid portion is located at the distal end of the flexible scope (e.g., distal end 724), and the flexible portion is located at or near the proximal end of the flexible scope (e.g., middle portion 722). In some embodiments, the flexible scope 720 includes two or more rigid portions (e.g., connecting rods 726-1 and 726-3) connected by a flexible portion (e.g., connecting rod 726-2). In some embodiments, the rigid portion and the flexible portion are connected by a connecting rod (e.g., connecting rod 726-2) and / or a joint (e.g., joint 728-1 or 728-2), and the flexible portion can move relative to the rigid portion (e.g., tilt, rotate, shift, etc.).

[0227] In some embodiments, the flexible scope 720 introduces additional degrees of freedom compared to a rigid scope. For example, in addition to the insertion, removal, translation, and rotation operations that can be performed on a rigid scope, at least a portion of the flexible scope 720 (e.g., a distal portion or a middle portion) can be articulated relative to another portion of the flexible scope 720 (e.g., tilt, rotate, pivot, flex, be caused to perform rotational movement, etc.), thereby enabling the user to view the target anatomical structure at a viewing angle that cannot be achieved with a rigid scope. In some embodiments, the flexible scope 720 has a smaller volume than a rigid scope, thereby freeing up space in the surgeon's workspace.

[0228] In some embodiments, the robotic system 200 (e.g., via the processor 380) is configured to operate the flexible scope 720 in a plurality of modes (e.g., drive modes), such as an articulation mode (e.g., an articulation drive mode), a track mode (e.g., a track drive mode), and an automatic insertion and retraction mode (e.g., an automatic insertion and retraction drive mode), as described in further detail below. For example, in some embodiments, the robotic system 200 operates the flexible scope 720 in an articulation mode at a first time, in a track mode at a second time different from the first time, and in an automatic insertion and retraction mode at a third time different from the first time and the second time. In some embodiments, the plurality of modes includes two modes in the aforementioned modes. In some embodiments, the plurality of modes includes four or more modes.

[0229] Fig.27B Operation of the flexible scope 720 in an articulation drive mode according to some embodiments is illustrated. In some embodiments, in the articulation drive mode, the distal portion 724 of the flexible scope 720 can be moved (730) (e.g., articulated, articulated, tilted, rotated, pivoted, rotated, etc.) in an upward direction, a downward direction, a leftward direction, or a rightward direction (e.g., by the processor 380). For example, the processor 380 can provide (e.g., to one or more actuators (such as motors or actuator controllers)) an electrical signal for causing movement of the distal portion 724 of the flexible scope 720. Fig.27B Three positions or states of the flexible scope 720 are illustrated when the distal portion 724 is moved or rotated (730) from an initial position or orientation ("a") (e.g., aligned with the middle portion 722) (e.g., facing north) to a second position or orientation ("b") (e.g., facing northwest) or to a third position or orientation ("c") (e.g., facing northeast). In some embodiments, as Fig.27B As illustrated, the middle portion 722 of the flexible scope 720 remains stationary in the articulation drive mode.

[0230] Fig.27C Operation of the flexible scope 720 in a track-driven mode according to some embodiments is illustrated. In some embodiments, in the track-driven mode, the flexible scope 720 (e.g., at the distal portion 724) can be moved (732) (e.g., in an orbital motion) about a target point 734 (e.g., a track point) in space (e.g., a track point) (e.g., by the processor 380). For example, when the flexible scope 720 is operated in the track-driven mode, as the flexible scope moves (e.g., laterally), the distal end of the flexible scope 720 rotates or tilts to face the target point 734. Fig.27C , three exemplary positions or postures of the flexible scope 720 when the distal end 724 of the flexible scope 720 orbits around the target point 734 are shown in (i), (ii) and (iii).

[0231] In some embodiments, the target point 734 is a fixed point in space. In some embodiments, the target point 734 is predefined (e.g., selected) by a user. In some embodiments, the target point 734 is predefined by the robotic system. In some embodiments, the target point 734 is automatically selected by the robotic medical system 200 (e.g., independent of user input). For example, the target point 734 can be selected as the location of a specific organ or anatomical structure of the patient.

[0232] Fig.27D 30 illustrates the operation of the flexible scope 720 in the automatic insertion and retraction drive mode according to some embodiments. In some embodiments, in the automatic insertion and retraction mode, the flexible scope 720 can be automatically (e.g., by the processor 380) inserted (e.g., moved in the direction 736) into the surgical site from the current position (e.g., without manual force applied to the flexible scope 720). In some embodiments, as Fig.27D As illustrated, the distal portion 724 of the flexible scope bends as the flexible scope is inserted (e.g., along the sheath). In the automatic insertion and retraction mode, the flexible scope 720 can also be automatically retracted (e.g., moved in direction 738) from the surgical site (e.g., without manual force applied to the flexible scope 720) to the port or cannula of the robotic arm to which the flexible scope 720 is connected (e.g., by the processor 380). In some embodiments, the distal portion 724 of the flexible scope 720 straightens when retracted.

[0233] Flexible scopes are beneficial because they enable users (e.g., surgeons) to obtain different views of target anatomical structures that may not be obtained using rigid scopes. However, operating (e.g., driving) flexible scopes in various drive modes may be challenging. For example, in the joint motion drive mode, it may be difficult for a user to identify the current shape or configuration of the flexible scope, or the degree or direction of the joint motion, or in order to recover from a bent state to a straight state, the scope should be moved in which direction. In the track drive mode, it may be difficult for a user to determine the distance between the flexible scope and the target point around which the flexible scope is orbitally moved, or the position of the flexible scope in the track, or the shape of the flexible scope. In the automatic insertion and retraction drive mode, it may be difficult for a user to understand the current state or position (along the insertion path) of the flexible scope when the flexible scope is inserted or retracted, or whether the flexible scope is in a bent state or has been straightened. Therefore, a robotic medical system that can facilitate user-visible positioning or orientation of a flexible scope is needed.

[0234] E. Exemplary User Interface

[0235] According to some embodiments, the robotic system includes a user interface that can be configured to display dynamic visual representations of the flexible scope described above for different drive modes. In some embodiments, the user interface is configured to display a dynamic visual representation of the flexible scope based on determining that the robotic arm is coupled to the flexible scope (e.g., regardless of the drive mode).

[0236] FIG. 28A to FIG. 28D An exemplary user interface displayed on a display device when the flexible scope is operating in an articulation drive mode according to some embodiments is illustrated.

[0237] Fig.28A 8 shows a user interface 802 displayed on a display device 800 according to some embodiments. In some embodiments, as Fig.28A As shown, the user interface 802 includes an image or representation of a camera's field of view (eg, a view of a surgical site such as a treatment site) or a scope (eg, the flexible scope 720). Fig.28A Also shown is user interface 802 including images or representations of medical devices 500 and 550 that are within the field of view of the camera or scope.

[0238] In some embodiments, such as Fig.28AAs illustrated, the user interface 802 includes indicators 804 (e.g., visual indicators) identifying medical instruments at the surgical site (e.g., "Tool A," "Tool B," "Tool C," "Tool D"). In some embodiments, the indicators 804 include numbers associated with the robotic arms coupled to the respective medical instruments. For example, the indicator 804-2 includes the number "2," which indicates that Tool B is coupled to the second robotic arm of the robotic medical system.

[0239] In some embodiments, such as Fig.28A As illustrated, the user interface 802 includes an indicator 806 (e.g., a visual indicator, a scope label, etc.) for indicating the position and / or orientation of a flexible scope (e.g., the flexible scope 720) or a portion thereof (e.g., the distal end 724 of the flexible scope 720) relative to a reference position. The reference position can be a neutral position (e.g., a position corresponding to a fully straightened scope, a position determined based on an insertion port, etc.). In some embodiments, the indicator 806 has an initial size similar to the indicator 804 and then expands (e.g., expands to a position such as when the camera coupled to the flexible scope is or has become active (e.g., as determined by the robotic medical system) Fig.28A size shown).

[0240] Fig.28A The illustration in FIG. 8 shows that the indicator 806 includes a graphical element such as a grid 809, which includes a reference axis 807-1 (e.g., a horizontal axis), a reference axis 807-2 (e.g., a vertical axis), and a marker 808 (e.g., a dot, a point, etc.). The marker 808 indicates the direction and / or degree of articulation (e.g., movement, bending, pivoting, etc.) of the flexible scope. Fig.28A In the example of , indicator 806 indicates that the distal end of the flexible endoscope is articulated (e.g., moved, pivoted, bent, etc.) upward and left relative to its reference position by showing a mark 808 positioned above and to the left of an origin formed by reference axes 807-1 and 807-2 (e.g., the intersection of reference axes 807-1 and 807-2).

[0241] In some embodiments, indicator 806 includes a number associated with the robotic arm to which the flexible scope is coupled. Fig.28A The indicator 806 is shown to include the number "3", which indicates that the flexible scope is coupled to the third robotic arm of the robotic medical system.

[0242] In some embodiments, such as Fig.28A As illustrated, the user interface 802 includes an indicator 810 (eg, along a perimeter of the user interface 802 or a portion thereof) for indicating the position (or orientation) of the medical device relative to the field of view. Fig.28A, indicator 810-1 is positioned along the left edge of user interface 802, thereby indicating that medical device 500 is positioned on the left side of the field of view. Fig.28A As shown, indicator 810 - 1 includes information identifying the medical device 500 (eg, a number associated with a robotic arm coupled to the medical device 550 , such as number “ 1 ” indicating a first robotic arm).

[0243] In some embodiments, indicator 810 (or any other indicator for any medical tool) is displayed when the corresponding medical tool is outside the field of view. In some embodiments, indicator 810 (or any other indicator for any medical tool) is displayed regardless of whether the corresponding medical tool is inside or outside the field of view.

[0244] Fig.28B It is shown that in some embodiments, the indicator 806 includes a representation 812 (e.g., a real-time representation or a three-dimensional representation or rendering) corresponding to the shape of the flexible scope to further help the user understand the shape of the scope. In some embodiments, the representation 812 includes a rendering of the connecting rods 814. In some embodiments, the number of connecting rods 814 (e.g., 814-1 to 814-3) in the representation 812 corresponds to the number of connecting rods (e.g., connecting rods 726-1 to 726-3) of the flexible scope.

[0245] Fig.28B and Fig.28C The transition between shows the articulation of the distal end of the flexible scope from Fig.28B Up and left position to Fig.28C In some embodiments, in response to detecting a change in the articulation of the flexible scope, the user interface 802 (e.g., via the processor 380) updates one or more graphical elements of the indicator 806 to reflect the changed position and / or shape of the scope. Fig.28C , user interface 802 displays an updated graphical element (eg, marker) 811 positioned above and to the right of the origin formed by reference axes 807-1 and 807-2 to indicate that the distal end of the scope has moved (articulated) to an upward and rightward position. Fig.28C Also shown is the user interface 802 displayed with an updated representation 813 reflecting the current shape of the flexible scope. Fig.28B and 28C As illustrated, indicator 806 may be or function as a dynamic indicator that changes in real time as the scope moves.

[0246] In some embodiments, the user interface 802 displays a representation of the field of view of the camera of the flexible scope that is updated in response to the movement of the flexible scope. Fig.28C, when the flexible scope is articulated upward and rightward relative to its reference position, the user interface 802 displays an updated representation 820 of the camera's field of view that is consistent with the Fig.28B The representation 818 of the field of view of the camera in has been translated to the left (related to the rightward movement of the distal end of the flexible scope).

[0247] Fig.28D A user interface 802 including an indicator 822 (e.g., a bar, an articulation bar, etc.) according to some embodiments is illustrated. In this example, the indicator 822 includes a horizontal bar 822-1 and a vertical bar 822-2. In some embodiments, the horizontal bar 822-1 includes a graphical element (e.g., an arrow) 824-1 indicating a left articulation direction relative to a reference position 826-1 (e.g., the center of the horizontal bar) and a graphical element (e.g., an arrow) 824-2 indicating a right articulation direction relative to the reference position 826-1. In some embodiments, the vertical bar 822-2 includes a graphical element (e.g., an arrow) 824-3 indicating an upward articulation direction relative to the reference position 826-2 (e.g., the center of the vertical bar) and a graphical element (e.g., an arrow) 824-4 indicating a downward articulation direction relative to the reference position 826-2.

[0248] In some embodiments, the indicator 822 includes a graphic element 828-1 indicating the direction of movement of the distal end of the flexible scope along the horizontal direction. In some embodiments, the indicator 822 includes a graphic element 828-2 indicating the direction of movement of the distal end of the flexible scope along the vertical direction. Fig.28D In the example of , the distal end of the flexible scope is in a rightward and upward direction relative to its reference position. Fig.28D In the user interface 802 of FIG. 8 , the graphical element 828-1 is positioned to the right of the reference position 826-1, and the graphical element 828-2 is positioned above the reference position 826-2. In some embodiments, the distal end of the flexible scope moves in a rightward and upward direction relative to its reference position (e.g., articulates, bends, flexes, etc.), and the user interface 802 indicates the direction of movement using the graphical element 828-1 and the graphical element 828-2.

[0249] In some embodiments, each of graphical elements 828-1 and 828-2 has (or is characterized by) a respective length that represents the degree of articulation of the flexible scope. In some embodiments, the lengths of graphical elements 828-1 and 828-2 are updated dynamically (e.g., in real time) as the scope moves. For example, graphical element 828-1 has a length that increases as the scope articulates to the right away from the reference position (e.g., real timeElement 828-2 has a length 830-2 that increases (e.g., in real time) as the scope articulates upward away from the reference position and decreases as the scope articulates downward toward the reference position.

[0250] FIG. 29A to FIG. 29D An exemplary user interface displayed on a display device when the flexible scope is operating in a track drive mode according to some embodiments is illustrated.

[0251] Fig.29A and Fig.29B The user interface 802 is shown including an indicator 832. In some embodiments, the indicator 832 includes a graphical element 834 (e.g., a dot, a sphere, a cross mark, an "X" symbol, a hollow circle, a solid circle, etc.) representing an orbital point (e.g., a target point, such as the target point 734, etc.) about which the flexible scope performs orbital motion.

[0252] During surgery, the surgeon may actuate an input device on the surgeon console 240 to control the depth to which the flexible endoscope is inserted into the surgical site. In some embodiments, the graphical element 834 has (e.g., is characterized by) a size (e.g., length, width, height, diameter, etc.) that can be updated (e.g., modified) according to changes in the insertion depth of the flexible endoscope. For example, from FIG. 29A to FIG. 29B The transition from exemplifies that the flexible endoscope is moved closer to the surgical site. In some embodiments, as the flexible endoscope is moved closer to the surgical site (or toward a target organ, such as an organ positioned at the center of the camera's field of view), the diameter of the graphical element 834 displayed in the user interface 802 increases (e.g., is enlarged), such as from FIG. 29A to FIG. 29B In some embodiments, Fig.29A The smaller diameter of the graphic element 834 in FIG. 8 indicates a greater distance between the endoscope and the surgical site, while Fig.29B A larger diameter of the graphic element 834 in indicates a smaller distance between the endoscope and the surgical site.

[0253] In some embodiments, the user interface 802 displays a representation of the camera's field of view that varies according to the depth to which the flexible scope 720 is inserted into the surgical site. Fig.29B , the user interface 802 displays a representation 842 of the camera's field of view having the same Fig.29A The magnification of the field of view depicted in FIG840 is a greater degree of magnification than that depicted in FIG840 .

[0254] In some embodiments, such as Fig.29A and Fig.29BAs illustrated, the indicator 832 includes a graphical element 836 (e.g., a line, a dashed line, etc.) representing a line of sight from the track point (or target point) to a port corresponding to the flexible endoscope (e.g., a port through which the flexible endoscope is inserted). In some embodiments, the indicator 832 includes a graphical element 838 for indicating a direction to the endoscope port. In some embodiments, when the endoscope port, camera, and track point are aligned (e.g., the port is behind the camera and the track point is in front of the camera along a common line of sight), the graphical element 836 is omitted. For example, the graphical element 836 (e.g., together with the graphical element 838) can be aggregated or merged with the graphical element 834 so that when the endoscope port, camera, and track point are aligned, the graphical element 836 is no longer displayed in the user interface 802.

[0255] Fig.29C A user interface 802 with an orbital sphere is illustrated according to some embodiments. In this example, the user interface 802 displays an indicator 844 that includes a graphical element 846 (e.g., a sphere, a 3D sphere, a globe, etc.) representing the orbital sphere that depicts the range (e.g., the full range) of the orbital motion of the flexible scope. In some embodiments, the indicator 844 includes a graphical element 848 (e.g., a first circle, a hollow circle, etc.) representing an orbital point around which the flexible scope orbits. In some embodiments, the indicator 844 includes a graphical element 850 (e.g., a marker, a second circle, a solid circle, etc.) that represents the position of the flexible scope in the orbital sphere (e.g., in real time). Fig.29C In the example of FIG. 1 , the distal end of the scope is located at the top and left position relative to the track point. In some embodiments, when the scope is aligned with the track point (e.g., Fig.27C ), graphical element 850 ceases to be displayed. For example, graphical element 850 may be aggregated or merged with graphical element 848 so that when the endoscope port, camera, and track point are aligned, graphical element 836 is no longer displayed in user interface 802.

[0256] In some embodiments, the graphical element 846 has a size (e.g., diameter, length, size, etc.) that is related to the distance between the endoscope and the target point. In some embodiments, the size of the graphical element 846 is dynamic and changes (e.g., in real time) according to the distance between the endoscope and the target point. For example, the size (e.g., diameter) of the graphical element 846 increases as the endoscope is inserted and moved toward the surgical site (e.g., when the distance between the endoscope and the target point decreases), and decreases as the endoscope is retracted away from the surgical site (e.g., when the distance between the endoscope and the target point increases). In some embodiments, the user interface 802 also displays graphical elements 848 and 850 at updated positions on the graphical element 846 according to the change in the size of the graphical element 846. The superposition of the graphical elements 846, 848, and 850 on the surgical field of view provides the surgeon with insights into the position and configuration of the endoscope and how the endoscope can be straightened (e.g., by adjusting the endoscope from a position corresponding to the representation 850 to a position corresponding to the representation 848).

[0257] Fig.29D A user interface 802 with a 3D axis indicator 854 is illustrated according to some embodiments. The indicator 854 includes graphical elements 856 representing axes (e.g., x-axis 856-1, y-axis 856-2, and z-axis 856-3). The orientation or rotation of the axes 856 represents the orientation or position of the flexible scope relative to the port. In some embodiments, the user interface displays the indicator 854 with the graphical elements 856 (e.g., as shown in FIG. 1 ) based on determining that the scope and the track point are not aligned. Fig.27C In some embodiments, when the scope is aligned with the track point (eg, the scope is a straight scope), graphical element 856-3 ceases to be displayed or becomes a dot.

[0258] FIG. 30A to FIG. 30C An exemplary user interface displayed on a display device when a flexible scope is operating in an automatic insertion and retraction mode according to some embodiments is illustrated.

[0259] Fig. 30A The display of indicator 860 in user interface 802 is illustrated. In some embodiments, indicator 860 includes a graphical element 861 corresponding to the shape (e.g., outline) of a flexible scope that is coupled to the robotic arm and is used in the present surgical procedure. In this example, the flexible scope is configured as a flexible scope that performs 90° of articulation. Indicator 860 also includes a graphical element 861 (e.g., a shadow, rendering, etc.) superimposed on graphical element 862 and representing the real-time position or state of the scope. Fig. 30AIn the example of , the endoscope is fully inserted and therefore graphical element 862 completely fills graphical element 861 (e.g., completely superimposes the graphical element).

[0260] Fig. 30B The display of indicator 860 in user interface 802 is illustrated when the scope is partially retracted. Fig. 30B , graphical element 862 partially fills graphical element 861. The shaded portion of the endoscope (eg, represented by graphical elements 861 and 862 together) represents the current shape and position of the endoscope in real time and is dynamically updated (eg, as the endoscope continues to be retracted or reinserted).

[0261] In some embodiments, the user interface 802 displays a text element 864 (such as "INSERTING" or "RETRACTING") to indicate whether the scope is being inserted or retracted.

[0262] Fig. 30C A progress bar according to some embodiments is illustrated. In this example, the user interface 802 displays an indicator 866 (e.g., a bar) that includes a graphical element 868 (e.g., a bar, a shaded portion, a shadow, a rendering, etc.) indicating the degree of insertion and / or retraction of the endoscope. In some embodiments, the indicator 866 includes a graphical element 870 (e.g., a text element) corresponding to the fully inserted (e.g., maximum insertion) position of the endoscope. In some embodiments, the indicator 866 includes a graphical element 872 (e.g., an icon, a text element, etc.) indicating whether the flexible endoscope has undergone an articulation. For example, in some embodiments, the graphical element 872 is visually emphasized (e.g., lit, highlighted, etc.) when or when the flexible endoscope has undergone an articulation. Fig. 30C In the example of , the flexible scope is in a partially inserted and articulated state. In this example, graphical element 868 indicates the extent of insertion, and the visual emphasis of graphical element 872 indicates that the scope has been articulated.

[0263] FIG. 30D to FIG. 30F Examples of indicators 866 indicating different states of the scope are illustrated.

[0264] Fig.30D An indicator 866 corresponding to a fully retracted and straight state of a flexible scope is illustrated according to some embodiments. In some embodiments, the indicator 866 includes a graphical element 874 (e.g., a text element) corresponding to the fully retracted position of the scope. In some embodiments, the indicator 866 includes a graphical element 870 (e.g., a text element) corresponding to the fully inserted position of the scope, such as with respect to Fig. 30C In Fig.30DIn some embodiments, the indicator 866 also includes a graphic element 872. In some embodiments, the graphic element 872 is Fig.30D The illustration is visually de-emphasized or omitted (or stopped displaying) to indicate that the scope is not articulating.

[0265] Fig.30E An indicator 866 is illustrated corresponding to a partially inserted and straight (e.g., not articulated) state of the flexible scope, according to some embodiments. Fig.30E , graphical element 868 indicates the extent of insertion (e.g., partial insertion). Fig.30D As shown in the indicator 866, the graphic element 872 may be Fig.30E The illustration is visually de-emphasized or alternatively omitted (or stopped displaying) to indicate that the scope is not articulating.

[0266] Fig.30F An indicator 866 is illustrated with the flexible scope fully inserted and articulated (e.g., partially articulated, fully articulated, etc.) in accordance with some embodiments. Fig.30F , graphical element 870 indicates the extent of insertion of the endoscope. Fig.30F As shown, graphical element 870 is visually emphasized to indicate that the endoscope is fully inserted. In addition, graphical element 872 indicates whether the endoscope has been articulated. For example, Fig.30F As shown, graphical element 872 is visually emphasized to indicate that the scope has been articulated.

[0267] Fig.31A Different configurations of flexible scopes according to some embodiments are illustrated, such as scopes with 0°, 45°, and 90° articulation. In some embodiments, the shape or configuration of the scope (e.g., 0° scope, 45° scope, or 90° scope) when retraction or insertion is initiated is displayed in the user interface 802 (e.g., as FIG. 30A to FIG. 30B ), with graphic element 862 indicating the progress of insertion or retraction.

[0268] FIG. 31B to FIG. 31F Exemplary user interface elements displayed on a display device to indicate a status of a flexible scope according to some embodiments are illustrated.

[0269] Fig.31B Indicator 880 according to some embodiments is illustrated. In some embodiments, indicator 880 is shown in a graphic projection (e.g., a non-front view graphic projection). In some embodiments, indicator 880 is shown in a perspective view, an oblique view, or an auxiliary view (such as an isometric view (e.g., a front view). Fig.31BIn some embodiments, the indicator 880 includes one or more reference planes (e.g., xz plane 882 and yz plane 884) or graphical representations thereof, or is displayed together with the one or more reference planes or graphical representations thereof. In some embodiments, the one or more reference planes are semi-transparent (e.g., semi-permeable). This facilitates simultaneous display of the position of the scope relative to the one or more reference planes. Fig.31B Indicator 880 (e.g., 880-1 to 880-4) in different states are shown. For example, indicator 880 in state 880-1 indicates that the scope is bent upward (e.g., in the direction toward the y-axis), indicator 880 in state 880-2 indicates that the scope is bent to the right (e.g., in the direction toward the x-axis), indicator 880 in state 880-3 indicates that the scope is bent downward (e.g., away from the y-axis), and indicator 880 in state 880-4 indicates that the scope is bent to the left (e.g., away from the x-axis). In some embodiments, indicator 880 is placed in different states indicating that the scope is bent in a direction that is not parallel and perpendicular to the reference plane.

[0270] Fig.31C Indicator 886 is illustrated according to some embodiments. Fig.31C The indicator 886 shown is similar to Fig.31B The indicator 880 is shown. However, in Fig.31C , indicator 886 is shown with a reference user interface element 888. In some embodiments, the reference user interface element 888 is depicted as a two-dimensional representation. In some embodiments, the reference user interface element 888 has a circular shape (e.g., a circle, an ellipse, an oval, etc.). In some embodiments, the reference user interface element 888 is depicted as a three-dimensional representation. In some embodiments, the reference user interface element 888 is a graphical representation of a sphere, a dome, or a semi-dome. In some embodiments, the reference user interface element 888 includes one or more axis indicators (e.g., an x-axis indicator 892 and a y-axis indicator 894). In some embodiments, the reference user interface element 888 (and one or more axis indicators) are translucent (e.g., semi-transparent). This facilitates simultaneous display of the position of the scope relative to the reference user interface element 888 and one or more axis indicators. Although Fig.31CRepresentations of the indicator 886 in multiple states are shown (e.g., the indicator 886 in a state indicating that the endoscope is bent upward is shown with a solid line, and the indicator 886 in a state indicating that the endoscope is bent to the left, right, and downward is shown with a dashed line), but in some embodiments, the indicator 886 is shown in only a single state (e.g., the indicator 886 represented by a solid line is shown without a dashed line to indicate a state in which the endoscope is bent upward). In some embodiments, a portion of the reference user interface element 888 is visually distinguished (e.g., highlighted) to indicate the orientation or posture of the endoscope. For example, a portion (e.g., one of the four predefined quarters) of the reference user interface element 888 (e.g., a semi-dome) is visually distinguished (e.g., distinguished from the rest of the four predefined quarters of the semi-dome in terms of transparency, color, etc.) to highlight the orientation or state of the flexible endoscope (or the orientation or state to which the flexible endoscope is moving).

[0271] Fig.31D Indicator 896 is illustrated according to some embodiments. Fig.31D The indicator 896 shown is similar to Fig.31B 880, except that indicator 896 shows insertion and / or progress of the scope in addition to flexing of the scope. For example, indicator 896 in states 896-1, 896-2, and 896-3 indicates different degrees of insertion of the scope. Indicator 896 in state 896-4 shows flexing of the scope to indicate that the scope is in a flexed state (or that the scope has begun to flex) by representing the scope in a flexed form. In some embodiments, one or more reference user interface elements (e.g., Fig.31B The reference plane shown or Fig.31C In some embodiments, the one or more reference user interface elements are displayed based on determining that the scope is bent (e.g., the one or more reference user interface elements are displayed only after the scope is bent and not before). In some embodiments, the one or more reference user interface elements are stopped from being displayed based on determining that the scope is straight (e.g., the one or more reference user interface elements are removed when the scope is restored from a bent state to a straight state). In some embodiments, the one or more reference user interface elements are always displayed. In some embodiments, the one or more reference user interface elements are not shown.

[0272] In some embodiments, the indicator 880 or 896 is displayed in a picture-in-picture window, such as Fig.31EWindow 3102, 3104, or 3106 is shown. In some embodiments, the picture-in-picture window is placed in a corner of the user interface 802. This reduces distraction to the operator (e.g., surgeon).

[0273] In some embodiments, Fig.31E The picture-in-picture windows are shown being moved away from the user interface 802 so that they cease to overlap with the user interface 802 (e.g., the picture-in-picture windows (such as windows 3102, 3104, and 3106) are moved to a position below the user interface 802), as shown. Fig.31F In some embodiments, the user interface 802 is scaled (e.g., reduced) and / or moved (e.g., the center of the user interface 802 is moved upward) to provide space on the display device 800 for displaying windows that leave the user interface 802, such as Fig.31F shown.

[0274] In some embodiments, FIG. 28A to FIG. 20 D. FIG. 29A to FIG. 29D , FIG. 30A to FIG. 30F and FIG. 31A to FIG. 31F The illustrated visual representation or one or more portions thereof are color-coded. In some embodiments, at least a portion of the indicator 896 (e.g., a straight portion) may be color-coded to indicate the percentage of insertion. For example, at least the straight portion of the indicator 896 has a first color, such as green, when the insertion degree of the scope is between 0% and 80%; a second color, such as yellow, when the insertion degree of the scope is between 80% and 99%; and a third color, such as red, when the scope is in a 100% insertion state. In some embodiments, at least a portion of the indicator 896 may be color-coded to indicate the degree of deflection of the flexible end of the scope (e.g., at least a portion of the indicator 896 has a specific color, such as red, to indicate that the scope may not be able to be further deflected, or that the scope is in a maximum deflection state). In some embodiments, the degree of insertion of the scope or the degree of deflection of the scope is indicated by color without displaying the entire indicator 896. For example, a color indicator without a scope shape is used to indicate the degree of insertion of the scope or the degree of deflection of the scope. In some embodiments, the insertion degree of the scope or the deflection degree of the scope is indicated using an animation associated with the indicator 896. For example, the indicator 896 can be displayed to vibrate for at least a predefined time to indicate the insertion degree of the scope or the deflection degree of the scope (e.g., the scope has reached a fully inserted or fully deflected state).

[0275] In some embodiments, FIG. 28A to FIG. 20 D. FIG. 29A to FIG. 29D , FIG. 30A to FIG. 30F and FIG. 31A to FIG. 31FThe illustrated visual representation is provided with one or more audible signals (eg, dedicated sounds, such as a first sound indicating that the scope is fully inserted and a second sound different from the first sound indicating that the scope is fully deflected).

[0276] In some embodiments, when the surgeon activates the "Camera Clutch" mode, the FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D , FIG. 30A to FIG. 30F and FIG. 31A to FIG. 31F In some embodiments, the surgeon activates the "camera clutch" mode by pressing a camera pedal (e.g., foot pedal 244) of the surgeon console 240, or by activating a button on the console touch screen display, or by touching a point of arm at the bedside.

[0277] In some embodiments, FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D , FIG. 30A to FIG. 30C and FIG. 31A to FIG. 31F The illustrated visual representation is displayed based on a determination (eg, by the processor 380 ) that the robotic medical system is operating the flexible scope (eg, whether or not the flexible scope is being operated in a particular drive mode).

[0278] As mentioned above, FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D as well as FIG. 30A to FIG. 30C An exemplary user interface is illustrated. One of ordinary skill in the art will appreciate that one or more processors (e.g., central processing unit, graphics processing unit, accelerated processing unit, application specific integrated circuit, etc.) may be configured to generate electrical signals (e.g., video signals) so that one or more display devices may display the video. FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D and FIG. 30A to FIG. 30C The illustrated user interface.

[0279] F. Exemplary Process for Displaying Visual Indicators

[0280] FIG. 32A to FIG. 32F An example of a method of performing a task by a robotic system (e.g., Fig.21 and Fig. 22 Flowchart of method 900 performed by one or more processors (e.g., processor 380) of the illustrated robotic medical system 200, surgical robotic system, or robotic surgical platform, etc. The surgical robotic system includes a memory that stores instructions for execution by the one or more processors.

[0281] The robotic system includes a first robotic arm (eg, a robotic manipulator) coupled to a flexible scope (eg, a flexible scope 720, such as a flexible laparoscope or a flexible endoscope) (eg, Fig. 22210 - 1 in the robot arm).

[0282] The robotic system includes a viewer (e.g., a display device 232 located on tower 230, or a display device 242 included in physician console 240) for displaying a field of view of the surgical site derived from a flexible scope (e.g., a flexible scope coupled to a camera at a distal end of the scope) (the flexible scope introduces new degrees of freedom, allowing the surgeon to control the camera in more ways than when using a rigid scope).

[0283] The robotic system includes one or more processors (eg, processor 380 ), and memory (eg, memory 382 ) storing instructions for execution by the one or more processors.

[0284] The robotic system operates (902) the flexible scope in a particular mode (e.g., a drive mode) of a plurality of modes (e.g., a corresponding (drive) mode of a plurality of (drive) modes). In some embodiments, the plurality of modes includes an articulation mode, a track mode, and / or an automated insertion and retraction mode. In some embodiments, the plurality of modes also includes one or more additional modes.

[0285] The robotic system, based on determining (904) that the flexible scope is operating in a particular mode among the plurality of modes, provides a first visual indicator (e.g., Fig.28A , Fig.28B and Fig.28C Indicator 806, Fig.28D Indicator 822 in Fig.29A and Fig.29B Indicator 832, Fig.29C Indicator 844 in Fig.29D Indicator 854 in Fig. 30A and Fig. 30B Indicator 860 or Fig. 30C The first visual indicator indicates at least one of the following: a position (e.g., a first position) or an orientation (e.g., a first orientation) of the flexible endoscope relative to a reference position (e.g., a position and / or orientation of a distal end of the flexible endoscope).

[0286] In some embodiments, the flexible scope is inserted (906) into the patient through a first port (e.g., port 608), and the reference position is determined based on (e.g., corresponds to) the position of the first port. For example, in some embodiments, when the flexible scope is inserted through the port into the surgical site, the flexible scope is aligned with a line of sight from the cannula to the port.

[0287] In some embodiments, when the robotic system is operating the flexible scope in a first mode of multiple modes (e.g., an articulation drive mode), the robotic system (e.g., via the one or more processors) causes (908) a distal portion of the flexible scope (e.g., at the distal end 724 of the flexible scope 720, at a link 726 located at the distal portion of the flexible scope, at a joint 728 at the distal portion of the flexible scope, etc.) to move (e.g., articulation movement, tilting, rotation, pivoting movement, rotational movement, etc.) from a first position or a first orientation to a second position different from the first position or a second orientation different from the first orientation (e.g., movement relative to, close to, the distal portion of the flexible scope).

[0288] For example, movement of the distal portion of the flexible scope places the flexible scope in a second position (different from the first position) or a second orientation (different from the first orientation), and the first visual indicator is updated to indicate the second position or second orientation of the flexible scope (or its distal portion).

[0289] In some embodiments, movement of the distal end of the flexible scope includes movement in an upward direction, a downward direction, a leftward direction, and / or a rightward direction.

[0290] In some embodiments, the flexible scope includes a proximal end (eg, intermediate portion 722) that remains stationary when the distal portion is moved from a first position or orientation to a second position or orientation.

[0291] In some embodiments, the robotic system updates (910) the first visual indicator to indicate a second position or a second orientation of the distal portion of the flexible scope relative to a reference position (e.g., relative to a neutral position of the flexible scope) (e.g., when the flexible scope is fully aligned (straight) with the line of sight from the cannula to the port). Fig.28B , indicator 806 is updated to indicate a second position or second orientation of the distal portion of the flexible scope.

[0292] In some embodiments, the first visual indicator includes (912) one or more reference axes and a graphical element (e.g., a marker) positioned relative to the one or more reference axes to identify the direction and / or extent of movement. For example, indicator 806 includes reference axis 807 and a graphical element (e.g., a marker) 808 positioned relative to reference axis 807 to identify the direction and / or extent of movement.

[0293] In some embodiments, the robotic system provides (e.g., automatically, in real time) an electrical signal for presenting an updated graphical element (e.g., graphical element 811) based on determining (e.g., automatically, in real time) that a distal portion of the flexible endoscope has moved from a first position and / or first orientation to a second position and / or second orientation (914), the updated graphical element indicating a change in position and / or orientation of the flexible endoscope associated with the movement.

[0294] In some embodiments, the robotic system determines (916) (e.g., in real time) a shape of the flexible scope. For example, the robotic system can determine the shape of the flexible scope from a control signal provided to the flexible scope. The robotic system provides (918) an image (e.g., a rendered image, such as a 3D rendered image) (e.g., representation 812) that corresponds to (e.g., indicates, identifies, or includes) the determined shape.

[0295] In some embodiments, the image includes (920) two or more links (eg, link 814) corresponding to two or more links (eg, link 726) of the flexible scope.

[0296] In some embodiments, the robotic system provides an electrical signal for presenting an updated image (e.g., representation 813) indicating the changed shape based on determining (922) (e.g., in real time) that the shape of the flexible scope has changed from a first shape to a second shape (e.g., due to a change in the direction and / or extent of joint movement).

[0297] In some embodiments, the first visual indicator includes (924) a plurality of graphic elements (e.g., two bars (white portions of the bars), rectangles, ellipses, etc.). The plurality of graphic elements includes a first graphic element (e.g., graphic element 828-1) (e.g., a bar, a rectangle, a line-shaped object, an ellipse, etc.) and a second graphic element (e.g., graphic element 828-2) that is different from the first graphic element (e.g., a bar, a rectangle, a line-shaped object, an ellipse, etc.). The first graphic element represents a first component (e.g., direction and / or degree) of movement of the distal end of the flexible endoscope along a first direction (e.g., left or right). The second graphic element represents a second component (e.g., direction and / or degree) of movement of the distal end of the flexible endoscope along a second direction (e.g., upward or downward). The first direction is different from the second direction (e.g., the second direction is perpendicular to the first direction, and the first direction and the second direction are not parallel to each other).

[0298] In some embodiments, the robotic system provides an electrical signal for updating the position of the first graphical element based on determining (926) (e.g., in real time) that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude. In some embodiments, the robotic system provides an electrical signal for updating the position of the second graphical element based on determining (928) (e.g., in real time) that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude.

[0299] In some embodiments, the robotic system provides an electrical signal for updating the length (e.g., length 830-1) of a first graphical element (e.g., graphical element 828-1) based on determining (930) (e.g., in real time) that the movement of the flexible scope in a first direction (e.g., left or right) has changed from a first magnitude to a second magnitude. In some embodiments, the robotic system provides an electrical signal for updating the length (e.g., length 830-2) of a second graphical element (e.g., graphical element 828-2) based on determining (932) (e.g., in real time) that the movement of the flexible scope in a second direction (e.g., up or down) has changed from a third magnitude to a fourth magnitude.

[0300] In some embodiments, the field of view of the surgical site is derived from a camera coupled to the flexible endoscope (e.g., at a distal portion of the endoscope). The robotic system expands (934) a first visual indicator (e.g., an endoscope label) based on determining that the camera is active. For example, Fig.28A As illustrated, in some implementations, indicator 806 can have an initial size similar to indicator 804. The size of indicator 806 is expanded when a camera coupled to flexible scope 720 is active.

[0301] In some embodiments, when the robotic system is operating the flexible scope in a second mode (e.g., track drive mode) of a plurality of modes (e.g., track drive mode), the robotic system causes (936) the flexible scope to move around a target point (e.g., target point 734) (e.g., track point) in space.

[0302] In some embodiments, the target point is predefined (e.g., selected) by the surgeon. In some embodiments, the target point is predefined by the robotic system. In some embodiments, the target point is randomly selected by the robotic system. In some embodiments, the target point corresponds to the center of the field of view of the camera when the track drive mode is activated. In some embodiments, the target point is located offset from the center of the field of view of the camera.

[0303] In some embodiments, the first visual indicator includes (938) a first graphical element (e.g., graphical element 834) representing the target point (e.g., a solid circle, a hollow circle, a sphere, a cross mark).

[0304] In some embodiments, the robotic system provides (940) an electrical signal for superimposing a first graphical element on the displayed field of view (e.g., superimposing a first point on the displayed field of view of the surgical site). Fig.29A , a graphical element 834 is superimposed on a representation 840 of the field of view.

[0305] In some embodiments, the first visual indicator includes (942) a second graphical element (e.g., graphical element 836) representing a (e.g., direct) line of sight from the target point to the port of the flexible endoscope (e.g., straight line, solid line, dashed line).

[0306] In some embodiments, the first visual indicator includes a text element (e.g., element 838) that indicates the direction of the endoscope port (e.g., "Endoscope Port").

[0307] In some embodiments, when the port, camera, and target point are aligned (meaning the port is directly behind the camera), the second graphical element converges / merges with the first graphical element.

[0308] In some embodiments, the first graphical element (e.g., graphical element 834) has (e.g., is characterized by) a first size (e.g., diameter, length, width, area, etc.). The robotic system moves the first graphical element (e.g., graphical element 834) in response to changes in the depth of insertion of the flexible endoscope into the surgical site. Fig.29A and Fig.29B The size of the graphical element 834 in is adjusted (944) (e.g., changed, altered, etc.) (e.g., in real time) from a first size to a second size.

[0309] In some embodiments, the first size is predefined by one or more processors of the robotic system (eg, processor 380).

[0310] In some embodiments, the first size indicates a distance from the flexible scope to the target organ.

[0311] In some embodiments, the first size indicates the insertion distance of the scope (eg, the depth of the scope inserted into the body). For example, a larger size indicates that the scope is inserted further into the body, while a smaller size indicates that the scope is closer to the surface.

[0312] In some embodiments, the size of the first graphical element increases based on determining that the insertion depth has increased (e.g., the flexible scope is inserted further into the surgical site). In some embodiments, the size of the first graphical element decreases based on determining that the insertion depth has decreased (e.g., the flexible scope is being moved further away from the surgical site).

[0313] In some embodiments, the first visual indicator (e.g., indicator 844) includes (946) a second graphical element (e.g., graphical element 846) (e.g., a sphere, a globe, etc.) that represents the range of orbital motion of the flexible scope.

[0314] In some embodiments, the first visual indicator (e.g., indicator 844) includes (948) a third graphical element (e.g., graphical element 850) (e.g., a solid circle, a mark such as an "x", etc.) that represents the (e.g., real-time, current) position of the flexible endoscope.

[0315] In some embodiments, the robotic system provides (950) a method for placing a first graphical element, a second graphical element, and a third graphical element (e.g., Fig.29C The electrical signals of the graphic elements 846, 848 and 850 in the image are superimposed on the displayed field of view.

[0316] In some embodiments, the first visual indicator (e.g., indicator 854) includes one or more axes (e.g., three axes, such as axis 856) that represent the flexible scope relative to the one or more axes (e.g., Fig.29D In some embodiments, the one or more axes are positioned adjacent to a top edge, a bottom edge, a left edge, or a right edge of the user interface 802. In some embodiments, the one or more axes are positioned in a corner of the user interface 802.

[0317] In some embodiments, when the robotic system is operating the flexible endoscope in a third mode of the multiple modes (e.g., automated insertion and retraction mode), the robotic system causes (952) the flexible endoscope to be automatically (e.g., without manual force being applied to the flexible endoscope) inserted (e.g., from a cannula of a first robotic arm into a surgical site, inserted from a port into a surgical site, etc.) and / or (automatically) retracted (e.g., retracted from a surgical site into a port, retracted from a surgical site into a cannula of a first robotic arm, etc.).

[0318] In some embodiments, the robotic system determines (954) (e.g., in real time) the shape of the flexible scope during activation of the third mode (or during automatic insertion or automatic retraction). For example, the robotic system may determine the shape of the flexible scope based on a control signal provided to the flexible scope. Alternatively, the robotic system may determine the shape of the flexible scope based on a signal indicating the position or state of a corresponding joint of the flexible scope. The robotic system provides (956) an image (e.g., a rendered image, such as a 3D rendered image) corresponding to the determined shape (e.g., Fig. 30A and Fig. 30B Graphic element 861 in.

[0319] In some embodiments, the robotic system updates (958) (e.g., automatically, in real time) the image (e.g., Fig. 30B 862 in the graphic element).

[0320] In some embodiments, the first visual indicator (e.g., indicator 866) includes (960) a graphical element (e.g., graphical element 868) (e.g., a bar, a shading, etc.) indicating the extent of insertion (e.g., fully inserted, partially inserted, etc.) and / or the extent of retraction (e.g., fully retracted, partially retracted, etc.). In some embodiments, the first visual indicator (e.g., indicator 866) includes a graphical element (e.g., graphical element 872) indicating the articulation of the flexible scope (e.g., it is straight, articulated, etc.).

[0321] In some embodiments, the robotic system includes (962) a device coupled to a surgical tool (e.g., Fig.21 and Fig.24A Medical devices 212, Fig.25 The surgical tool 602 and Fig.26A and Fig.26B The surgical tools 704 and 708 in the apparatus may be provided by a second robotic arm (e.g., a robotic manipulator) (e.g., robotic arm 210-2).

[0322] In some embodiments, the robotic system provides (964) an electrical signal for presenting a second visual indicator (e.g., indicator 804) corresponding to the surgical tool on the viewer. In some embodiments, the second visual indicator identifies a medical instrument (e.g., surgical instrument) at the surgical site. In some embodiments, the second visual indicator 804 includes an association between the robotic arm and the corresponding medical instrument to which it is coupled. For example, in Fig.28A , indicator 804 - 2 includes the number “2,” which indicates that tool B is coupled to the second robotic arm of the robotic medical system.

[0323] In some embodiments, the first visual indicator includes (966) information identifying a first robotic arm coupled to the flexible scope (e.g., a robotic arm number). Fig.28A , Fig.28B and Fig.28C , the visual indicator 806 includes a number "3" identifying a robotic arm (eg, robotic arm 3) coupled to the flexible endoscope.

[0324] In some embodiments, the first visual indicator (e.g., FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D as well as FIG. 30A to FIG. 30C ) is displayed on the viewer simultaneously with the display of the field of view of the surgical site (968).

[0325] In some embodiments, the first visual indicator (e.g., FIG. 28A to FIG. 28D , Fig.29D and Fig. 30C The indicator (indicator in) is displayed (970) around the display of the field of view of the surgical site (e.g., displayed along the periphery of the display of the field of view of the surgical site).

[0326] In some embodiments, the first visual indicator (e.g., FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D as well as FIG. 30A to FIG. 30C An indicator in (972) is displayed on the display of the field of view of the surgical site.

[0327] In some embodiments, the viewer is (974) part of the surgeon's console (eg, the display device 242 is included in the physician console 240).

[0328] In some embodiments, the surgeon console includes (976) an input device (e.g., a user interface of the display device 242, an input device such as a button, a switch, a touch-sensitive surface or a gimbal, a foot pedal 244, etc.). The robotic system receives (978) input on the input device. The robotic system determines (980) that the flexible scope is operating in a particular mode based on the received input.

[0329] In some embodiments, the input device includes (982) a foot pedal (eg, foot pedal 244).

[0330] In some embodiments, the first visual indicator includes a graphical representation of the flexible scope projected in a non-front view graphic.

[0331] In some embodiments, the robotic system displays one or more reference user interface elements based on determining that the flexible scope is deflected.

[0332] In some embodiments, the robotic system ceases displaying the one or more reference user interface elements based on determining that the flexible scope is straight.

[0333] 3. Implement systems and terminology .

[0334] Fig.33 is a schematic diagram illustrating electronic components of a medical robotic system (e.g., a surgical robotic system) according to some embodiments.

[0335] The robotic medical system (e.g., surgical robotic system) includes one or more processors 380 with memory for executing any of the methods described herein (e.g., with respect to Fig.25 , Fig.26A , Fig.26B , FIG. 27A to FIG. 27D , FIG. 28A to FIG. 28D , FIG. 29A to FIG. 29D , FIG. 30A to FIG. 30C as well as FIG. 32A to FIG. 32F The one or more processors 380 also communicate with an input / output controller 384 (via a system bus or any suitable circuit). The input / output controller 384 receives sensor data from one or more sensors 388-1, 388-2, etc., and relays the sensor data to the one or more processors 380. The input / output controller 384 also receives instructions and / or data from the one or more processors 380 and relays the instructions and / or data to one or more actuators, such as first motors 387-1 and 387-2, etc. In some embodiments, the input / output controller 384 is connected to one or more actuator controllers 386 and provides instructions and / or data to at least a subset of the one or more actuator controllers 386, which in turn provides control signals to selected actuators. In some embodiments, one or more actuator controllers 386 are integrated with the input / output controller 384, and the input / output controller 384 provides control signals directly to one or more actuators 387 (in the absence of a separate actuator controller). Although FIG. 30 illustrates the presence of one actuator controller 386 (e.g., one actuator controller for the entire medical robotic system; in some embodiments, additional actuator controllers may be used (e.g., one actuator controller for each actuator, etc.). In some embodiments, the one or more processors 380 communicate with one or more displays 381 for displaying information as described herein.

[0336] It should be noted that, as used herein, the term "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0337] The function of determining whether a tool is inside or outside the surgical field of view provided by a camera or a scope and rendering one or more indicators representing the position or direction of one or more medical tools described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or a processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or a processor.

[0338] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0339] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0340] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0341] As used herein, the term “exemplary” means “serving as an example, instance, or illustration,” and does not necessarily indicate any preference or advantage of the example over any other configurations or implementations.

[0342] As used herein, the term "and / or" encompasses any combination of the listed elements. For example, "A, B, and / or C" includes the following element sets: only A, only B, only C, A and B without C, A and C without B, B and C without A, and combinations of all three elements A, B, and C.

[0343] The foregoing description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternative and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A robot system, comprising: a first robotic arm coupled to the flexible scope; a viewer for displaying a field of view of a surgical site derived from the flexible endoscope; one or more processors; and A memory configured to store instructions for execution by the one or more processors, the stored instructions comprising instructions for the following operations: operating the flexible speculum in a particular mode among a plurality of modes; as well as Based on determining that the flexible scope is operating in the particular mode among the plurality of modes, an electrical signal is provided for presenting a first visual indicator corresponding to the respective mode on the viewer, the first visual indicator indicating at least one of: a position or orientation of the flexible scope relative to a reference position.

2. The robot system according to claim 1, wherein: The flexible scope is inserted into a patient through a first port, and the reference position is determined based on a position of the first port.

3. The robot system according to any one of claims 1 to 2, wherein: When the robotic system is operating the flexible scope in a first mode of the plurality of modes, the robotic system causes movement of a distal portion of the flexible scope from a first position or orientation to a second position different from the first position or orientation different from the first orientation.

4. The robot system according to claim 3, wherein: The stored instructions include instructions for updating the first visual indicator to indicate the second position or the second orientation of the distal portion of the flexible scope relative to the reference position.

5. The robot system according to claim 4, wherein: The first visual indicator includes one or more reference axes and a graphical element positioned relative to the one or more reference axes to identify at least one of a direction or an extent of the movement.

6. The robot system according to any one of claims 4 to 5, wherein: The stored instructions also include instructions for: Based on determining that the distal portion of the flexible endoscope has performed at least one of the following operations: (i) moving from the first position to the second position or (ii) moving from the first orientation to the second orientation, providing an electrical signal for presenting an updated graphical element, wherein the updated graphical element indicates a change in at least one of the position or orientation of the flexible endoscope associated with the movement.

7. The robot system according to any one of claims 3 to 6, wherein: The stored instructions also include instructions for: determining a shape of the flexible speculum; and An image corresponding to the determined shape is provided.

8. The robot system according to claim 7, wherein: The stored instructions also include instructions for: Based on determining that the shape of the flexible scope has changed from a first shape to a second shape, an electrical signal is provided for presenting an updated image indicative of the changed shape.

9. The robot system according to any one of claims 7 to 8, wherein: The image includes two or more links corresponding to two or more links of the flexible scope.

10. The robot system according to any one of claims 3 to 9, wherein: The first visual indicator comprises a plurality of graphical elements, the plurality of graphical elements comprising a first graphical element and a second graphical element different from the first graphical element, wherein: The first graphical element represents a first component of the movement of the distal end of the flexible scope along a first direction; and The second graphical element represents a second component of the movement of the distal end of the flexible scope along a second direction, wherein the first direction is different from the second direction.

11. The robot system according to claim 10, wherein: The stored instructions also include instructions for: providing an electrical signal for updating a position of the first graphical element based on determining that the movement of the distal end of the flexible scope along the first direction has changed from a first magnitude to a second magnitude; as well as Based on determining that the movement of the distal end of the flexible scope along the second direction has changed from a third magnitude to a fourth magnitude, an electrical signal for updating the position of the second graphical element is provided.

12. The robot system according to any one of claims 10 to 11, wherein: The stored instructions also include instructions for: providing an electrical signal for updating the length of the first graphical element based on determining that the movement of the flexible scope in the first direction has changed from a first magnitude to a second magnitude; as well as Based on determining that the movement of the flexible scope in the second direction has changed from a third magnitude to a fourth magnitude, an electrical signal for updating the length of the second graphical element is provided.

13. The robotic system according to any one of claims 1 to 12, wherein: The field of view of the surgical site is obtained from a camera coupled to the flexible scope; and The stored instructions also include instructions for expanding the first visual indicator based on determining that the camera is active.

14. The robot system according to any one of claims 1 to 13, wherein: When the robotic system is operating the flexible scope in a second mode of the plurality of modes, the robotic system causes movement of the flexible scope around a target point in space.

15. The robot system according to claim 14, wherein: The first visual indicator includes a first graphical element representing the target point.

16. The robot system according to claim 15, wherein: The stored instructions also include instructions for providing an electrical signal for superimposing the first graphical element on the field of view.

17. The robot system according to any one of claims 15 to 16, wherein: The first visual indicator includes a second graphical element representing a line of sight from the target point to a port of the flexible scope.

18. The robotic system according to any one of claims 15 to 17, wherein: The first graphical element has a first size; and The stored instructions also include instructions for: The first graphical element is resized from the first size to a second size based on a change in the depth to which the flexible scope is inserted into the surgical site.

19. The robot system according to any one of claims 15 to 18, wherein: The first visual indicator includes a second graphical element representing a range of orbital motion of the flexible scope.

20. The robot system according to claim 19, wherein: The first visual indicator includes a third graphical element representing a position of the flexible scope.

21. The robot system according to claim 20, wherein: The stored instructions also include instructions for providing electrical signals for superimposing the first graphical element, the second graphical element, and the third graphical element on the field of view.

22. The robot system according to any one of claims 1 to 21, wherein: The first visual indicator comprises: one or more axes; and A graphical element representing a direction of insertion of the flexible scope relative to the one or more axes.

23. The robotic system according to any one of claims 1 to 22, wherein: When the robotic system is operating the flexible scope in a third mode among the plurality of modes, the robotic system automatically inserts and / or retracts the flexible scope.

24. The robot system according to claim 23, wherein: The stored instructions also include instructions for: determining a shape of the flexible speculum during activation of the third mode; and An image corresponding to the determined shape is provided.

25. The robotic system of claim 24, wherein the stored instructions further comprise instructions for updating the image when the flexible scope is automatically inserted or retracted.

26. The robotic system according to any one of claims 23 to 25, wherein: The first visual indicator includes a graphical element indicating at least one of: an extent of insertion, an extent of retraction, or articulation of the flexible speculum.

27. The robotic system according to any one of claims 1 to 26, further comprising: A second robotic arm is coupled to the surgical tool.

28. The robotic system of claim 27, wherein: The stored instructions also include instructions for providing an electrical signal for presenting a second visual indicator corresponding to the surgical tool on the viewer.

29. The robotic system according to any one of claims 1 to 28, wherein: The first visual indicator includes information identifying the first robotic arm coupled to the flexible scope.

30. The robotic system according to any one of claims 1 to 29, wherein: The first visual indicator is displayed on the viewer simultaneously with the display of the field of view of the surgical site.

31. The robot system according to claim 30, wherein: The first visual indicator is displayed around the display of the field of view of the surgical site.

32. The robotic system according to any one of claims 30 to 31, wherein: The first visual indicator is displayed on the display of the field of view of the surgical site.

33. The robotic system according to any one of claims 1 to 32, wherein: The viewer is part of the surgeon's console.

34. The robotic system of claim 33, wherein: The surgeon console includes an input device; and The stored instructions include instructions for: receiving input on the input device; and A determination is made based on the received input that the flexible scope is operating in the particular mode.

35. The robotic system of claim 34, wherein: The input device includes a foot pedal.

36. The robotic system according to any one of claims 1 to 35, wherein: The first visual indicator includes a graphically projected representation of the flexible scope.

37. The robot system according to claim 36, wherein: The stored instructions include instructions for displaying one or more reference user interface elements based on determining that the flexible scope is flexed.

38. The robotic system of claim 37, wherein: The stored instructions include instructions for ceasing to display the one or more reference user interface elements based on determining that the flexible scope is straight.

39. The robotic system according to any one of claims 36 to 38, wherein: The stored instructions include instructions for displaying one or more reference user interface elements based on a determination of an extent of insertion of the flexible speculum.

40. A robot system comprising: a first robotic arm coupled to the flexible scope; a viewer for displaying a field of view of a surgical site derived from the flexible endoscope; one or more processors; and A memory configured to store instructions for execution by the one or more processors, the stored instructions comprising instructions for the following operations: operating the flexible scope; and An electrical signal is provided for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of: a position or an orientation of the flexible scope relative to a reference position.

41. An electronic device in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope, the electronic device comprising: one or more processors; and A memory configured to store instructions for execution by the one or more processors, the stored instructions comprising instructions for the following operations: operating the flexible speculum in a particular mode among a plurality of modes; as well as Based on determining that the flexible scope is operating in the particular mode of the plurality of modes, providing an electrical signal for presenting a first visual indicator corresponding to the respective mode on the viewer, the first visual indicator indicating at least one of: The position or orientation of the flexible scope relative to a reference position.

42. An electronic device in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope, the electronic device comprising: one or more processors; and A memory configured to store instructions for execution by the one or more processors, the stored instructions comprising instructions for the following operations: operating the flexible scope; and An electrical signal is provided for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of: a position or an orientation of the flexible scope relative to a reference position.

43. A computer readable storage medium storing instructions for execution by one or more processors in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope, the stored instructions comprising instructions for: operating the flexible scope in a particular mode among a plurality of modes; and, Based on determining that the flexible scope is operating in the particular mode among the plurality of modes, an electrical signal is provided for presenting a first visual indicator corresponding to the respective mode on the viewer, the first visual indicator indicating at least one of: a position or orientation of the flexible scope relative to a reference position.

44. A computer readable storage medium storing instructions for execution by one or more processors in communication with a robotic system having a first robotic arm coupled to a flexible endoscope and a viewer for displaying a field of view of a surgical site derived from the flexible endoscope, the stored instructions comprising instructions for: operating the flexible scope; and An electrical signal is provided for presenting a first visual indicator on the viewer, the first visual indicator indicating at least one of: a position or an orientation of the flexible scope relative to a reference position.

Images