Physiologist console generating tactile vibrations for remote operation

By introducing tactile interface devices and feedback mechanisms into the medical system, fine control of the robot arm is achieved, which solves the problem of complexity of endoscopic operation in laparoscopic surgery and improves the accuracy and efficiency of the surgery.

CN118922143APending Publication Date: 2024-11-08AURIS HEALTH INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380029552.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-25
Filing Date
2023-03-17
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

When performing minimally invasive operations such as laparoscopic surgery, existing medical robot systems are difficult to provide high-precision endoscopic operation and fine control of the patient's internal organs, and the operation is complex, which affects the physician's work efficiency and accuracy.

Method used

A remote operation medical system enabled by robots is designed, including a haptic interface device, input sensor and processor. Through kinetic haptic feedback and vibrating haptic feedback, it realizes fine control and operation of the robot arm, combining the robot arm and surgical instruments. The collaborative work provides kinesthetic and vibration feedback to assist physician operations.

Benefits of technology

It improves the accuracy and efficiency of the surgery, reduces the operation complexity of the doctor, enhances the control ability of the endoscopy, reduces the risk of the surgery, and improves the reliability and safety of the surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118922143A_ABST
    Figure CN118922143A_ABST
Patent Text Reader

Abstract

A medical system may include a first haptic interface device, one or more input sensors, one or more processors, and a memory storing instructions. The instructions, when executed by the one or more processors, enable the one or more processors to: receive a first input signal from the one or more input sensors; sending a kinesthetic haptic feedback signal to the first haptic interface device based at least on the first input signal for kinesthetic haptic feedback; receiving a second input signal from one or more input sensors; and transmitting a vibro-tactile feedback signal to the first tactile interface device based at least on a second input signal for vibro-tactile feedback.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to controllers, and in particular to controllers for robotically enabled teleoperated systems, including medical systems. Background Art

[0002] Medical procedures such as laparoscopic surgery may involve accessing and visualizing the internal area of ​​a patient. During a laparoscopic procedure, a medical instrument may be inserted into the internal area through a laparoscopic portal.

[0003] In certain procedures, a robotic-enabled medical system may be used to control the insertion and / or manipulation of an instrument and its end effector. The robotic-enabled medical system may include a robotic arm or other instrument positioning device. The robotic-enabled medical system may also include a controller for controlling the positioning of the instrument during the procedure. Summary of the invention

[0004] In a first aspect, a robotic-enabled teleoperated medical system is described. The medical system includes a first tactile interface device, one or more input sensors, one or more processors, and a memory storing instructions. When the instructions are executed by one or more processors, the instructions cause the one or more processors to: receive a first input signal from the one or more input sensors; send a kinesthetic tactile feedback signal to the first tactile interface device based at least on the first input signal for kinesthetic tactile feedback; receive a second input signal from the one or more input sensors; and send a vibrotactile feedback signal to the first tactile interface device based at least on the second input signal for vibrotactile feedback.

[0005] The system may include one or more of the following features in any combination: (a) a first robotic arm; (b) wherein the first input signal is based on a difference between a master command and a movement of the first robotic arm; (c) a second robotic arm different from the first robotic arm; (d) wherein the second input signal is derived from a collision between the first robotic arm and the second robotic arm; (e) wherein the second input signal is derived from a criterion including a distance between the first robotic arm and the second robotic arm being less than a first distance threshold; (f) wherein the second input signal is derived from a criterion including a collision between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm; (g) wherein the second input signal is derived from a criterion including a distance between the first surgical instrument coupled to the first robotic arm and the second surgical instrument coupled to the second robotic arm being less than a second distance threshold; (h) the second input signal is derived from a criterion including a detection of a fault in the medical system; (i) the second input signal is derived from a criterion including a detection of a fault in the medical system; (i) wherein the first tactile interface device comprises a criterion for detecting a first tactile interface device at a predefined boundary; (j) a second tactile interface device that is different and separate from the first tactile interface device; (k) wherein the second input signal originates from a criterion comprising a collision between the first tactile interface device and the second tactile interface device; (l) wherein the second input signal originates from a criterion comprising a distance between the first tactile interface device and the second tactile interface device being less than a third distance threshold; (m) wherein the kinesthetic tactile feedback comprises a continuous force applied against the user input; (n) wherein the second input signal originates from a criterion comprising a plurality of events; and different tactile feedback signals are selected as vibration tactile feedback signals for corresponding events in the plurality of events; (o) wherein the vibration tactile feedback comprises a component having a frequency of at least 10 Hz; and / or (p) wherein the first tactile interface device further comprises a motor for providing a torque about a rolling axis of the first tactile interface device in response to a user input on the first tactile interface device.

[0006] On the other hand, a method is provided, which involves or includes: receiving a first input signal from one or more input sensors; sending a kinesthetic tactile feedback signal to a first tactile interface device based at least on the first input signal for kinesthetic tactile feedback; receiving a second input signal from the one or more input sensors; and sending a vibration tactile feedback signal to the first tactile interface device based at least on the second input signal for vibration tactile feedback.

[0007] In another aspect, a robotic-enabled teleoperated medical system is described. The medical system includes a first tactile interface device, one or more input sensors, one or more processors, and a memory storing instructions. When the instructions are executed by the one or more processors, the instructions cause the one or more processors to: receive a first input signal from the one or more input sensors; send a first tactile feedback signal to the first tactile interface device based at least on the first input signal for first tactile feedback; receive a second input signal from the one or more input sensors; and send a second tactile feedback signal to the first tactile interface device based at least on the second input signal for second tactile feedback. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosed aspects will hereinafter be described in conjunction with the accompanying drawings, which are provided to illustrate and not to limit the disclosed aspects, wherein like reference numerals represent like elements.

[0009] Figure 1 Embodiments of a cart-based robotic system arranged for use in diagnostic and / or therapeutic bronchoscopy procedures are illustrated.

[0010] Figure 2 Depicted Figure 1 Other aspects of the robotic system.

[0011] Figure 3 An example of a device arranged for ureteroscopy is shown. Figure 1 Implementation scheme of a robotic system.

[0012] Figure 4 An example of a device arranged for use in a vascular procedure Figure 1 Implementation scheme of a robotic system.

[0013] Figure 5 An embodiment of a table-based robotic system arranged for use in a bronchoscopy procedure is illustrated.

[0014] Figure 6 Provided Figure 5 Alternative view of the robotic system.

[0015] Figure 7 An exemplary system configured to stow a robotic arm is illustrated.

[0016] Figure 8 An embodiment of a table-based robotic system configured for use in a ureteroscopy procedure is illustrated.

[0017] Fig. 9 An embodiment of a table-based robotic system configured for laparoscopic procedures is illustrated.

[0018] Fig.10Illustrated Figures 5 to 9 Embodiment of a stage-based robotic system with pitch and tilt adjustment.

[0019] Fig.11 Provided Figures 5 to 10 Detailed illustration of the interface between the stage and the column of the stage-based robotic system.

[0020] Fig.12 An alternative embodiment of a table-based robotic system is illustrated.

[0021] Fig.13 Illustrated Fig.12 End view of a stage-based robotic system.

[0022] Fig.14 An end view of a table-based robotic system with a robotic arm attached thereto is illustrated.

[0023] Fig.15 An exemplary instrument driver is illustrated.

[0024] Fig.16 An exemplary medical instrument having a pair of instrument drivers is illustrated.

[0025] Fig.17 An alternative design of an instrument driver and instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument.

[0026] Fig.18 An instrument having an instrument-based insertion architecture is illustrated.

[0027] Fig.19 An exemplary controller is illustrated.

[0028] Fig. 20 Depicts a block diagram illustrating an example embodiment of estimating Figures 1 to 10 The position of one or more elements of the robotic system (such as Figures 16 to 18 A positioning system that determines the position of the device.

[0029] Fig.21A is a block diagram illustrating an embodiment of a robotically enabled medical system including a controller for a robotically enabled medical instrument.

[0030] Fig. 21B It is an example Fig.21A A block diagram of an embodiment of a controller that can be configured for mixed impedance and admittance control.

[0031] Fig. 22 is an isometric view of an embodiment of a gimbal for a controller.

[0032] Fig.23 A control system 200 is depicted according to some embodiments.

[0033] Fig.24 It shows how the output signals are combined according to some embodiments.

[0034] Fig.25A Depicted are example damping functions with different damping coefficients according to some embodiments.

[0035] Fig.25B Depicted are example control signals to a tactile interface device (HID) or output responses from the HID according to some embodiments.

[0036] Fig.25C Depicted are example control signals to or output responses from a HID according to some embodiments.

[0037] Fig.25D Depicted are some embodiments of Fig.25B and Fig.25C Example control signals of combinations of control signals shown.

[0038] Fig.26 is a flow chart illustrating an embodiment of a method for providing feedback to a user. DETAILED DESCRIPTION

[0039] 1. Overview.

[0040] Various aspects of the present disclosure may be integrated into a robotically enabled medical system that is capable of performing a variety of medical procedures, including both minimally invasive procedures such as laparoscopy, and non-invasive procedures such as endoscopy. In the endoscopy procedure, the system may be able to perform bronchoscopy, ureteroscopy, gastroscopy, etc.

[0041] In addition to performing a wide range of procedures, the system may provide additional benefits such as enhanced imaging and guidance to assist the physician. Additionally, the system may provide the physician with the ability to perform procedures from an ergonomic position without the need for awkward arm motions and positions. Additionally, the system may provide the physician with the ability to perform procedures with improved ease of use such that one or more of the system's instruments may be controlled by a single user.

[0042] For illustrative purposes, various embodiments will be described below in conjunction with the accompanying drawings. It should be understood that many other specific implementations of the disclosed concepts are possible, and various advantages can be achieved using the disclosed specific implementations. Titles are included herein for reference and to help locate the various sections. These titles are not intended to limit the scope of the concepts described therein. Such concepts may have applicability throughout the specification.

[0043] A. Robotic system - cart.

[0044] Robotically enabled medical systems can be configured in a variety of ways, depending on the specific procedure. Figure 1 An embodiment of a cart-based, robotic-enabled system 10 arranged for use in a diagnostic and / or therapeutic bronchoscopy procedure is illustrated. During a bronchoscopy procedure, the robotic system 10 may include a cart 11 having one or more robotic arms 12 to deliver a medical instrument such as a steerable endoscope 13 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice access point (i.e., the mouth of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, the cart 11 may be positioned proximate to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 12 may be actuated to position the bronchoscope relative to the access point. When performing gastrointestinal (GI) procedures using a gastroscope (a specialized endoscope for GI procedures), the bronchoscope may also be utilized. Figure 1 The arrangement in . Figure 2 An exemplary embodiment of a cart is depicted in greater detail.

[0045] Continue to refer Figure 1 , once the cart 11 is correctly positioned, the robotic arm 12 can insert the steerable endoscope 13 into the patient robotically, manually, or in a combination thereof. As shown, the steerable endoscope 13 may include at least two telescopic parts, such as an inner guide portion and an outer sheath portion, each of which is coupled to a separate instrument driver from a set of instrument drivers 28, each of which is coupled to the distal end of a separate robotic arm. This linear arrangement of the instrument drivers 28 that facilitates coaxial alignment of the guide portion with the sheath portion produces a "virtual track" 29, which can be repositioned in space by manipulating one or more robotic arms 12 to different angles and / or positions. The virtual tracks described herein are depicted using dashed lines in the accompanying drawings, and therefore the dashed lines do not depict any physical structure of the system. The translation of the instrument driver 28 along the virtual track 29 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 13 from the patient. The angle of the virtual track 29 can be adjusted, translated, and pivoted based on clinical applications or physician preferences. For example, in bronchoscopy, the angle and position of the virtual track 29 as shown represents a compromise between providing the physician with access to the endoscope 13 while minimizing the friction caused by bending the endoscope 13 into the patient's mouth.

[0046] After insertion, the endoscope 13 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 13 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of a separate instrument driver 28 also allows the guide portion and the sheath portion to be driven independently of each other.

[0047] For example, the endoscope 13 can be guided to deliver a biopsy needle to a target, such as, for example, a lesion or nodule in a patient's lung. The needle can be deployed downward along a working channel that extends the length of the endoscope to obtain a tissue sample to be analyzed by a pathologist. Depending on the pathological results, additional tools can be deployed downward along the working channel of the endoscope for additional biopsies. After identifying that the nodule is malignant, the endoscope 13 can deliver tools through the endoscope to remove potential cancerous tissue. In some cases, diagnostic and therapeutic treatments may need to be delivered in separate procedures. In these cases, the endoscope 13 can also be used to deliver a benchmark to "mark" the location of the target nodule. In other cases, diagnostic and therapeutic treatments can be delivered during the same procedure.

[0048] The robotic system 10 may also include a movable tower 30 that may be connected to the cart 11 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 11. Placing such functionality in the tower 30 allows for a smaller form factor cart 11 that may be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 30 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 30 may be stowed in a remote location to be out of the way during a procedure.

[0049] To support the robotic system described above, the tower 30 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 30 or in the cart 11, the execution of these instructions may control the entire system or a subsystem thereof. For example, when executed by a processor of a computer system, the instructions may cause components of the robotic system to actuate associated brackets and arm mounts, actuate a robotic arm, and control a medical device. For example, in response to receiving a control signal, a motor in a joint of the robotic arm may position the arm into a particular posture.

[0050] The tower 30 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to systems that may be deployed through the endoscope 13. These components may also be controlled using the computer system of the tower 30. In some embodiments, irrigation and aspiration capabilities may be delivered directly to the endoscope 13 via separate cables.

[0051] The tower 30 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 11, resulting in a smaller, more mobile cart 11.

[0052] The tower 30 may also include support equipment for sensors deployed throughout the robotic system 10. For example, the tower 30 may include optoelectronic equipment for detecting, receiving, and processing data received from optical sensors or cameras throughout the robotic system 10. In conjunction with the control system, such optoelectronic equipment may be used to generate real-time images for display in any number of consoles deployed throughout the system (including display in the tower 30). Similarly, the tower 30 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 30 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0053] The tower 30 may also include a console 31 in addition to other consoles available in the rest of the system (e.g., a console mounted on top of a cart). The console 31 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in the robotic system 10 is typically designed to provide both preoperative information and real-time information for robotic control and procedures, such as navigation and positioning information for the endoscope 13. When the console 31 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the health or vital signs of the patient and the operation of the system, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, the console 31 is housed in a body separate from the tower 30.

[0054] The tower 30 may be coupled to the cart 11 and the endoscope 13 via one or more cables or connections (not shown). In some embodiments, support functions from the tower 30 may be provided to the cart 11 via a single cable, thereby simplifying and eliminating clutter in the operating room. In other embodiments, specific functions may be coupled in separate wiring and connections. For example, while power may be provided to the cart via a single cable, support for control, optics, fluidics, and / or navigation may also be provided via separate cables.

[0055] Figure 2 Provided by Figure 1Detailed illustration of an embodiment of a cart for a cart-based robotic-enabled system is shown. The cart 11 generally includes an elongated support structure 14 (commonly referred to as a "column"), a cart base 15, and a console 16 at the top of the column 14. The column 14 may include one or more brackets, such as for supporting one or more robotic arms 12 ( Figure 2 The cradle 17 (alternatively "arm support") may include a separately configurable arm mount that rotates along a vertical axis to adjust the base of the robotic arm 12 to better position it relative to the patient. The cradle 17 also includes a cradle interface 19 that allows the cradle 17 to translate vertically along the column 14.

[0056] The bracket interface 19 is connected to the column 14 through slots, such as slot 20, which are positioned on opposite sides of the column 14 to guide the vertical translation of the bracket 17. The slot 20 contains a vertical translation interface to position and maintain the bracket at various vertical heights relative to the cart base 15. The vertical translation of the bracket 17 allows the cart 11 to adjust the reach of the robotic arm 12 to meet a variety of table heights, patient sizes, and physician preferences. Similarly, the separately configurable arm mounts on the bracket 17 allow the robotic arm base 21 of the robotic arm 12 to be angled in a variety of configurations.

[0057] In some embodiments, the slot 20 may be supplemented with a slot cover that is flush and parallel to the slot surface to prevent dust and fluid from entering the internal cavity of the column 14 and the vertical translation interface when the carriage 17 is vertically translated. The slot cover can be deployed by a pair of spring reels positioned near the vertical top and bottom of the slot 20. The cover is coiled in the reel before deployment, and only extends and retracts from the coiled state of the cover as the carriage 17 vertically translates up and down. When the carriage 17 translates toward the reel, the spring load of the reel provides the force to retract the cover into the reel, while also maintaining a tight seal when the carriage 17 translates away from the reel. The cover can be connected to the carriage 17 using, for example, a bracket in the carriage interface 19 to ensure appropriate extension and retraction of the cover when the carriage 17 translates.

[0058] Column 14 may internally include mechanisms such as gears and motors designed to mechanized translate carriage 17 using a vertically aligned lead screw in response to control signals generated in response to user input (eg, input from console 16 ).

[0059] The robotic arm 12 may generally include a robotic arm base 21 and an end effector 22 separated by a series of links 23, the series of links being connected by a series of joints 24, each joint including an independent actuator, each actuator including an independently controllable motor. Each independently controllable joint represents an independent degree of freedom available to the robotic arm. Each of the arms 12 has seven joints and therefore provides seven degrees of freedom. Multiple joints result in multiple degrees of freedom, thereby allowing "redundant" degrees of freedom. The redundant degrees of freedom allow the robotic arm 12 to position its corresponding end effector 22 at a specific position, orientation, and trajectory in space using different link positions and joint angles. This allows the system to position and guide the medical device from a desired point in space, while allowing the physician to move the arm joint to a clinically advantageous position away from the patient to create greater access while avoiding arm collisions.

[0060] The cart base 15 balances the weight of the column 14, bracket 17, and arm 12 on the floor. Thus, the cart base 15 houses heavier components such as electronics, motors, power supplies, and components that enable the cart to be moved and / or fixed. For example, the cart base 15 includes rollable wheel-shaped casters 25 that allow the cart to be easily moved around the room before a procedure. After reaching the appropriate position, the casters 25 can be secured using wheel locks to keep the cart 11 in place during the procedure.

[0061] The console 16 positioned at the vertical end of the column 14 allows both a user interface and a display screen (or a dual-purpose device, such as, for example, a touch screen 26) for receiving user input to provide both preoperative and intraoperative data to the physician user. Potential preoperative data on the touch screen 26 may include preoperative planning, navigation and mapping data derived from a preoperative computerized tomography (CT) scan and / or a record from a preoperative patient interview. The intraoperative data on the display may include optical information provided from tools, sensors and coordinate information from sensors and important patient statistics, such as respiration, heart rate and / or pulse. The console 16 may be positioned and tilted to allow the physician to approach the console from the side of the column 14 opposite to the bracket 17. From this position, the physician can observe the console 16, the robot arm 12 and the patient while operating the console 16 from behind the cart 11. As shown, the console 16 also includes a handle 27 for helping to manipulate and stabilize the cart 11.

[0062] Figure 3An embodiment of a robotically enabled system 10 arranged for ureteroscopy is illustrated. In a ureteroscopy procedure, a cart 11 may be positioned to deliver a ureteroscope 32 (a procedure-specific endoscope designed to traverse a patient's urethra and ureters) to the patient's lower abdominal region. In a ureteroscopy, it may be desirable to align the ureteroscope 32 directly with the patient's urethra to reduce friction and forces on sensitive anatomical structures in this area. As shown, the cart 11 may be aligned at the foot of the table to allow the robotic arm 12 to position the ureteroscope 32 for direct linear access to the patient's urethra. The robotic arm 12 may insert the ureteroscope 32 from the foot of the table along a virtual track 33 through the urethra directly into the patient's lower abdomen.

[0063] After insertion into the urethra, the ureteroscope 32 can be navigated into the bladder, ureters, and / or kidneys for diagnostic and / or therapeutic applications using similar control techniques as in bronchoscopy. For example, the ureteroscope 32 can be guided into the ureters and kidneys to break up accumulated kidney stones using a laser or ultrasonic lithotripsy device deployed down the working channel of the ureteroscope 32. After lithotripsy is complete, the resulting stone fragments can be removed using a basket deployed down the ureteroscope 32.

[0064] Figure 4 An embodiment of a robotic-enabled system similarly arranged for vascular procedures is illustrated. In a vascular procedure, the robotic system 10 may be configured so that the cart 11 can deliver a medical device 34, such as a steerable catheter, to an access point in the femoral artery of the patient's leg. The femoral artery presents both a larger diameter for navigation and a relatively less circuitous and tortuous path to the patient's heart, which simplifies navigation. As in a ureteroscopy procedure, the cart 11 may be positioned toward the patient's leg and lower abdomen to allow the robotic arm 12 to provide a virtual track 35 for direct linear access to the femoral artery access point in the patient's thigh / hip region. After insertion into the artery, the medical device 34 may be guided and inserted by translating the instrument drive 28. Alternatively, the cart may be positioned around the patient's upper abdomen to reach alternative vascular access points, such as, for example, the carotid and brachial arteries near the shoulder and wrist.

[0065] B. Robotic system - unit.

[0066] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment within the operating room by removing a cart, which allows for greater access to the patient. Figure 5An embodiment of such a robotically enabled system arranged for use in a bronchoscopy procedure is illustrated. System 36 includes a support structure or column 37 for supporting a platform 38 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 39 of system 36 includes an instrument drive 42 designed to manipulate an elongated medical instrument, such as a slender medical instrument, through or along a virtual track 41 formed by the linear alignment of the instrument drive 42. Figure 5 In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of the patient by placing the emitter and detector around the table 38.

[0067] Figure 6 An alternative view of the system 36 without the patient and medical device for discussion purposes is provided. As shown, the column 37 may include one or more brackets 43 shown as annular in the system 36, on which one or more robot arms 39 may be based. The bracket 43 may translate along a vertical column interface 44 extending along the length of the column 37 to provide different vantage points from which the robot arm 39 may be positioned to reach the patient. The bracket 43 may rotate around the column 37 using a mechanical motor positioned within the column 37 to allow the robot arm 39 to access multiple sides of the table 38, such as, for example, both sides of the patient. In an embodiment with multiple brackets, the brackets may be positioned individually on the column and may translate and / or rotate independently of the other brackets. Although the bracket 43 does not need to be around the column 37 or even circular, the annular shape as shown facilitates the rotation of the bracket 43 around the column 37 while maintaining structural balance. The rotation and translation of the bracket 43 allows the system to align medical devices such as endoscopes and laparoscopes to different entry points on the patient. In other embodiments (not shown), the system 36 may include a patient table or bed with an adjustable arm support in the form of a pole or track that extends alongside the patient table or bed. One or more robotic arms 39 may be attached to the adjustable arm support (e.g., via a shoulder with an elbow joint) that may be vertically adjusted. By providing vertical adjustment, the robotic arms 39 are advantageously able to be compactly stowed beneath the patient table or bed and subsequently raised during a procedure.

[0068] The arm 39 may be mounted on the carriage via a set of arm mounts 45 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 39. Additionally, the arm mounts 45 may be positioned on the carriage 43 such that when the carriage 43 is appropriately rotated, the arm mounts 45 may be positioned on the same side of the table 38 (e.g., Figure 6 As shown), on the opposite side of the platform 38 (as shown Fig. 9 shown) or on an adjacent side of stage 38 (not shown).

[0069] The column 37 provides structural support for the stage 38 and provides a path for the vertical translation of the carriage. Internally, the column 37 can be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage based on the lead screw. The column 37 can also transmit power and control signals to the carriage 43 and the robotic arm 39 mounted thereon.

[0070] The base 46 has Figure 2 The cart base 15 in the illustrated cart 11 functions similarly, accommodating heavier components to balance the table / bed 38, column 37, bracket 43, and robotic arm 39. The table base 46 may also incorporate rigid casters to provide stability during the procedure. Casters deployed from the bottom of the table base 46 may extend in opposite directions on both sides of the base 46 and retract when the system 36 needs to be moved.

[0071] continue Figure 6 , the system 36 may also include a tower (not shown) that divides the functions of the system 36 between the table and the tower to reduce the form factor and volume of the table. As in the previously disclosed embodiments, the tower can provide various support functions to the table, such as processing, computing and control capabilities, power, fluid and / or optical and sensor processing. The tower can also be movable to be positioned away from the patient, thereby improving the physician's access and eliminating clutter in the operating room. In addition, placing the components in the tower allows more storage space in the table base for potential stowage of the robotic arm. The tower may also include a main controller or console that provides a user interface such as a keyboard and / or suspension for user input, and a display screen (or touch screen) for preoperative and intraoperative information such as real-time imaging, navigation and tracking information. In some embodiments, the tower may also include a holder for a gas tank to be used for insufflation.

[0072] In some embodiments, the table base can fold up and store the robotic arm when not in use. Figure 7 A system 47 for stowing a robotic arm in an embodiment of a table-based system is illustrated. In the system 47, a carriage 48 can be translated vertically into a base 49 to stow a robotic arm 50, an arm mount 51, and the carriage 48 within the base 49. A base cover 52 can be translated and retracted open to deploy the carriage 48, the arm mount 51, and the arm 50 around a column 53, and closed to stow the carriage, the arm mount, and the arm to protect them when not in use. The base cover 52 can be sealed along the edges of its opening with a membrane 54 to prevent dust and fluids from entering when closed.

[0073] Figure 8An embodiment of a robotically enabled table-based system configured for a ureteroscopy procedure is illustrated. During ureteroscopy, the table 38 may include a rotating portion 55 for positioning the patient at an angle to the column 37 and the table base 46. The rotating portion 55 may rotate or pivot about a pivot point (e.g., located below the patient's head) so as to position the bottom portion of the rotating portion 55 away from the column 37. For example, the pivoting of the rotating portion 55 allows a C-arm (not shown) to be positioned above the patient's lower abdomen without competing for space with the column (not shown) below the table 38. By rotating the bracket (not shown) about the column 37, the robotic arm 39 can insert the ureteroscope 56 directly into the patient's groin area along the virtual track 57 to reach the urethra. During ureteroscopy, stirrups 58 may also be fixed to the rotating portion 55 of the table 38 to support the position of the patient's legs during the procedure and allow full access to the patient's groin area.

[0074] In a laparoscopic procedure, minimally invasive instruments may be inserted into the patient's anatomical structure through one or more small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instrument includes an elongated rigid member, such as a shaft, for accessing the patient's anatomical structure. After the patient's abdominal cavity is inflated, the instrument may be guided to perform a surgical or medical task, such as grasping, cutting, ablating, suturing, etc. In some embodiments, the instrument may include a scope, such as a laparoscope. Fig. 9 Embodiments of a robotically enabled table-based system configured for laparoscopic procedures are illustrated. Fig. 9 As shown, the carriage 43 of the system 36 can be rotated and vertically adjusted to position the pair of robotic arms 39 on opposite sides of the table 38 so that the arm mounts 45 can be used to position instruments 59 through minimal incisions on both sides of the patient to reach his / her abdominal cavity.

[0075] To accommodate laparoscopic procedures, the robotic-enabled table system can also tilt the platform to a desired angle. Fig.10 Embodiments of a robotically enabled medical system with pitch or tilt adjustment are illustrated. Fig.10 As shown, system 36 can accommodate the tilt of table 38 to position one portion of the table at a greater distance from the floor than another portion. Additionally, arm mount 45 can be rotated to match the tilt so that arm 39 maintains the same planar relationship with table 38. To accommodate steeper angles, column 37 can also include telescoping portion 60 that allows vertical extension of column 37 to prevent table 38 from contacting the floor or colliding with base 46.

[0076] Fig.11A detailed illustration of the interface between the table 38 and the column 37 is provided. The pitch rotation mechanism 61 can be configured to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom. The pitch rotation mechanism 61 can be implemented by positioning orthogonal axes 1, 2 at the column-to-table interface, each axis being actuated by a separate motor 3, 4 in response to an electrical pitch angle command. Rotation along one screw 5 will enable tilt adjustment in one axis 1, while rotation along another screw 6 will enable tilt adjustment along another axis 2. In some embodiments, a spherical joint can be used to change the pitch angle of the table 38 relative to the column 37 in multiple degrees of freedom.

[0077] For example, pitch adjustment is particularly useful when attempting to position the table in the Trendelenburg position (i.e., positioning the patient's lower abdomen at a higher orientation from the floor than the patient's upper / upper abdomen) for lower abdominal surgery. The Trendelenburg position allows the patient's internal organs to glide toward his / her upper abdomen by gravity, thereby clearing the abdominal cavity to allow minimally invasive tools to be entered and to perform lower abdominal surgical or medical procedures, such as laparoscopic prostatectomy.

[0078] Fig.12 and Fig.13 An isometric view and an end view of an alternative embodiment of a table-based surgical robotic system 100 are illustrated. The surgical robotic system 100 includes a table 101 that can be configured to support one or more robotic arms (see, e.g., Fig.14 ) is provided with one or more adjustable arm supports 105 of the table 101. In the illustrated embodiment, a single adjustable arm support 105 is shown, but additional arm supports may be provided on opposite sides of the table 101. The adjustable arm support 105 may be configured so that it can be moved relative to the table 101 to adjust and / or change the orientation of the adjustable arm support 105 and / or any robotic arm mounted thereto relative to the table 101. For example, the adjustable arm support 105 may be adjusted in one or more degrees of freedom relative to the table 101. The adjustable arm support 105 provides high flexibility to the system 100, including the ability to easily stow the one or more adjustable arm supports 105 and any robotic arm attached thereto below the table 101. The adjustable arm support 105 may be raised from a stowed orientation to a position below the upper surface of the table 101. In other embodiments, the adjustable arm support 105 can be raised from the stowed position to a position above the upper surface of the table 101 .

[0079] The adjustable arm support 105 may provide several degrees of freedom, including lift, lateral translation, tilt, etc. Fig.12 and Fig.13 In the exemplary embodiment of the present invention, the arm support 105 is configured to have four degrees of freedom. Fig.12102a of the table 101. The first degree of freedom allows the adjustable arm support 105 to be adjusted in the z-direction ("Z lift"). For example, the adjustable arm support 105 may include a bracket 109 that is configured to move upward or downward along or relative to the column 102a of the support table 101. The second degree of freedom allows the adjustable arm support 105 to be tilted. For example, the adjustable arm support 105 may include a rotational joint that allows the adjustable arm support 105 to be aligned with the bed in a Trendelenburg position. The third degree of freedom allows the adjustable arm support 105 to "pivot up," which can be used to adjust the distance between one side of the table 101 and the adjustable arm support 105. The fourth degree of freedom allows the adjustable arm support 105 to translate along the longitudinal length of the table.

[0080] Fig.12 and Fig.13 The surgical robot system 100 in the embodiment of the present invention may include a table supported by a column 102a mounted to a base 103. The base 103 and the column 102a support the table 101 relative to the support surface. The floor axis 131 and the support axis 133 are located at Fig.13 Shown in.

[0081] The adjustable arm support 105 can be mounted to the column 102a. In other embodiments, the arm support 105 can be mounted to the table 101 or the base 103. The adjustable arm support 105 can include a bracket 109, a rod or track connection 111, and a rod or track 107. In some embodiments, one or more robotic arms mounted to the track 107 can translate and move relative to each other.

[0082] The bracket 109 may be attached to the column 102a via a first joint 113 that allows the bracket 109 to move relative to the column 102a (e.g., such as up and down along a first or vertical axis 123). The first joint 113 may provide a first degree of freedom ("Z lift") to the adjustable arm support 105. The adjustable arm support 105 may include a second joint 115 that provides a second degree of freedom (tilt) to the adjustable arm support 105. The adjustable arm support 105 may include a third joint 117a that provides a third degree of freedom ("upward pivot") to the adjustable arm support 105. Additional joints 119 may be provided (in Fig.13 ), the additional joint mechanically constrains the third joint 117a to maintain the orientation of the track 107 when the track connector 111 rotates about the third axis 127. The adjustable arm support 105 may include a fourth joint 121, which may provide the adjustable arm support 105 with a fourth degree of freedom (translation) along a fourth axis 129.

[0083] Fig.14An end view of a surgical robotic system 140A having two adjustable arm supports 105A, 105B mounted on opposite sides of a table 101 is illustrated. A first robotic arm 142A is attached to a rod or track 107A of the first adjustable arm support 105B. The first robotic arm 142A includes a base 144A attached to the track 107A. The distal end of the first robotic arm 142A includes an instrument drive mechanism 146A that can be attached to one or more robotic medical instruments or tools. Similarly, the second robotic arm 142B includes a base 144B attached to the track 107B. The distal end of the second robotic arm 142B includes an instrument drive mechanism 146B. The instrument drive mechanism 146B can be configured to be attached to one or more robotic medical instruments or tools.

[0084] In some embodiments, one or more of the robotic arms 142A, 142B include an arm with seven or more degrees of freedom. In some embodiments, one or more of the robotic arms 142A, 142B may include eight degrees of freedom, including an insertion axis (including 1 degree of freedom for insertion), a wrist (including 3 degrees of freedom for wrist pitch, yaw, and roll), an elbow (including 1 degree of freedom for elbow pitch), a shoulder (including 2 degrees of freedom for shoulder pitch and yaw), and a base 144A, 144B (including 1 degree of freedom for translation). In some embodiments, the insertion degrees of freedom may be provided by the robotic arms 142A, 142B, while in other embodiments, the instrument itself provides insertion via an instrument-based insertion architecture.

[0085] C. Device drivers and interfaces.

[0086] The end effector of the system's robotic arm includes (i) an instrument driver (alternatively referred to as an "instrument drive mechanism" or "instrument device manipulator") that incorporates an electromechanical device for actuating a medical device; and (ii) a removable or detachable medical device that may not contain any electromechanical components, such as a motor. This dichotomy may be driven by the need to sterilize medical instruments used in medical procedures; and the inability to adequately sterilize expensive capital equipment due to its complex mechanical assemblies and sensitive electronics. Therefore, the medical device may be designed to be disassembled, removed, and interchanged from the instrument driver (and therefore from the system) for separate sterilization or disposal by a physician or physician's staff. In contrast, the instrument driver does not need to be changed or sterilized, and can be covered for protection.

[0087] Fig.15An example instrument driver is illustrated. The instrument driver 62, positioned at the distal end of the robotic arm, includes one or more drive units 63 arranged in parallel axes to provide controlled torque to a medical device via a drive shaft 64. Each drive unit 63 includes a separate drive shaft 64 for interacting with the instrument, a gear head 65 for converting motor shaft rotation into a desired torque, a motor 66 for generating the drive torque, an encoder 67 for measuring the speed of the motor shaft and providing feedback to the control circuit, and a control circuit 68 for receiving control signals and actuating the drive unit. Each drive unit 63 is independently controlled and motorized, and the instrument driver 62 can provide multiple (e.g., Fig.15 In operation, the control circuit system 68 will receive the control signal, transmit the motor signal to the motor 66, compare the resulting motor speed measured by the encoder 67 with the desired speed, and modulate the motor signal to generate the desired torque.

[0088] For the procedures requiring a sterile environment, the robotic system can be combined with a drive interface, such as a sterile adapter connected to a sterile cover, which is located between the instrument driver and the medical device. The main purpose of the sterile adapter is to transfer angular motion from the drive shaft of the instrument driver to the drive input of the instrument, while maintaining the physical separation between the drive shaft and the drive input and thus maintaining sterility. Therefore, an exemplary sterile adapter may include a series of rotational inputs and rotational outputs intended to cooperate with the drive shaft of the instrument driver and the drive input on the instrument. The sterile cover composed of a thin flexible material (such as transparent or translucent plastic) connected to the sterile adapter is designed to cover capital equipment, such as instrument drivers, robot arms and carts (in cart-based systems) or tables (in tables-based systems). The use of the cover will allow capital equipment to be positioned near the patient while still being located in an area that does not require sterilization (i.e., non-sterile area). On the other side of the sterile cover, medical devices can dock with patients in areas that require sterilization (i.e., sterile areas).

[0089] D.Medical devices.

[0090] Fig.16An example medical device with a paired instrument driver is illustrated. Similar to other instruments designed for use with a robotic system, the medical device 70 includes an elongated shaft 71 (or elongated body) and an instrument base 72. The instrument base 72, also referred to as an "instrument handle" due to its intended design for manual interaction by a physician, can typically include a rotatable drive input 73 (e.g., a socket, pulley, or reel) that is designed to mate with a drive output 74 of a drive interface on an instrument driver 75 extending through a distal end of a robotic arm 76. When physically connected, latched, and / or coupled, the mating drive input 73 of the instrument base 72 can share an axis of rotation with the drive output 74 in the instrument driver 75 to allow torque to be transferred from the drive output 74 to the drive input 73. In some embodiments, the drive output 74 may include a spline that is designed to mate with a socket on the drive input 73.

[0091] The elongated shaft 71 is designed to be delivered through an anatomical opening or lumen (e.g., as in endoscopy) or through a minimally invasive incision (e.g., as in laparoscopy). The elongated shaft 71 can be flexible (e.g., having properties similar to an endoscope) or rigid (e.g., having properties similar to a laparoscope), or a customized combination of both flexible and rigid portions. When designed for laparoscopy, the distal end of the rigid elongated shaft can be connected to an end effector extending from an articulated wrist formed by a connecting fork having at least one degree of freedom and a surgical tool or medical instrument (such as, for example, a grasper or scissors) that can be actuated based on forces from a tendon when the drive input rotates in response to a torque received from the drive output 74 of the instrument driver 75. When designed for endoscopy, the distal end of the flexible elongated shaft can include a manipulable or controllable curved segment that articulates and bends based on a torque received from the drive output 74 of the instrument driver 75.

[0092] The torque from the instrument driver 75 is transmitted to the elongated shaft 71 down the shaft 71 using tendons. These individual tendons (such as pull wires) can be individually anchored to individual drive inputs 73 within the instrument handle 72. From the handle 72, the tendons are guided downward along one or more pull cavities of the elongated shaft 71 and anchored at the distal portion of the elongated shaft 71, or in the wrist at the distal portion of the elongated shaft. During surgical procedures such as laparoscopic, endoscopic, or hybrid procedures, these tendons can be connected to a distally mounted end effector, such as a wrist, a grasper, or scissors. In such an arrangement, the torque applied to the drive input 73 transmits tension to the tendons, causing the end effector to actuate in some manner. In some embodiments, during a surgical procedure, the tendons can cause the joint to rotate around an axis, causing the end effector to move in one direction or another. Alternatively, the tendons can be connected to one or more jaws of a grasper at the distal end of the elongated shaft 71, wherein the tension from the tendons causes the grasper to close.

[0093] In endoscopy, the tendon may be coupled to a bending or articulation segment located along the elongated shaft 71 (e.g., at the distal end) via an adhesive, a control ring, or other mechanical fastener. When fixedly attached to the distal end of the bending segment, torque applied to the drive input 73 will be transmitted downward along the tendon, causing the softer bending segment (sometimes referred to as the segment or region capable of articulation) to bend or articulate. Along the non-bending segment, it may be advantageous to spiral or swirl a separate traction lumen that guides a separate tendon along the wall (or inside) of the endoscope shaft to balance the radial forces caused by the tension in the traction wire. For specific purposes, the angle of the spiral and / or the spacing therebetween may be varied or designed, wherein a tighter spiral presents less axial compression under load force, while a lower spiral amount causes greater axial compression under load force, but also presents limited bending. In another case, the traction lumen may be guided parallel to the longitudinal axis of the elongated shaft 71 to allow controlled articulation in the desired bending or articulation segment.

[0094] In endoscopy, the elongated shaft 71 accommodates multiple components to assist in robotic procedures. The shaft may include a working channel for deploying surgical tools (or medical devices), irrigation pieces, and / or suction pieces to the operating area at the distal end of the shaft 71. The shaft 71 may also accommodate wires and / or optical fibers to transmit signals to / from an optical assembly at the distal end, which may include an optical camera. The shaft 71 may also accommodate an optical fiber to carry light from a proximally located light source (such as a light emitting diode) to the distal end of the shaft.

[0095] At the distal end of the instrument 70, the distal tip may also include an opening for a working channel for delivering tools for diagnosis and / or treatment, irrigation, and suction to the surgical site. The distal tip may also include a port for a camera (such as a fiberscope or a digital camera) to capture images of the internal anatomical space. Relatedly, the distal tip may also include a port for a light source for illuminating the anatomical space when the camera is used.

[0096] exist Fig.16 In the example of , the axis of the drive shaft, and therefore the drive input axis, is orthogonal to the axis of the elongated shaft. However, this arrangement complicates the ability of the elongated shaft 71 to roll. Rolling the elongated shaft 71 along its axis while keeping the drive input 73 stationary can cause undesirable tangling of the tendons as they extend out of the drive input 73 and into the traction cavity within the elongated shaft 71. Such resulting tangling of the tendons can disrupt any control algorithm designed to predict the movement of the flexible elongated shaft during an endoscopic procedure.

[0097] Fig.17 An alternative design of an instrument driver and an instrument is illustrated, wherein the axis of the drive unit is parallel to the axis of the elongated shaft of the instrument. As shown, a circular instrument driver 80 includes four drive units, and its drive output 81 is aligned in parallel at the end of a robot arm 82. The drive units and their corresponding drive outputs 81 are contained in a rotating assembly 83 of an instrument driver 80 driven by a drive unit in the drive unit in the assembly 83. In response to the torque provided by the rotating drive unit, the rotating assembly 83 rotates along a circular bearing, which connects the rotating assembly 83 to the non-rotating portion 84 of the instrument driver. Electric power and control signals can be conveyed to the rotating assembly 83 from the non-rotating portion 84 of the instrument driver 80 by electrical contact, and the electrical contact can be maintained by the rotation of the brush slip ring connection (not shown). In other embodiments, the rotating assembly 83 can be responsive to a separate drive unit integrated into the non-rotating portion 84, and is therefore not parallel to other drive units. The rotating mechanism 83 allows the instrument driver 80 to allow the drive unit and its corresponding drive output 81 to rotate around the instrument driver axis 85 as a single unit.

[0098] Similar to previously disclosed embodiments, the instrument 86 may include an elongated shaft portion 88 and an instrument base 87 (shown as having a transparent outer surface for discussion purposes) that includes a plurality of drive inputs 89 (such as sockets, pulleys, and spools) configured to receive drive outputs 81 in the instrument driver 80. Unlike previously disclosed embodiments, the instrument shaft 88 extends from the center of the instrument base 87, with the axis of the instrument base being substantially parallel to the axis of the drive inputs 89, rather than being substantially parallel to the axis of the drive inputs 89. Fig.16 as orthogonal in the design.

[0099] When coupled to the rotation assembly 83 of the instrument driver 80, the medical device 86, including the instrument base 87 and the instrument shaft 88, rotates in combination with the rotation assembly 83 about the instrument driver axis 85. Since the instrument shaft 88 is located at the center of the instrument base 87, the instrument shaft 88 is coaxial with the instrument driver axis 85 when attached. Therefore, the rotation of the rotation assembly 83 causes the instrument shaft 88 to rotate about its own longitudinal axis. In addition, when the instrument base 87 rotates with the instrument shaft 88, any tendons connected to the drive input 89 in the instrument base 87 do not tangle during rotation. Therefore, the parallelism of the axes of the drive output 81, the drive input 89, and the instrument shaft 88 allows the shaft to rotate without tangling any control tendons.

[0100] Fig.18 An instrument with an instrument-based insertion architecture according to some embodiments is illustrated. The instrument 150 can be coupled to any of the instrument drivers described above. The instrument 150 includes an elongated shaft 152, an end effector 162 connected to the shaft 152, and a handle 170 coupled to the shaft 152. The elongated shaft 152 includes a tubular member having a proximal portion 154 and a distal portion 156. The elongated shaft 152 includes one or more channels or grooves 158 along its outer surface. The groove 158 is configured to receive one or more wires or cables 180 passing through the groove. Therefore, one or more cables 180 extend along the outer surface of the elongated shaft 152. In other embodiments, the cable 180 may also extend through the elongated shaft 152. The manipulation of the one or more cables 180 (e.g., via the instrument driver) causes the actuation of the end effector 162.

[0101] The instrument handle 170 (also referred to as the instrument base) may generally include an attachment interface 172 having one or more mechanical input portions 174, such as a socket, pulley, or spool, which are designed to reciprocately engage with one or more torque connectors on the attachment surface of the instrument driver.

[0102] In some embodiments, the instrument 150 includes a series of pulleys or cables that enable the elongated shaft 152 to translate relative to the handle 170. In other words, the instrument 150 itself includes an instrument-based insertion architecture that accommodates instrument insertion, thereby minimizing reliance on a robotic arm to provide insertion of the instrument 150. In other embodiments, the robotic arm may be largely responsible for instrument insertion.

[0103] E. Controller.

[0104] Any of the robotic systems described herein may include an input device or controller for manipulating an instrument attached to the robotic arm. In some embodiments, the controller may be coupled to the instrument (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) such that manipulation of the controller causes corresponding manipulation of the instrument, e.g., via master-slave control.

[0105] Fig.19 is a perspective view of an embodiment of the controller 182. In this embodiment, the controller 182 includes a hybrid controller that can have both impedance and admittance control. In other embodiments, the controller 182 can utilize only impedance or passive control. In other embodiments, the controller 182 can utilize only admittance control. By being a hybrid controller, the controller 182 can advantageously have lower perceived inertia when in use. In the illustrated embodiment, the controller 182 is configured to allow manipulation of two medical devices. As illustrated, the controller 182 may include two handles 184. In some embodiments, a pair of handles 184 operates a single instrument, while in other embodiments, each handle in the pair of handles 184 operates its own corresponding instrument. Each handle 184 is connected to a gimbal 186. Each gimbal is connected to a positioning platform 188. In some embodiments, the handle 184 is considered to be distal to the gimbal 186, which is considered to be distal to the positioning platform 188. The handle 184 and the gimbal 186 are at Fig. 22 This is shown in more detail in FIG. 1 and will be described below.

[0106] like Fig.19 As shown, in the illustrated embodiment, each positioning platform 188 includes a selectively compliant assembly robot arm 198 (SCARA arm) having a plurality of connectors that are coupled to the column 194 via a prismatic joint 196. The prismatic joint 196 is configured to translate along the column 194 (e.g., along the rail 197) to allow the handle 184 to translate in the z direction, thereby providing a first degree of freedom. The SCARA arm 198 is configured to allow the handle 184 to move in the xy plane, thereby providing two additional degrees of freedom. Thus, Fig.19 Each of the illustrated positioning platforms 188 is configured to provide three positional or translational degrees of freedom and allow an operator to position the handle 184 anywhere in three-dimensional (e.g., x, y, z) space (within the reach of the positioning platform).

[0107] In some embodiments, the column 194 (and rail 197) extends along an axis aligned with the vertical direction of the workspace (e.g., the z direction as illustrated), which can be aligned with the direction of gravity. An advantage of this positioning platform 188 is that it can provide gravity compensation. In other words, the prismatic joint 196 of the positioning platform 188 can maintain a constant orientation of the gimbal 106 relative to gravity.

[0108] In some embodiments, the positioning platform 188 may have other configurations. For example, in all embodiments, the positioning platform 108 need not include a prismatic joint and / or a SCARA arm.

[0109] In some embodiments, a load sensor ( Fig.19 182). Adding a load sensor enables the controller to have admittance control in addition to impedance control. Under admittance control, the perceived inertia of the controller 182 can be reduced. This is because the mass of the gimbal 186 and / or the positioning platform can be hidden via the load sensor. This may be because the load sensor can measure the force applied to the controller and can be used to provide an output to drive the motor in the controller 182 to assist in the movement of the controller 182. The amount of hidden mass depends on the location of the load sensor. In some embodiments, the mass proximal to the load sensor can be partially or substantially hidden, while the mass distal to the load sensor will not be hidden.

[0110] In some embodiments, by positioning the load sensor distally on the controller 182 (e.g., at Fig.19In the embodiment of the present invention, the mass of the positioning platform 188 (shown in the gimbal 186) can be partially or substantially hidden when operating the controller 182. Similarly, the mass of the positioning platform 188 (which has a relatively higher mass than the gimbal 186) can also be partially or substantially hidden when operating the controller 182. Hiding the mass advantageously results in lower perceived inertia for the clinician. In the absence of a load sensor, in order to move the handle 184 in the z direction, the operator provides sufficient force to the handle 184 to lift the handle 184, the gimbal 186 and the SCARA arm 198 upward. In addition, it can be imagined that less force will be required to move the handle in the xy plane compared to movement in the z direction. This difference will likely result in an uneven operating experience for the operator, which will make the controller 182 difficult to use. Therefore, by including a load sensor, as described herein, the controller 182 can help the user translate the handle 184 in the x, y and z directions, and provide a more uniform and controlled operating experience. In some embodiments, the load sensor enables the positioning platform 188 to be operated substantially or completely under admittance control. The moment of inertia of the gimbal 186 can be relatively low compared to the positioning platform 188. This may be because the gimbal 186 is generally much smaller than the positioning platform 188. Therefore, at least some portions of the gimbal 186 can be suitable for impedance control.

[0111] One advantage of such a hybrid impedance / admittance controller 182 as described herein is that the perceived inertia of the system can be relatively lower than a system that relies solely on impedance control. In addition, the mechanical structure of the hybrid controller 182 can be simpler because the admittance control can be used to supplement and smooth the movement of the system. In contrast, the mechanical structure of an impedance-only system is typically very complex in an attempt to normalize the forces used to move the system in different directions and minimize the perceived inertia.

[0112] In some embodiments, by using a hybrid controller 182 as described herein, it is possible that the mass and inertia of the gimbal 186 can actually be increased relative to the gimbal of an impedance-only controller because so much of the total mass and inertia of the controller 182 can be hidden by the admittance control of the positioning platform. In some embodiments, increasing the size of the gimbal can allow for the use of larger motors, which can allow the controller to provide stronger haptic feedback forces than other systems that require the use of lightweight gimbals and motors to avoid increasing the total mass and inertia.

[0113] like Fig.19As shown, the hybrid controller 182 can be viewed as a plurality of links and joints in series, for example, as a serial link manipulator. The handle 184, the gimbal 186, and the positioning platform 188 each include one or more links that are operably coupled, wherein the most proximal link is adjacent to the column 194 of the positioning platform 188, and the most distal link is part of the handle 184 itself. In some embodiments, one or more load sensors ( Fig.19 ) is inserted into the controller 182 to provide admittance control of at least some portions of the controller 182. Other portions of the controller 182 may be controlled by a clinician or operator through impedance control (or in some cases, passive control). In some embodiments, the connectors and joints proximal to the load sensor may be directly or indirectly affected by the load sensor. Thus, manipulation of these proximal connectors and joints may be assisted by admittance control. In some embodiments, the connectors and joints distal to the load sensor may not be directly or indirectly affected by the load sensor. Thus, manipulation of these distal connectors and joints may be assisted by impedance control. For example, in some embodiments, the load sensor is positioned in the universal bracket 186 so that the distal joints 128, 130, 132 ( Fig. 22 The distal connections and joints may be moved by impedance control. In contrast, the connections and joints proximal to the load sensors (such as those in the positioning platform 188) may be directly or indirectly affected by the load sensors. In other words, the manipulation of the axis of the gimbal 186 at these joints is not directly or indirectly based on the output of the load sensors. These proximal connections and joints may be moved by admittance control. In contrast, the connections and joints proximal to the load sensors (such as those in the positioning platform 188) may be directly or indirectly affected by the load sensors. In other words, the manipulation of the axis at these joints is directly or indirectly based on the output of the load sensors. These proximal connections and joints may be moved by admittance control.

[0114] In some embodiments, one or more load sensors are positioned in the controller. For example, in some embodiments, a load sensor (not shown) is positioned in the body of each gimbal in the gimbal 106. By providing a load sensor, a portion of the controller 182 can be operated under admittance control, thereby advantageously reducing the perceived inertia of the controller when in use. In some embodiments, the positioning platform 108 is configured for admittance control, and the gimbal 106 is constructed for impedance control. In other embodiments, the gimbal 106 is configured for admittance control, and the positioning platform 108 is constructed for impedance control. Therefore, for some embodiments, the translational degree of freedom or the azimuth degree of freedom of the positioning platform 108 can rely on admittance control, while the rotational degree of freedom of the gimbal 106 relies on impedance control.

[0115] F. Navigation and control.

[0116] Conventional endoscopy may involve the use of fluoroscopy (e.g., as may be delivered by a C-arm) and other forms of radiation-based imaging modalities to provide intracavitary guidance to the operator physician. In contrast, the robotic system contemplated by the present disclosure may provide non-radiation-based navigation and positioning devices to reduce physician exposure to radiation and reduce the amount of equipment in the operating room. As used herein, the term "positioning" may refer to determining and / or monitoring the position of an object in a reference coordinate system. Technologies such as preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to achieve a radiation-free operating environment. In other cases where radiation-based imaging modalities are still used, preoperative mapping, computer vision, real-time EM tracking, and robotic command data may be used alone or in combination to improve the information obtained only by radiation-based imaging modalities.

[0117] Fig. 20 is a block diagram illustrating a positioning system 90 for estimating the position of one or more elements of a robotic system, such as the position of an instrument, according to an example embodiment. The positioning system 90 may be a set of one or more computer devices configured to execute one or more instructions. The computer device may be embodied by a processor (or multiple processors) and a computer readable memory in one or more of the components discussed above. By way of example and not limitation, the computer device may be located at Figure 1 The tower 30 shown, Figures 1 to 4 The cart shown, Figures 5 to 14 The bed shown, etc.

[0118] like Fig. 20 As shown, the positioning system 90 may include a positioning module 95 that processes the input data 91-94 to generate position data 96 for the distal end of the medical device. The position data 96 may be data or logic representing the position and / or orientation of the distal end of the instrument relative to a reference system. The reference system may be a reference system relative to the patient's anatomy or a known object such as an EM field generator (see below for a discussion of EM field generators).

[0119] The various input data 91-94 are now described in more detail. Preoperative mapping can be accomplished using a collection of low-dose CT scans. The preoperative CT scans are reconstructed into three-dimensional images that are visualized, for example, as "slices" of cross-sectional views of the patient's internal anatomy. When analyzed as a whole, an image-based model of the anatomical cavities, spaces, and structures of the patient's anatomical structure (such as the patient's lung network) can be generated. Techniques such as centerline geometry can be determined and approximated from the CT images to form a three-dimensional volume of the patient's anatomical structure, which is referred to as model data 91 (also referred to as "preoperative model data" when generated using only preoperative CT scans). The use of centerline geometry is discussed in U.S. Patent Application No. 14 / 523,760, the contents of which are incorporated herein in their entirety. Network topology models can also be derived from CT images and are particularly suitable for bronchoscopy.

[0120] In some embodiments, the instrument can be equipped with a camera to provide visual data 92. The positioning module 95 can process the visual data to implement one or more vision-based position tracking. For example, preoperative model data can be used in conjunction with the visual data 92 to implement computer vision-based tracking of a medical instrument (e.g., an endoscope or an instrument advanced through a working channel of an endoscope). For example, using the preoperative model data 91, the robotic system can generate a library of expected endoscopic images from the model based on the expected path of travel of the endoscope, each image connected to a location within the model. In operation, the robotic system can reference the library to compare real-time images captured at a video camera (e.g., a camera at the distal end of an endoscope) with those in the image library to assist in positioning.

[0121] Other computer vision based tracking techniques use feature tracking to determine the motion of the camera, and therefore the motion of the endoscope. Some features of the localization module 95 may identify circular geometric structures in the preoperative model data 91 that correspond to anatomical cavities and track changes in those geometric structures to determine which anatomical cavity is selected, as well as track relative rotational and / or translational motion of the camera. The use of topological maps may further enhance vision based algorithms or techniques.

[0122] Optical flow (another computer vision-based technique) can analyze the displacement and translation of image pixels in a video sequence in the visual data 92 to infer camera movement. Examples of optical flow techniques can include motion detection, object segmentation calculations, brightness, motion compensated encoding, stereo disparity measurements, etc. Through multiple iterations of multi-frame comparisons, the movement and position of the camera (and therefore the endoscope) can be determined.

[0123] The positioning module 95 can use real-time EM tracking to generate the real-time position of the endoscope in a global coordinate system that can be aligned to the patient's anatomical structure represented by the preoperative model. In EM tracking, an EM sensor (or tracker) comprising one or more sensor coils in one or more positions and orientations embedded in a medical device (e.g., an endoscopic tool) measures changes in the EM field generated by one or more static EM field generators positioned at known locations. The position information detected by the EM sensor is stored as EM data 93. The EM field generator (or transmitter) can be placed close to the patient to generate a low-intensity magnetic field that can be detected by the embedded sensor. The magnetic field induces a small current in the sensor coil of the EM sensor, which can be analyzed to determine the distance and angle between the EM sensor and the EM field generator. These distances and orientations can be "aligned" to the patient's anatomical structure (e.g., a preoperative model) during surgery to determine a geometric transformation that aligns a single position in the coordinate system with a position in the preoperative model of the patient's anatomical structure. Once registered, an embedded EM tracker in one or more locations of a medical device (eg, the distal tip of an endoscope) can provide a real-time indication of the progress of the medical device through the patient's anatomy.

[0124] The robotic commands and kinematic data 94 may also be used by a positioning module 95 to provide orientation data 96 for the robotic system. The pitch and yaw of the device derived from the joint motion commands may be determined during preoperative calibration. In surgery, these calibration measurements may be used in conjunction with known insertion depth information to estimate the position of the instrument. Alternatively, these calculations may be analyzed in conjunction with EM, vision, and / or topological modeling to estimate the position of the medical device within the network.

[0125] like Fig. 20 As shown, the positioning module 95 can use a variety of other input data. For example, although Fig.15 Not shown, but an instrument utilizing shape sensing fibers may provide shape data that positioning module 95 may use to determine the position and shape of the instrument.

[0126] The localization module 95 may use the input data 91-94 in combination. In some cases, such a combination may use a probabilistic approach, where the localization module 95 assigns a confidence weight to a position determined from each of the input data 91-94. Thus, in situations where the EM data may be unreliable (such as where EM interference may be present), the confidence in the position determined by the EM data 93 may be reduced, and the localization module 95 may rely more heavily on the visual data 92 and / or the robot command and kinematics data 94.

[0127] As discussed above, the robotic systems discussed herein can be designed to incorporate a combination of one or more of the above techniques. A computer-based control system for a robotic system located in a tower, bed, and / or cart can store computer program instructions, for example, in a non-transitory computer-readable storage medium (such as a permanent magnetic storage drive, a solid-state drive, etc.), which when executed causes the system to receive and analyze sensor data and user commands, generate control signals for the entire system, and display navigation and positioning data, such as the position of an instrument in a global coordinate system, an anatomical map, etc.

[0128] 2. Controller for a robot-enabled teleoperation system

[0129] A robot-enabled teleoperated system (such as the above-described system) may include an input device or controller configured to allow an operator (e.g., a physician performing a robot-enabled medical procedure) to manipulate and control one or more instruments (or a robotic arm holding one or more instruments). In some embodiments, the robot-enabled teleoperated system includes a controller for operating one or more medical tools. Those skilled in the art will appreciate that the controllers described herein may also be applied in non-medical environments. For example, the controller may be used to manipulate tools involving hazardous materials. In addition, in some embodiments, the controllers described herein may be used to grasp objects in a physical environment and / or a virtual environment. In some embodiments, the controller may be self-sufficient as a service robot that interacts with a human operator. In some embodiments, the controller may be coupled (e.g., communicatively, electronically, electrically, wirelessly, and / or mechanically) to an instrument (such as a medical instrument) so that manipulation of the controller causes corresponding manipulation of the instrument. In some embodiments, the controller and the instrument are arranged as a master-slave pair. In some embodiments, the controller may be referred to as a manipulator, a simulator, a master device, an interface, etc. In some embodiments, the controller may include a plurality of connectors assembled in parallel or in series.

[0130] The controller can be used as an input device for the operator to control the action of a medical device (such as an endoscopic, intracavitary, laparoscopic or open surgical instrument). The operator's movement of the controller can guide the movement of the medical device. For example, when the operator translates the controller in three-dimensional space (e.g., up, down, left, right, back, forward), the system can cause a corresponding translation of the medical device. Similarly, if the operator rotates the controller (e.g., around any of three orthogonal axes), the system can cause a corresponding rotational movement of the medical device. The controller can also be configured to receive an input that allows the operator to actuate the medical device. For example, if the medical device includes a gripper, the controller can receive an input that allows the operator to open and close the gripper.

[0131] The controller may also provide tactile feedback to the operator. For example, in some embodiments, the force or torque applied to the medical device may be transmitted back to the operator through the controller. In some embodiments, providing tactile feedback to the operator through the controller provides the user with an improved operation, control, or driving experience. In some embodiments, tactile cues may be provided to make it easier for the operator to interact with the controller and operate the system.

[0132] In some embodiments, the controller is also used to align the operator's hand with the orientation of the medical device, such as when switching medical devices. For example, if the medical device is positioned inside a patient during a medical procedure, it is important that the medical device does not move accidentally or unintentionally. Therefore, when the operator wishes to control a medical device that is already positioned inside a patient, the controller can first be moved to match the orientation of the medical device while the device remains in place. With the controller correctly oriented to match the orientation of the medical device, the operator can then use the controller to manipulate the medical device.

[0133] In some embodiments, a robot-enabled medical system includes a controller having seven degrees of freedom that follows the movement of an operator's hand, wherein the seven degrees of freedom include three positional degrees of freedom (e.g., translational movement in x, y, z space), three rotational degrees of freedom (e.g., rotational movement around a pitch axis, a roll axis, and a yaw axis), and one (or more) instrument actuation degrees of freedom (e.g., angular degrees of freedom). In some embodiments, the instrument actuation degrees of freedom can control the opening and closing of an end effector (such as a clamp or gripper) of a medical device to hold an object. In some embodiments, the instrument actuation degrees of freedom can be omitted. In some embodiments, the controller may include a greater or lesser number of degrees of freedom. For example, in some embodiments, the controller may include more than three positional degrees of freedom or more than three rotational degrees of freedom to provide one or more redundant degrees of freedom. In some embodiments, the redundant degrees of freedom may provide additional mechanical flexibility for the controller, for example, to avoid singularities caused by the mechanical structure of the controller.

[0134] Fig.21A A block diagram of an embodiment of a robotic-enabled medical system 100 is illustrated, including a schematic diagram of an embodiment of a controller 302 and a schematic diagram of an embodiment of a robotic-enabled medical device 310. As briefly described above, the controller 302 can be coupled (e.g., mechanically, electrically, or functionally) to the robotic-enabled medical device 310 such that manipulation of the controller 302 causes substantially corresponding movement of the robotic-enabled medical device 310, and forces exerted on the robotic-enabled medical device 310 can be transmitted back to the controller and tactilely communicated to an operator. In some embodiments, the controller 302 and the robotic-enabled medical device 310 are arranged in a master-slave configuration.

[0135] In the illustrated embodiment of the system 100, the controller 302 includes a handle 104, a gimbal 106, and a positioning platform 108. The handle 104 may be configured to be held by an operator. As illustrated, in some embodiments, the handle 104 is coupled to the gimbal 106 and the positioning platform 108. As described above, the handle 104 may include one or more degrees of freedom to actuate the instrument. The gimbal 106 may be configured to provide one or more rotational degrees of freedom to allow the operator to rotate the handle 104. In some embodiments, the gimbal 106 is configured to provide at least three rotational degrees of freedom. For example, the gimbal 106 may be configured to allow the operator to rotate the handle 104 around a pitch axis, a roll axis, and a yaw axis. Reference Figures 17 to 19 B describes an example gimbal 106 in more detail. The positioning platform 108 can be configured to provide one or more translational (also referred to herein as positional) degrees of freedom to allow an operator to translate the handle 104. In some embodiments, the positioning platform 108 is configured to provide at least three positional degrees of freedom. For example, the positioning platform 108 can be configured to allow an operator to translate the handle 104 in three-dimensional space (e.g., x, y, and z directions). Fig.19 An example positioning platform 108 is described in further detail. The gimbal 106 and positioning platform 108 together may enable a user to manipulate the handle 104 .

[0136] In the illustrated implementation, the robotically enabled medical device 310 includes an instrument or tool 312 (which may include an end effector), an instrument driver 314, and a robotic arm 316 (or other instrument positioning device). The medical tool 312 may be, for example, Fig. 9 The laparoscopic instrument 59 shown, as well as other types of endoscopic or laparoscopic medical instruments as described throughout this application and will be apparent to those of ordinary skill in the art. The medical tool 312 may include one or more end effectors. The end effector may be positioned at the distal end of the medical tool 312. The end effector may be configured for insertion into a patient. In some embodiments, the end effector may be a grasper, a clamp, a cutter, a collar device, or scissors, etc. In some embodiments, the medical tool 312 may include a mirror or a camera.

[0137] The medical tool 312 can be attached to an instrument driver 314. The instrument driver 314 can be configured to actuate the medical tool 312 as described above. For example, the instrument driver 314 can be configured to pull one or more pull wires of the medical tool 312 to actuate the medical tool 312. In some embodiments, the instrument driver 314 can be an instrument drive mechanism as described above. The instrument driver 314 can be attached to a robotic arm 316, for example, Fig.14As shown. The robotic arm 316 can be configured to be articulated or moved to further manipulate and position the medical tool 312. Example medical instruments / tools, instrument drivers, and robotic arms are described above. Figures 1 to 15 is shown in the system.

[0138] The controller 302 can be coupled to the robotically enabled medical instrument 310 such that manipulation of the handle 104 causes substantially corresponding movement of the medical tool 312, and the forces applied to the medical tool 312 can be tactilely transmitted to the operator through the handle 104. Manipulation of the handle 104 can be measured or determined by measuring the forces and movements of the gimbal 106 and the positioning platform 108. The movement of the medical tool 312 can be caused by the articulation and movement of the instrument driver 314 and / or the robotic arm 316. Thus, by manipulating the handle 104, the operator can control the medical tool 312.

[0139] As will be described below, in some embodiments, the controllers described herein are configured to operate using both admittance control and impedance control. These and other features of the controllers are further discussed in the following sections.

[0140] A. Hybrid controller.

[0141] Fig. 21B is a block diagram of an embodiment of a controller 302 configured to operate using both impedance control and admittance control. Such a controller 302 may be referred to as a hybrid controller. However, in some embodiments, a controller configured to operate using only impedance control (without admittance control) or only admittance control (without impedance control) may be used.

[0142] Impedance control and admittance control are two control schemes used to control robotic systems. Under impedance control, the system measures displacement (e.g., changes in position and velocity) and outputs force. For example, with impedance control, the system can measure the distance or speed at which the operator moves the controller, and based on the measurement, a force is generated on the controller (e.g., by an actuating motor). Under impedance control, the operator's movement of the controller can backdrive multiple parts of the instrument. In many cases, the use of impedance control can result in large perceived inertia. This may be because, for example, impedance control relies on the operator to move the controller. Under impedance control, the operator overcomes the perceived mass or inertia of the controller to move it, making the controller feel heavy. For impedance control, the operator physically overcomes most or all of the inertia in the system to move the controller. Some controllers rely solely on impedance control.

[0143] Under admittance control, the system measures the force and / or torque applied by the operator to the controller and outputs the corresponding rate and / or position of the controller. In some aspects, admittance control is opposite to impedance control. In some embodiments, the use of admittance control can reduce the perceived inertia or mass of the system. Admittance control can be used to change the dynamics of a controller that is perceived to have high mass or inertia. In some cases, by using admittance control, the operator does not need to overcome all inertia in the system to move the controller. For example, under admittance control, when a user applies a force on the controller, the system can measure the force and help the user move the controller by driving one or more motors associated with the controller, thereby obtaining the desired rate and / or position of the controller. In other words, for admittance control, a force sensor or load sensor measures the force that the operator is applying to the controller, and moves the controller and the coupled robot-enabled medical device 310 in a way that feels light. Admittance control can feel lighter than impedance control because under admittance control, the perceived inertia of the controller can be hidden because the motor in the controller can help accelerate the mass. In contrast, with impedance control, the user is responsible for all or substantially all mass acceleration.

[0144] like Fig. 21B As shown in the embodiment illustrated in , the controller 302 includes a handle 104, a gimbal 106, and a positioning platform 108. As described above, the gimbal 106 can be configured to provide one or more rotational degrees of freedom (e.g., three or four), and the positioning platform 108 can be configured to provide one or more rotational degrees of freedom (e.g., three or four). The gimbal 106 and the positioning platform 108 can allow a user to move the handle 104 in three-dimensional space and rotate the handle 104 around a pitch axis, a roll axis, and a yaw axis. Manipulation of the handle 104 causes movement of the corresponding medical device. In addition, the handle 104, the gimbal 106, and the positioning platform 108 can be configured to provide tactile feedback to the operator indicating the force applied to the medical device.

[0145] like Fig. 21B , in some embodiments, in the controller 302, the gimbal 106 is configured for impedance control, and the positioning platform 108 is configured for admittance control. Therefore, for some embodiments, the translational degree of freedom or the positional degree of freedom of the positioning platform 108 relies on the admittance control, while the rotational degree of freedom of the gimbal 106 relies on the impedance control. As further described below, this type of hybrid controller 302 may have certain advantages. In other embodiments, the gimbal 106 is configured for admittance control, and the positioning platform 108 is configured for impedance control. In some embodiments, both the gimbal 106 and the positioning platform may be configured for admittance control, or both may be configured for impedance control.

[0146] To utilize admittance control, the controller 302 includes at least one force sensor or load sensor 112. The load sensor 112 is configured to measure the force applied by the operator to the controller 302 (generally, the force applied to the handle 104). The output signal (measurement of force) of the load sensor 112 is used to provide a control signal to control the movement of the controller 302, such as the positioning platform 108. The robotic-enabled medical device 310 will follow the movement of the handle 104 (e.g., by activating one or more motors in the instrument driver 314 or the robotic arm 316). In some embodiments, the load sensor 112 can be a three-degree-of-freedom load sensor that measures force in three directions.

[0147] In the illustrated embodiment, the load cell 112 is positioned within the gimbal 106. Fig. 22 The load sensor 112 is shown to be positioned at different locations within the gimbal 106. Other locations for the load sensor 112 are also possible. In some embodiments, the load sensor 112 is positioned in the positioning platform 108. In some embodiments, more than one load sensor 112 (e.g., two, three, four, or more load sensors) is included, which can be positioned in the handle 104, the gimbal 106, and / or the positioning platform 108.

[0148] In some embodiments, the load sensor 112 is positioned distally (closer to the handle 104) in the controller 302. In some embodiments, this allows the use of admittance control to hide the perceived mass of portions of the controller 302 that are proximal to the load sensor 112 (e.g., portions of the controller 302 that are on the opposite side of the load sensor 112 from the handle 104).

[0149] As described above, in some embodiments, the load sensor 112 (or force sensor) is positioned in the gimbal 106. In some embodiments, the gimbal 106 provides a rotational degree of freedom for the controller 302 with impedance control, while the positioning platform 108 provides a positional degree of freedom for the controller 302 with admittance control (e.g., based on the output of the load sensor 112 positioned in the gimbal 106). There are many ways in which the load sensor 112 can be positioned within the gimbal 106. The degree to which the controller 302 senses the reduction in inertia can be based in part on the position of the load sensor 112 within the gimbal 106. Two exemplary embodiments are described in this section, which show load sensors 112 positioned in two different parts of the gimbal 106. Other embodiments are also possible.

[0150] Fig. 22106 is an isometric view of an embodiment of the gimbal 106. As illustrated, for some embodiments, the gimbal 106 is positioned at the distal end of the positioning platform 108 ( Fig. 22 108). As used in this patent application, in the context of the controller 302, the term distal refers to a direction toward the handle 104 (e.g., the handle 104 is the most distal component of the controller 302), and the term proximal refers to the opposite direction. Thus, the proximal end of the gimbal 106 can be attached to the distal end of the positioning platform 108. Thus, the handle 104 can be positioned at the distal end of the gimbal 106.

[0151] In some embodiments, the handle 104 is configured to be held by an operator. The handle 104 can be configured to simulate or imitate the medical device that the controller 302 is used to control. In some embodiments, the handle includes a gripper handle (e.g., a radially symmetrical gripper handle), a stylus, a paddle handle, etc. In the illustrated embodiment, the handle 104 includes two actuating arms 120 configured to provide the instrument actuation freedom discussed above. While holding the handle 104, the operator can adjust the angle between the actuating arms 120 to control the corresponding angle associated with the controlled medical device. For example, in the case where the medical device is a gripper, scissors, etc., the angle between the actuating arms 120 can be used to control the angle between the two jaws of the gripper.

[0152] In the illustrated embodiment, the gimbal 106 includes three arms or links connected by joints. Fig. 22 As shown, the gimbal 106 includes a first connector 122, a second connector 124, and a third connector 126. Fig. 22 As illustrated, the gimbal 106 also includes a first joint 128, a second joint 130, a third joint 132, and a fourth joint 134. These joints allow the various connections to rotate, thereby providing the gimbal 106 with the rotational degrees of freedom discussed above.

[0153] The handle 104 is connected to the distal end of the first connector 122 via a first joint 128. The first joint 128 can be configured to allow the handle 104 to rotate relative to the first connector 122. In the illustrated embodiment, the first joint 128 allows the handle 104 to rotate about a roll axis 136. In some embodiments, the roll axis 136 is aligned with the longitudinal axis of the handle 104. The first joint 128 can be a revolute joint.

[0154] The proximal end of the first connector 122 is connected to the distal end of the second connector 124 by the second joint 130. The second joint 130 can be configured to allow the handle 104 and the first connector 122 to rotate relative to the second connector 124. In the illustrated embodiment, the second joint 130 allows the handle 104 and the first connector 122 to rotate around the yaw axis 138. In some embodiments, the yaw axis 138 extends through the second joint 130 and intersects the roll axis 136 at the center point of the handle 104. The second joint 130 can be a revolute joint. As shown in the figure, for some embodiments, the first connector 122 includes an L-shape. In some embodiments, the first connector 122 is configured to have a groove formed therein for receiving the second connector 124 and allowing the second connector 124 to rotate relative to the first connector 122.

[0155] The proximal end of the second connector 124 is connected to the distal end of the third connector 126 via the third joint 132. The third joint 132 can be configured to allow the handle 104, the first connector 122, and the second connector 124 to rotate relative to the third connector 126. In the illustrated embodiment, the third joint 132 allows the handle 104, the first connector 122, and the second connector 124 to rotate around the pitch axis 139. In some embodiments, the pitch axis 139 extends through the third joint 132 and intersects the roll axis 136 and the yaw axis 138 at the center point of the handle 104. The third joint 132 can be a revolute joint. As shown in the figure, for some embodiments, the second connector 124 includes an L-shape. In some embodiments, the L-shaped second connector 124 is received in a groove of the L-shaped first connector 122 (such as Fig.17 In other embodiments, the L-shaped first connector 122 may be received in a groove of the L-shaped second connector 124.

[0156] In the illustrated embodiment, the first joint 128, the first link 122, the second joint 130, the second link 124, and the third joint 132 provide three rotational degrees of freedom, thereby allowing the rotation of the handle 104 to be adjusted in pitch, roll, and yaw. In the illustrated embodiment, the gimbal 106 also includes a third link 126 and a fourth joint 134 that provide redundant rotational degrees of freedom. This need not be included in all embodiments, but can provide greater mechanical flexibility for the gimbal 106.

[0157] As shown, the distal end of the third connector 126 is connected to the proximal end of the second connector 124 by the third joint 132. The proximal end of the third connector 126 is connected to the distal end of the positioning platform 108 by the fourth joint 134. The fourth joint 134 can be configured to allow the handle 104, the first connector 122, the second connector 124 and the third connector 126 to rotate relative to the positioning platform 108. In the illustrated embodiment, the fourth joint 134 allows the handle 104, the first connector 122, the second connector 124 and the third connector 126 to rotate around the axis 141. In some embodiments, the axis 141 is parallel to the yaw axis 138. In some embodiments, the yaw axis 138 and the axis 141 are coaxial, but as illustrated, this is not necessarily the case in all embodiments. The axis 141 (and the yaw axis 138) can be parallel to the direction of gravity to keep the orientation of the universal bracket relative to the direction of gravity, as described above. The fourth joint 134 can be a revolute joint. As shown, for some embodiments, the third connector 126 comprises an L-shape.

[0158] A tactile interface device (HID) is a device that provides tactile feedback, and includes any controller in the controller for controlling a robotic system, a robotic arm, and / or an instrument as described herein. In some embodiments, the input sensor is different from the tactile interface device. In some embodiments, the HID includes a tactile feedback device and an input device (e.g., a gripper, such as handle 104), which is configured to be held by an operator. In some embodiments, the HID includes a component designed or selected to have minimal mechanical dissipation effects (such as friction and damping). In this way, the HID is designed to be transparent to the user / operator, so that when the user moves the HID in free space, the user does not feel too much resistance or impedance, thereby allowing the user to complete surgical tasks with minimal burden and interference imposed by the HID.

[0159] Fig.23A control system 200 for providing tactile feedback to an operator is depicted according to some embodiments. The control system 200 receives patient-side feedback 202 from the operation and / or state of one or more robotic arms 12, and receives HID feedback 204 from the operation and / or state of one or more input devices (e.g., controller 302). Both the patient-side feedback 202 and the HID feedback 204 are provided to an impedance control unit 206. In some embodiments, the patient-side feedback 202 includes information indicating an event, state, or operation associated at least in part with a medical device 310 (e.g., tool motion differences, collisions between two medical devices such as referred to as "sword fighting", collisions between robotic arms, collisions between robotic arms and the surrounding environment, such as collisions between robotic arms and static components in an operating room, failures, etc.). In some embodiments, the HID feedback 204 includes information indicating an event, state, or operation associated with one or more input devices (e.g., controller 302), such as variable damping, gimbal navigation, gimbal collisions, loose grip during re-clutching, failures, etc. In some embodiments, patient-side feedback 202 and / or HID feedback 204 are generated (or provided or updated) based on input signals from one or more input sensors (e.g., sensors (such as encoders) coupled to one or more input devices (such as controller 102302). In some embodiments, impedance control unit 206 includes two modules: a rolling tactile module 208 and a tactile buzz module 210. In some embodiments, tactile buzz module 210 also includes two components: an event type classifier 212 and a profile generator 214.

[0160] In some embodiments, the outputs from both the rolling haptic module 208 and the tactile buzz module 210 are combined at the signal controller 216 (also referred to as a signal combiner) and delivered to the HID. In some embodiments, the output from the tactile buzz module 210 includes a vibration tactile feedback signal. In some embodiments, the output from the rolling haptic module 208 includes a kinesthetic tactile feedback signal. For example, the output control signal from the signal controller 216 is provided to a component of the HID (e.g., an actuator or a controller for the actuator). In some embodiments, the component is a torque / current control loop 218 of the HID, and the HID includes a gimbal 106. In some embodiments, the torque / current control loop 218 controls the motor of the gimbal 106. In some embodiments, the motor is a gimbal rolling motor of the gimbal 106.

[0161] In some embodiments, when an event in a first set of predefined events (e.g., tool motion discrepancy) is detected by (or based on) one or more input sensors, patient-side feedback 202 is generated (or provided or updated) (e.g., the patient-side feedback 202 includes information indicating an event in the first set of predefined events for providing kinesthetic tactile feedback). A tool motion discrepancy occurs when a primary command signal (e.g., a control signal) received by the robotic arm 12 causes an output motion of the robotic arm 12 (or a coupled surgical tool) to deviate from an expected motion corresponding to the primary command signal (e.g., the robotic arm 12 moves more than specified by the primary command / control signal, the robotic arm 12 moves less than specified by the control signal, the robotic arm 12 moves at a speed less than a predetermined threshold specified by the control signal, the robotic arm 12 moves in a different direction than specified by the control signal, etc.). In some embodiments, a tool motion discrepancy is detected when a processor of the robotic system 10 determines, based on one or more input sensors in the robotic arm 12 (e.g., encoders at or near one or more motors of the robotic arm 12, one or more inertial sensors, etc.), that the current position of the robotic arm deviates from a predicted position based on a control signal sent to the robotic arm 12. Based on the processor detecting or determining an event in the first set of predetermined events (e.g., a tool motion discrepancy), an output signal is sent to a rolling haptic module 208 of the impedance control unit 206 as patient-side feedback 202. In some embodiments, the patient-side feedback 202 results in the generation of kinesthetic haptic feedback.

[0162] In some embodiments, when an event in the second set of predefined events (e.g., a "sword fight" is about to occur or has occurred) is detected by (or based on) one or more input signals, patient-side feedback 202 is generated (or provided or updated) (e.g., the patient-side feedback 202 includes information indicating an event in the second set of predefined events for providing vibration tactile feedback). In some embodiments, the second set of predefined events is mutually exclusive with the first set of predefined events. An instrument "sword fight" occurs when an instrument axis is within or less than a threshold distance to a different instrument axis (e.g., less than 10 cm, less than 5 cm, less than 2 cm, or less than 1 cm) or a collision has occurred between two instrument axes. In some embodiments, the instrument axis is a portion of a medical instrument that is mechanically grasped directly or indirectly by the robotic arm 12 (e.g., via the instrument driver 28). For example, an instrument axis of a first medical instrument held and controlled by a first robotic arm may be within or less than a threshold distance to an instrument axis of a second medical instrument held and controlled by a second robotic arm. In some embodiments, an instrument "fight" is detected when the processor of the robotic system 10 determines that the current position of the first medical instrument manipulated (or held) by the first robotic arm is within a threshold distance from the current position of the second medical instrument manipulated (or held) by the second robotic arm based on one or more sensors in the first robotic arm and the second robotic arm (e.g., encoders, one or more inertial sensors at or near one or more motors of the robotic arms) and the sizes of the first medical instrument and the second medical instrument. When the processor detects the instrument "fight", an output signal is sent to the tactile buzz module 210 of the impedance control unit 206 as patient-side feedback 202. The patient-side feedback 202 causes the generation of vibrotactile tactile feedback. In some embodiments, tactile technology that relies on tactile stimulation transmits mechanical stimulation to the skin of the operator. The mechanical stimulation includes vibration stimulation, surface change stimulation, or friction change stimulation. In some embodiments, the tactile feedback generates a sensation such as vibration, friction, or micro-deformation.

[0163] In some embodiments, a robotic arm "sword fight" is considered to have occurred when a first robotic arm is within or less than a threshold distance to a different robotic arm (e.g., less than 10 cm, less than 5 cm, less than 2 cm, or less than 1 cm), or a collision has occurred between the two robotic arms. In some embodiments, a "sword fight" is considered to have occurred when a medical device held by a first robotic arm is within or less than a threshold distance to a different robotic arm (e.g., less than 10 cm, less than 5 cm, less than 2 cm, or less than 1 cm), or a collision has occurred between the medical device and the robotic arm.

[0164] In some embodiments, the patient-side feedback 202 includes information indicating an event in which an instrument axis and / or a robotic arm of a first medical device approaches or collides with a camera in a patient body workspace (for monitoring a portion of the patient body workspace). This information can be used to provide vibrotactile feedback. In some embodiments, the patient body workspace is a space inside the patient's body. A collision or near collision occurs between the instrument arm and the camera when the instrument axis and / or the robotic arm is within a threshold distance to the camera (e.g., less than 10 cm, less than 5 cm, less than 2 cm, or less than 1 cm) or when a collision has occurred between the camera and the instrument axis and / or the robotic arm.

[0165] In some embodiments, based on a determination that the current position of the robotic arm and / or an instrument manipulated by the robotic arm is less than a threshold distance from the camera, a collision (or impending collision) is detected by a processor of the robotic system 10 based on one or more input sensors in the robotic arm (e.g., encoders, one or more inertial sensors at or near one or more motors of the robotic arm). Based on the processor detecting a collision or near collision, an output signal is sent as patient-side feedback 202 to the tactile buzz module 210 of the impedance control unit 206.

[0166] In some embodiments, the patient-side feedback 202 includes information indicating a fault condition (e.g., detection of at least one of the fault conditions) (e.g., for providing vibrotactile feedback). Examples of fault conditions associated with the patient-side feedback 202 include detecting excessive forces (e.g., forces above a force threshold) on the robotic arm (e.g., caused by a collision between the robotic arm and the environment, such as a patient, a nurse, etc.).

[0167] In some embodiments, the HID feedback 204 includes information indicating a variable damping level provided by the HID during navigation of the gimbal 106. For example, based in part on how the user manipulates the HID, an appropriate variable damping resistance level is typically applied to the HID or one or more joints of the HID (e.g., joints 128, 130, 132, and 134) to modify the force or torque provided by the one or more joints. Variable damping (or changes thereto) can be used to provide kinesthetic tactile feedback. For example, an increase in the damping level may indicate a rolling tactile module 208 provided as an input signal (e.g., HID feedback 204) to the impedance control unit 206. In some embodiments, when the tool motion difference is determined based on information in the patient-side feedback 202, the impedance control unit 206 changes the damping level from the current variable damping level indicated by the HID feedback 204 provided to the impedance control unit 206 to provide or modify kinesthetic tactile feedback.

[0168] In some embodiments, the HID feedback 204 includes an indication that the first gimbal is approaching or colliding with the second gimbal (e.g., Fig.19 The left universal bracket and Fig.19 The information of the right gimbal (e.g., for providing vibration tactile feedback) is provided. A collision or near collision occurs between the first gimbal and the second gimbal when the two gimbals are within or less than a first threshold distance (e.g., less than 20 cm, 10 cm, less than 5 cm, less than 2 cm, or a collision has occurred). In some embodiments, a collision between the two gimbals is detected when the processor of the robotic system 10 determines that the current position of the first gimbal is within a first threshold distance from the current position of the second gimbal based on sensors in the first gimbal and the second gimbal (e.g., encoders, one or more inertial sensors at or near one or more motors of the gimbals).

[0169] In some embodiments, the HID feedback 204 includes information (e.g., position and / or speed) indicating navigation (e.g., movement) of the gimbal, which can be compared with the movement of the medical device 310 to determine whether a tool motion difference has occurred or whether the tool motion difference has exceeded a predetermined threshold (e.g., for providing kinesthetic tactile feedback). For example, when the gimbal is near the end of the travel range of a particular degree of freedom, the HID feedback 204 indicating the gimbal position or the gimbal is close to the end of the travel range is provided to the rolling tactile module 208 of the impedance control unit 206 for providing kinesthetic tactile feedback (e.g., increased damping). In some embodiments, when the gimbal exceeds a rate threshold during navigation, the HID feedback 204 including information indicating the speed of the gimbal or information indicating that the gimbal has exceeded a rate or speed threshold is provided to the rolling tactile module 208 of the impedance control unit 206 for providing kinesthetic tactile feedback.

[0170] In some embodiments, information indicating the navigation of the gimbal can be used to provide vibrotactile feedback. For example, when the gimbal reaches the end of the travel range of a particular degree of freedom, HID feedback 204 indicating that the gimbal has reached the end of the travel range is provided to the tactile buzz module 210 of the impedance control unit 206 for providing vibrotactile feedback.

[0171] In some embodiments, the HID feedback 204 includes information indicating whether the user needs to relax her grip on the HID. For example, when the user has over-articulated the gimbal, exceeding a certain threshold (e.g., after hitting a tactile boundary), the HID feedback 204 may include information indicating whether the user needs to relax her grip on the HID. When the processor detects an indication that the user needs to relax her grip on the HID, an output signal is sent from the HID feedback 204 to the tactile buzz module 210 of the impedance control unit 206. The HID feedback 204 causes a vibrotactile tactile feedback to be generated. The vibrotactile feedback provided in response to the information indicating whether the user needs to relax her grip on the HID informs the user that the gimbal has moved beyond the tool reach. In some cases, once the user's grip on the HID is relaxed, the system can redirect the gimbal (automatically) to match the tool posture. In some other cases, the user may need to reorient his or her hand position during a procedure, and pressing the clutch button may temporarily disengage the HID from the control operation of the instrument, allowing the user to reorient his or her hand position to re-grasp the HID during the procedure. When the user relaxes her grip, this gesture may be interpreted by the HID or robotic system as a return to the clutch state (e.g., reengaging the "clutch").

[0172] In some embodiments, HID feedback 204 is generated when a fault condition is detected. Examples of fault conditions associated with HID feedback 204 include detecting a collision of a gimbal (e.g., with an armrest or with another gimbal) or determining that a redundant joint cannot move (due to a collision).

[0173] In some embodiments, when the tactile buzz module 210 receives a signal from the patient-side feedback 202, a first type of tactile output is generated, and when a signal is received from the HID feedback 204, a second type of tactile output is generated. In some embodiments, the first type of tactile output is different from the second type of tactile output. In some embodiments, the first type of tactile output has a first vibration frequency component (e.g., a frequency between 10 Hz and 20 Hz), and the second type of tactile output has a second vibration frequency component different from the first vibration frequency component (e.g., a frequency between 50 Hz and 100 Hz or between 100 Hz and 200 Hz, etc.). In some embodiments, the first type of tactile output has a first vibration pattern (e.g., a sawtooth pattern), and the second type of tactile output has a second vibration pattern different from the first vibration pattern (e.g., a square wave pattern). In some embodiments, the first type of tactile output has a first duration (e.g., 0.4 seconds, 0.5 seconds, 0.6 seconds, 0.7 seconds, 0.8 seconds, 0.9 seconds, 1 second, or semi-continuous), and the second type of tactile output has a second duration different from the first duration (e.g., 0.1 seconds, 0.2 seconds, 0.3 seconds, etc.). In some embodiments, the first type of tactile output has a first number of chirps (3 chirps), and the second type of tactile output has a second number of chirps different from the first number of chirps (e.g., 5 chirps). The user may be able to more intuitively disambiguate between the two types of tactile outputs and more quickly identify problems that have occurred on the patient side or on the HID side.

[0174] When the system is operating, various feedbacks received from the patient side or the HID side may generate certain events, which in turn trigger either or both of the traditional haptic effect feedback and the buzz effect feedback.

[0175] In some embodiments, the impedance control unit 206 is implemented as one or more processors (e.g., a microprocessor, an application specific integrated circuit, etc.). For example, both the rolling haptic module 208 and the haptic buzz module 210 can be implemented in a single processor or two separate processors. In some embodiments, the gimbal roll motor torque control 218 is implemented in the same processor as the impedance control unit 206 or in a separate processor. In some embodiments, the robotic system 10 includes one or more processors for processing patient-side feedback 202 and / or HID feedback 204. In some embodiments, the circuit for processing patient-side feedback 202 and / or HID feedback 204 is included in one or more processors implementing the impedance control unit 206.

[0176] i. Scrolling tactile module

[0177] In some embodiments, the rolling haptic module 208 applies a damping function to a robotic joint (e.g., a robotic joint 24) to modify the resistance to the movement of the robotic joint (e.g., the damping function causes an increase in the resistance or torque to the movement of the robotic joint by applying a damping coefficient). In some embodiments, the joint velocity is received or determined, and the damping function applied to the robotic joint may also be based on the current velocity. In some embodiments, when the medical tool manipulated by the robotic joint moves in three-dimensional space, the current position and / or current velocity of the robotic joint is received or determined (e.g., by the control unit, by the impedance control 206). In some embodiments, each joint 24 reports its velocity and position (e.g., angular velocity and angular position) to the control unit. Additionally or alternatively, the joint 24 may report the angular force or torque applied by or to the joint.

[0178] In some embodiments, the control unit (e.g., impedance control 206) then determines the distance between the current position of the robot joint 24 (e.g., the rotational position of the robot joint 24) and the first motion limit of the robot joint (e.g., the rotational distance of the joint). In some embodiments, the rolling tactile module 208 implements kinesthetic tactile technology, which involves applying a force on the user that hinders the movement of the limb. Kinesthetic tactile feedback is generally not felt on the skin, but mainly on the muscles and tendons. In some embodiments, kinesthetic tactile feedback includes a continuous force applied against the user input. In some embodiments, kinesthetic tactile feedback involves the perception of body movement and the ability to detect changes in body position and movement without relying on information from the five senses. In some embodiments, kinesthetic tactile feedback simulates the real behavior of the robot arm.

[0179] The scroll haptic module 208 provides as its output an output control signal for generating kinesthetic haptic feedback at the HID.

[0180] ii. Tactile buzzing mode

[0181] The tactile buzz module 210 receives an input signal from the patient-side feedback 202 and / or the HID feedback 204. The input signal is sent to the event type classifier 212 in the tactile buzz 210. In some embodiments, the event type classifier 212 identifies the source of the received signal (e.g., from the patient-side feedback 202 or from the HID feedback 204) and the condition conveyed by the received signal (e.g., a gimbal collision, a failure, a sword fight, etc.), and provides an output signal to the profile generator 214. In some embodiments, the output signal is unique for each type of event or information. The profile generator 214 provides an output control signal corresponding to the identified event type indicated in the output signal (e.g., the profile generator 214 can provide a unique output control signal for each identified event type, so that the HID can provide a unique tactile or tactile feedback indicating the identified event type). The output control signal is used to generate a tactile signal at the HID. In some embodiments, a user can determine the type or profile of haptic signals to be associated with each event type (e.g., failure from HID feedback 2014, failure from patient-side feedback 202, gimbal collision, sword fight, medical device collision, camera collision, loose grip during re-clutch, etc.) In some embodiments, profile generator 214 automatically generates and assigns haptic signals for one or more of the event types.

[0182] The signal controller 216 generates a combined signal from the output control signal of the roll haptic module 208 and the output control signal of the haptic buzz module 210. In some embodiments, the signal controller 216 adds the output control signal from the roll haptic module 208 and the output control signal from the haptic buzz module 210. In some embodiments, the signal controller 216 adds a first weighted amount (e.g., a fraction of the output control signal, a multiple of the output control signal) of the output control signal from the roll haptic module 208 to a second weighted amount of the output control signal from the haptic buzz module 210 to generate a final output control signal for controlling a portion of the robot joint. In some embodiments, the portion of the robot joint corresponds to a gimbal roll motor, and the final output signal is delivered to the torque control loop 218 of the gimbal roll motor for torque control.

[0183] Fig.24How to combine the output signals from the rolling haptic module 208 and the haptic buzz module 210 according to some embodiments is shown. The main gimbal impedance control 209 representing the output from the rolling haptic module 208 is provided to the signal controller 216. In some embodiments, the main gimbal impedance control 209 is a non-periodic signal. The selected buzz profile 211 representing the output signal from the buzz haptic module 210 is also provided to the signal controller 216. In some embodiments, the selected buzz profile 211 is a periodic signal. In some embodiments, the selected buzz profile 211 is used to generate physical vibrations (e.g., periodic physical vibrations, such as alternating between high-frequency vibrations and low-frequency vibrations or alternating between a state in which vibrations are provided and a state in which vibrations are not provided). In some embodiments, the frequency of the physical vibrations is at least 10 Hz (e.g., at least 50 Hz, or between 50 Hz and 200 Hz). In some embodiments, the frequency is between 4 Hz and 10 Hz or about 6 Hz. The frequency profile of the vibration makes the buzzing effect easily identifiable by the user, thereby allowing information to be conveyed to the user (indicating an associated event) in a different and identifiable manner. The frequency of the physical vibration is selected to be away from normal remote operation main control (e.g., 5Hz or less) and scrolling haptic (e.g., kinesthetic haptic) feedback, so that the vibration only provides a vibrotactile haptic effect and does not affect existing operations. The waveform, amplitude, frequency, duration, repetition pattern, etc. can be further selected based on hardware modal analysis (e.g., hardware modal analysis can include determining different modes of vibration on the hardware and preventing or avoiding the use of excitation frequencies associated with these modes to avoid or reduce resonance) and various needs associated with haptic presentation for different applications.

[0184] In some embodiments, the buzz tactile or vibrotactile feedback has a varying amplitude (e.g., alternating high and low amplitude variations for events such as collisions or failures, and monotonically increasing or decreasing amplitude variations for events derived from excessive motion differences). In some embodiments, the signal controller 216 determines a weighted sum of the control signal from the main gimbal impedance control 209 and the output control signal from the selected buzz profile 211 (e.g., by adding a first weighting value corresponding to the control signal from the main gimbal impedance control 209 (e.g., a fraction of the output control signal or a multiple of the output control signal) and a second weighting value corresponding to the output control signal from the selected buzz profile 211) to generate a final output control signal for controlling a portion of the robot joint. In some embodiments, the portion of the robot joint corresponds to a gimbal roll motor, and the final output signal is delivered to a torque control loop 218 for torque control of the gimbal roll motor.

[0185] Return to Fig.23In some embodiments, the positioning platform 108 of the robotic system 10 includes a gimbal 106 at its distal end. The gimbal 106 also includes a plurality of motors. In some embodiments, a motor (e.g., G6) adjacent to the handle 104 (e.g., at the distal end of the gimbal 106) receives a superimposed vibration tactile control signal into its torque / current control loop in addition to receiving a kinesthetic tactile signal from the main gimbal impedance control to generate a buzzing vibration on top of the kinesthetic / rolling tactile signal. The G6 motor can provide a torque (e.g., a rotational resistance) about a rolling axis 136 to the handle 104 (and the first joint 128). In some embodiments, the rolling axis 136 is aligned with the longitudinal axis of the handle 104. In some embodiments, providing a vibration tactile signal at the distal end of the gimbal 106 helps the user sense the vibration tactile signal. In some embodiments, when the user applies torque to the handle 104 about the rolling axis 136, the rolling tactile signal causes the user to experience resistance (e.g., kinesthetic tactile feedback). The gimbal includes other motors, for example, motor G5 that provides torque about the yaw axis 138 and motor G4 that provides torque about the pitch axis 139 .

[0186] Various buzz profiles may be used to alert the user of different events, such as an impending physical collision between gimbals, between gimbals and positioning platform arms (e.g., the positioning platform 108 of a first gimbal collides with a second gimbal), between medical devices, or between robotic arms. The vibration buzz profile may also be used to alert the user of an impending physical collision between a gimbal and a console workspace boundary (including other components of the robotic system 10, such as a display, or any other object adjacent to the gimbals) (e.g., between the console 16 and the gimbal 106).

[0187] In some embodiments, the medical system includes a second tactile interface device that is different from the first tactile interface device (e.g., Fig.19 Two tactile interface devices are shown).

[0188] 3. Variable damping for tactile interface control.

[0189] In some embodiments, kinesthetic haptic feedback includes applying a torque (or force) by an actuator (e.g., a motor) to provide resistance to user input. In some embodiments, a rolling haptic associated with an HID involves applying variable damping during navigation and / or operation of the HID. In some embodiments, a position difference between a master device and a slave device is determined in order to provide a feedback force to an operator.

[0190] The variable damping method provides the appropriate level of damping resistance to the system and / or user based in part on how the user manipulates the HID. The damping algorithm may employ a damping function having multiple damping states. As an example, one damping state may provide a relatively low amount of resistance to the user (e.g., similar to moving a hand through water), while a different damping state may provide a relatively high amount of resistance to the user (e.g., imagine moving a hand through molasses). Another damping state may provide a variable amount of resistance depending on motion information received from a gimbal (e.g., from a joint). In some embodiments, the resistance may be proportional or inversely proportional to the motion information from the medical device and / or the gimbal.

[0191] A combination of multiple or variable damping states may be used to provide kinesthetic tactile feedback. For example, providing a low damping coefficient may indicate to the surgeon that there is no problem with the surgeon's gimbal movement, while providing a high damping coefficient may indicate to the surgeon that there may be a problem (e.g., approaching the end of the range of travel). Increasing the damping coefficient may indicate that an increased level of attention may be required (e.g., moving toward the end of the range of travel), and decreasing the damping coefficient may indicate that a decreased level of attention may be required (e.g., moving away from the end of the range of travel).

[0192] Fig.25A An example of a damping function used in the main gimbal impedance control 209 in the rolling haptic module 208 according to some embodiments is depicted. In some embodiments, a low damping coefficient is desired (state A.1), while a high damping coefficient is desired, for example, to provide a warning (state A.3). To avoid or minimize a sudden jump between the low damping state (state A.1) and the high damping state (state A.3), a transition damping region (state A.2) bridges state A.1 and state A.3.

[0193] The applied damping coefficient may be selected to modify the force or torque provided by one or more robot joints to provide resistance to the user input. For example, when the current speed or velocity of a portion of the robot user interface is within a first range (corresponding to, e.g., state A.1), the force or torque may be modified by a fixed amount, when the current speed or velocity of the portion of the robot user interface is within a second range (corresponding to, e.g., state A.2), the force or torque may be modified by a variable amount (e.g., increased in the depicted implementation), and when the current speed or velocity is within a second range greater than the first range (corresponding to, e.g., the depicted plateau of state A.3), the force or torque may be modified by another fixed amount.

[0194] Although the depicted damping regions provide a continuous transition, in some embodiments, the transition may not be continuous. For example, the transition may include several sub-regions, each with its own damping coefficient that is different from the damping coefficients of other sub-regions, eventually progressing to a high damping region.

[0195] Fig.25B An example of a portion of an output signal (e.g., voltage or current) provided to a HID (for generating haptic and / or tactile feedback) according to some embodiments is depicted. In some embodiments, at least a portion of the output signal from signal controller 216 originating from main gimbal impedance control 209 is provided to a torque / current control loop of a component of the HID. In some embodiments, the component of the HID is a gimbal roll motor (e.g., G6) of the HID. Fig.25B A control signal (e.g., current) provided to the HID as a function of time is shown, the control signal originating from the main gimbal impedance control 209 for providing kinesthetic tactile feedback. The same graph also depicts the tactile feedback or output (e.g., force) from the HID experienced by the user. In some embodiments, the tactile feedback is kinesthetic feedback. In some embodiments, the kinesthetic tactile feedback is provided for a duration of at least 0.5 seconds.

[0196] When the user begins to interact with the HID, a constant control signal may be sent to the HID to provide a fixed amount of resistance as a kinesthetic feedback for the user's movement of the HID. At time 220, one or more signals are received as part of the HID feedback 204 and / or the patient-side feedback 202, and the rolling haptic module 208 generates a variable damping signal to increase the resistance provided by the HID. For example, the tool motion difference causes the patient-side feedback 202 to be sent to the impedance control 206, and / or the gimbal navigation causes the HID feedback 204 to be sent to the impedance control 206. An example of a gimbal navigation that causes a signal from the HID feedback 204 to be sent is a gimbal navigating toward the end of the travel range of the gimbal. In some embodiments, at time 222, the gimbal approaches the end of its travel range, and a large damping coefficient is applied to make it more difficult to move the HID to the end of its travel range. When the HID moves away from the end of its travel range, the control signal decreases (to reduce the amount of damping).

[0197] Fig.25CA control signal (e.g., voltage or current) provided to the HID as a function of time is shown, the control signal originating from the tactile buzz module 210 for providing vibratory tactile haptic feedback to the user. At time 224, one or more signals are received from the HID feedback 204 and / or the patient-side feedback 202, and the tactile buzz module 210 causes physical vibrations to be generated at the HID. The physical vibrations are at least 50 Hz, or may be between 50 Hz and 200 Hz. The high-frequency profile of the vibrations makes the buzzing effect easily identifiable by the user, thereby allowing information to be conveyed to the user in a different and identifiable manner (such as compared to low-frequency kinesthetic feedback). For example, one or more of a sword fight, a collision, and / or other failures causes the patient-side feedback 202 to be sent to the impedance controller 206 to provide vibratory feedback. Additionally or alternatively, one or more of a gimbal collision, a gimbal navigation, a failure, or a relaxed grip during re-clutching causes the HID feedback 204 to be sent to the impedance controller 206 for providing vibratory feedback. An example of gimbal navigation that causes a signal from the HID feedback 204 to be sent to provide vibration feedback is movement of a gimbal that results in an impending collision with a second gimbal.

[0198] Fig.25D 2 shows a control signal (e.g., voltage or current) provided to the HID from a signal controller 216, which combines control signals from both the scrolling haptic module 208 and the haptic buzz module 210. In some embodiments, the control signal provided from the signal controller 216 is Fig.25B The signal shown is Fig.25C The sum (or weighted sum) of the signals is shown. Due to the different characteristics of kinesthetic haptic feedback and vibrotactile haptic feedback, the user may identify feedback from the HID at time 220, time 224, and time 222 that indicates various events associated with the medical device and / or the gimbal.

[0199] Fig.26 2 is a flow chart illustrating an example method 250 in which a tactile feedback device provides feedback to a user while the user uses a controller (e.g., a master device) to control a medical device (e.g., a slave device). The method 250 may be configured for hybrid control using both impedance control and admittance control. Although illustrated sequentially, the blocks of the method 250 may be implanted in other orders, or one or more of the blocks may occur substantially simultaneously. The method 250 begins at block 252, where the system receives a first input signal from one or more input sensors. At block 254, the system sends a first type of signal (e.g., a kinesthetic tactile feedback signal) to a first tactile interface device based at least on the first input signal for (e.g., providing or generating) a first type of feedback (e.g., kinesthetic tactile feedback).

[0200] At block 256, the system receives a second input signal from the one or more input sensors. At block 258, the system sends a second type of signal (e.g., a vibrotactile feedback signal) to the first tactile interface device based at least on the second input signal for (e.g., providing or generating) a second type of feedback (e.g., vibrotactile feedback).

[0201] 4. Implement systems and terminology.

[0202] Implementations disclosed herein provide systems, methods, and apparatus for a robotically enabled medical system. Various implementations described herein include a controller for a robotically enabled medical system.

[0203] It should be noted that, as used herein, the term "coupled" or other variations of the word coupled may indicate an indirect connection or a direct connection. For example, if a first component is "coupled" to a second component, the first component may be indirectly connected to the second component via another component or directly connected to the second component.

[0204] The position estimation and robot motion actuation functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term "computer-readable medium" refers to any available medium that can be accessed by a computer or processor. By way of example and not limitation, such a medium may include a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a compact disk read-only memory (CD-ROM) or other optical disk storage device, a magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired program code in the form of an instruction or data structure and can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transient. As used herein, the term "code" may refer to software, instructions, code or data that can be executed by a computing device or processor.

[0205] The method disclosed herein includes one or more steps or actions for implementing the method. The method steps and / or actions may be interchangeable with each other without departing from the scope of the claims. In other words, unless the correct operation of the method being described requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0206] As used herein, the term "plurality" means two or more. For example, a plurality of components indicates two or more components. The term "determine" encompasses a variety of actions, and thus, "determine" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, etc. Additionally, "determine" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), etc. Additionally, "determine" may include parsing, selecting, choosing, establishing, etc.

[0207] Unless explicitly stated otherwise, the phrase “based on” does not mean “based only on.” In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0208] As used herein, the term "about" or "approximately" refers to a measurement range of a length, thickness, amount, time period, or other measurable value. Such measurement ranges encompass variations of + / -10% or less, preferably + / -5% or less, more preferably + / -1% or less, and still more preferably + / -0.1% or less relative to the specified value, as long as such variations are appropriate in order to function in the disclosed devices, systems, and techniques.

[0209] The foregoing description of the disclosed specific implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these specific implementations will be apparent to those skilled in the art, and the general principles defined herein may be applied to other specific implementations without departing from the scope of the present invention. For example, it should be understood that one of ordinary skill in the art will be able to adopt a plurality of corresponding alternatives and equivalent structural details, such as equivalent ways of fastening, mounting, coupling or engaging tool parts, equivalent mechanisms for producing specific actuating motions, and equivalent mechanisms for delivering electrical energy. Therefore, the present invention is not intended to be limited to the specific implementations shown herein, but is given the widest scope consistent with the principles and novel features disclosed herein.

[0210] 5. A description of the technology that is the subject of the clause.

[0211] Some embodiments or implementations are described with reference to the following clauses:

[0212] Clause 1. A medical system comprising:

[0213] a first tactile interface device;

[0214] one or more input sensors;

[0215] one or more processors; and

[0216] a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to:

[0217] receiving a first input signal from the one or more input sensors;

[0218] Based at least on a first input signal for kinesthetic tactile feedback, the first tactile interface

[0219] The mouth device sends kinesthetic tactile feedback signals;

[0220] receiving a second input signal from the one or more input sensors; and

[0221] A vibrotactile feedback signal is sent to the first haptic interface device based at least on the second input signal for vibrotactile feedback.

[0222] Clause 2. The medical system of clause 1, further comprising a first robotic arm.

[0223] Clause 3. The medical system of clause 2, wherein the first input signal is based on a difference between a master command and a movement of the first robotic arm.

[0224] Clause 4. The medical system of clause 2 or clause 3, further comprising a second robotic arm different from the first robotic arm.

[0225] Clause 5. The medical system of clause 4, wherein the second input signal results from a collision between the first robotic arm and the second robotic arm.

[0226] Clause 6. The medical system of clause 4 or clause 5, wherein the second input signal is derived from a criterion including a distance between the first robotic arm and the second robotic arm being less than a first distance threshold.

[0227] Clause 7. The medical system of any one of Clauses 4 to 6, wherein the second input signal results from a criterion comprising a collision between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm.

[0228] Clause 8. A medical system according to any one of clauses 4 to 7, wherein the second input signal is derived from a criterion comprising a distance between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm being less than a second distance threshold.

[0229] Clause 9. The medical system of any one of clauses 4 to 8, wherein the second input signal is derived from a criterion comprising detection of a fault in the medical system.

[0230] Clause 10. The medical system of any one of clauses 4 to 9, wherein the second input signal results from a criterion comprising detection of the first tactile interface device at a predefined boundary.

[0231] Clause 11. The medical system of any one of Clauses 1 to 10, further comprising a second tactile interface device that is distinct and separate from the first tactile interface device.

[0232] Clause 12. The medical system of Clause 11, wherein the second input signal results from a criterion comprising a collision between the first tactile interface device and the second tactile interface device.

[0233] Clause 13. The medical system of Clause 11 or Clause 12, wherein the second input signal results from a criterion comprising a distance between the first tactile interface device and the second tactile interface device being less than a third distance threshold.

[0234] Clause 14. The medical system of any one of Clauses 1 to 13, wherein the kinesthetic tactile feedback comprises a continuous force applied against a user input.

[0235] Clause 15. A medical system according to any one of clauses 1 to 14, wherein:

[0236] The second input signal is derived from a criterion comprising a plurality of events; and

[0237] Different haptic feedback signals are selected as the vibrotactile feedback signal for respective events of the plurality of events.

[0238] Clause 16. The medical system of Clause 15, wherein the vibrotactile feedback comprises a component having a frequency of at least 10 Hz.

[0239] Clause 17. The medical system of any one of clauses 1 to 16, wherein the first tactile interface device further comprises a motor for providing a torque about a roll axis of the first tactile interface device in response to a user input on the first tactile interface device.

[0240] Clause 18. A method comprising:

[0241] receiving a first input signal from one or more input sensors;

[0242] sending a kinesthetic tactile feedback signal to a first tactile interface device based at least on the first input signal for kinesthetic tactile feedback;

[0243] receiving a second input signal from the one or more input sensors; and

[0244] A vibrotactile feedback signal is sent to the first haptic interface device based at least on the second input signal for vibrotactile feedback.

[0245] Clause 19. The method of clause 18, wherein the first input signal is used to control movement of a first robotic arm.

[0246] Clause 20. The method of clause 19, wherein the first input signal is based on a difference between a master command and a movement of the first robotic arm.

[0247] Clause 21. The method of clause 19 or clause 20, wherein the second input signal is used to control movement of a second robotic arm different from the first robotic arm.

[0248] Clause 22. The method of clause 21, wherein the second input signal results from a collision between the first robotic arm and the second robotic arm.

[0249] Clause 23. The method of clause 21 or clause 22, wherein the second input signal is derived from a criterion including a distance between the first robotic arm and the second robotic arm being less than a first distance threshold.

[0250] Clause 24. The method of any one of clauses 21 to 23, wherein the second input signal is derived from a criterion comprising a collision between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm.

[0251] Clause 25. A method according to any one of clauses 21 to 24, wherein the second input signal is derived from a criterion comprising a distance between a first surgical instrument connected to the first robotic arm and a second surgical instrument connected to the second robotic arm being less than a second distance threshold.

[0252] Clause 26. The method of any one of clauses 21 to 25, wherein the second input signal originates from a criterion comprising detection of a fault in the medical system.

[0253] Clause 27. The method of any one of clauses 21 to 26, wherein the second input signal results from a criterion comprising detection of the first tactile interface device at a predefined boundary.

[0254] Clause 28. The method of any of Clauses 18 to 27, further comprising determining a position of a second tactile interface device that is different and separate from the first tactile interface device.

[0255] Clause 29. The method of Clause 28, wherein the second input signal results from a criterion including a collision between the first tactile interface device and the second tactile interface device.

[0256] Clause 30. The method of clause 28 or clause 29, wherein the second input signal results from a criterion comprising a distance between the first tactile interface device and the second tactile interface device being less than a third distance threshold.

[0257] Clause 31. The method of any one of clauses 18 to 30, wherein the kinesthetic tactile feedback comprises a continuous force applied against a user input.

[0258] Clause 32. A method according to any one of clauses 18 to 31, wherein:

[0259] The second input signal is derived from a criterion comprising a plurality of events; and

[0260] Different haptic feedback signals are selected as the vibrotactile feedback signal for respective events of the plurality of events.

[0261] Clause 33. The method of Clause 32, wherein the vibrotactile feedback includes a component having a frequency of at least 10 Hz.

[0262] Clause 34. The method of any one of clauses 18 to 33, wherein the first tactile interface device further comprises a motor for providing a torque about a roll axis of the first tactile interface device in response to a user input on the first tactile interface device.

Claims

1. A medical system comprising: a first tactile interface device; one or more input sensors; one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving a first input signal from the one or more input sensors; sending a kinesthetic tactile feedback signal to the first tactile interface device based at least on the first input signal for kinesthetic tactile feedback; receiving a second input signal from the one or more input sensors; as well as A vibrotactile feedback signal is sent to the first haptic interface device based at least on the second input signal for vibrotactile feedback.

2. The medical system of claim 1, further comprising a first robotic arm. 3 . The medical system of claim 2 , wherein the first input signal is based on a difference between a master command and a movement of the first robotic arm.

4. The medical system according to claim 2 or 3, further comprising a second robotic arm different from the first robotic arm. 5 . The medical system of claim 4 , wherein the second input signal results from a collision between the first robotic arm and the second robotic arm.

6. The medical system of claim 4 or 5, wherein the second input signal is derived from a criterion comprising a distance between the first robotic arm and the second robotic arm being less than a first distance threshold.

7. The medical system of any one of claims 4 to 6, wherein the second input signal is derived from a criterion comprising a collision between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm.

8. The medical system of any one of claims 4 to 7, wherein the second input signal is derived from a criterion comprising a distance between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm being less than a second distance threshold.

9. The medical system of any one of claims 4 to 8, wherein the second input signal is derived from a criterion comprising the detection of a fault in the medical system.

10. The medical system according to any one of claims 4 to 9, wherein the second input signal results from a criterion comprising detection of the first tactile interface means at a predefined boundary.

11. The medical system of any one of claims 1 to 10, further comprising a second tactile interface device that is different and separate from the first tactile interface device.

12. The medical system of claim 11, wherein the second input signal results from a criterion comprising a collision between the first tactile interface device and the second tactile interface device.

13. The medical system of claim 11 or 12, wherein the second input signal is derived from a criterion comprising a distance between the first tactile interface device and the second tactile interface device being less than a third distance threshold.

14. The medical system of any one of claims 1 to 13, wherein the kinesthetic tactile feedback comprises a continuous force applied against a user input.

15. The medical system according to any one of claims 1 to 14, wherein: The second input signal is derived from a criterion comprising a plurality of events; and Different haptic feedback signals are selected as the vibrotactile feedback signal for respective events of the plurality of events.

16. The medical system of claim 15, wherein the vibrotactile feedback comprises a component having a frequency of at least 10 Hz.

17. The medical system of any one of claims 1 to 16, wherein the first tactile interface device further comprises a motor for providing a torque about a roll axis of the first tactile interface device in response to a user input on the first tactile interface device.

18. A method comprising: receiving a first input signal from one or more input sensors; sending a kinesthetic tactile feedback signal to a first tactile interface device based at least on the first input signal for kinesthetic tactile feedback; receiving a second input signal from the one or more input sensors; as well as A vibrotactile feedback signal is sent to the first haptic interface device based at least on the second input signal for vibrotactile feedback.

19. The method of claim 18, wherein the first input signal is used to control movement of a first robotic arm.

20. The method of claim 19, wherein the first input signal is based on a difference between a master command and a movement of the first robotic arm.

21. A method according to claim 19 or 20, wherein the second input signal is used to control movement of a second robotic arm different from the first robotic arm.

22. The method of claim 21, wherein the second input signal results from a collision between the first robotic arm and the second robotic arm.

23. The method of claim 21 or 22, wherein the second input signal is derived from a criterion comprising a distance between the first robotic arm and the second robotic arm being less than a first distance threshold.

24. The method of any one of claims 21 to 23, wherein the second input signal results from a criterion comprising a collision between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm.

25. The method of any one of claims 21 to 24, wherein the second input signal is derived from a criterion comprising a distance between a first surgical instrument coupled to the first robotic arm and a second surgical instrument coupled to the second robotic arm being less than a second distance threshold.

26. A method according to any one of claims 21 to 25, wherein the second input signal is derived from a criterion comprising the detection of a fault in a medical system.

27. A method according to any one of claims 21 to 26, wherein the second input signal originates from a criterion comprising detection of the first tactile interface means at a predefined boundary.

28. The method of any one of claims 18 to 27, further comprising determining a position of a second tactile interface device that is different and separate from the first tactile interface device.

29. The method of claim 28, wherein the second input signal results from a criterion comprising a collision between the first tactile interface device and the second tactile interface device.

30. The method of claim 28 or 29, wherein the second input signal results from a criterion comprising a distance between the first tactile interface device and the second tactile interface device being less than a third distance threshold.

31. A method according to any one of claims 18 to 30, wherein the kinesthetic tactile feedback comprises a continuous force applied against a user input.

32. A method according to any one of claims 18 to 31, wherein: The second input signal is derived from a criterion comprising a plurality of events; and Different haptic feedback signals are selected as the vibrotactile feedback signal for respective events of the plurality of events.

33. The method of claim 32, wherein the vibrotactile feedback comprises a component having a frequency of at least 10 Hz.

34. The method of any one of claims 18 to 33, wherein the first tactile interface device further comprises a motor for providing a torque about a roll axis of the first tactile interface device in response to a user input on the first tactile interface device.