Instrument Anomaly Detection in Surgical Robot Systems

Abstract

Machine learning models are used in detecting anomalies in the operation of instrument drive chains in surgical robotic systems. For example, normal operation (e.g., cable force) based on operation information (e.g., position and / or load) is predicted by the machine learning model, and the prediction is then compared to actual operation. The comparison results in early detection of anomalies based on the machine-learned prediction and corresponding incorporation of historical operation.

Description

[Technical Field]

[0001] Embodiments relate to detecting anomalies in instruments used in teleoperation by surgical robotic systems. [Background technology]

[0002] Minimally-invasive surgery (MIS), such as laparoscopic surgery, involves techniques intended to reduce tissue damage during surgical procedures. MIS may be performed using a robotic system that includes one or more robotic arms for manipulating surgical tools based on commands from a remote operator. In robotic MIS systems, it may be desirable to define and maintain high positional accuracy of surgical instruments supported by the robotic arms.

[0003] Surgical instruments for robotic arms may share a similar design; for example, a tool may have an end effector including a robotic wrist and one or more jaws. The end effector may include tools for grasping, cutting, and suturing, among other surgical tasks. A cable system couples the end effector to actuators within a tool drive that can drive multi-axis motion (e.g., pitch and yaw) of the end effector. For example, four actuators using four cables drive an end effector that has a robotic wrist with a pair of jaws. Summary of the Invention [Problem to be solved by the invention]

[0004] Abnormalities can occur due to wear on the surgical instrument or driveline, or due to interference. Unexpected forces can arise due to factors such as the presence of debris, increased friction due to frayed cables, and unexpected external side loads. Such events can lead to snapping of the instrument cable or impede tool tip movement, adversely affecting the surgeon's ability to perform the procedure. A faster response or replacement is desirable to avoid problems during use on the patient. [Means for solving the problem]

[0005] By way of preamble, the preferred embodiments described below include methods, systems, instructions, and computer-readable media for anomaly detection in instruments of a surgical system. A machine learning model is used in detecting anomalies in the operation of an instrument drive chain. For example, normal operation (e.g., cable force) based on operational information (e.g., position and / or load) is predicted by the machine learning model, and the prediction is then compared to actual operation. The comparison results in early detection of anomalies based on the machine-learned prediction and corresponding incorporation of historical operation.

[0006] In a first aspect, a method for detecting an anomaly in an instrument of a surgical system is provided. Measurements are received from a sensor. The sensor senses a drive chain of the instrument. During use of the instrument, a first movement of the drive chain is predicted by a machine learning model in response to input of the measurements. An anomaly is detected based on a comparison of the predicted first movement output by the machine learning model with an actual movement of the drive chain. An indication of the detection of the anomaly is output.

[0007] In a second aspect, a surgical robotic system for anomaly detection is provided. The surgical tool is connected to a respective number of actuators by a first number of cables. The surgical tool is connected such that actuation of the actuators moves the surgical tool. A first sensor is configured to sense positions of the actuators. A processor is configured to detect anomalies by applying a machine learning network. The machine learning network is configured to receive the positions and output predicted cable forces.

[0008] In a third aspect, a surgical robotic system for anomaly detection is provided. The robotic arm is configured to hold and operate a surgical tool. The processor is configured to detect anomalies by applying a machine learning model. The machine learning model is configured to receive past motion information and output predicted motion.

[0009] The present invention is defined by the following claims, but nothing in this section should be taken as a limitation on those claims. Additional aspects and advantages of the present invention are discussed below in conjunction with the preferred embodiments, and may then be claimed independently or in combination. [Brief explanation of the drawings]

[0010] Embodiments of the present invention are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference symbols indicate similar elements. It should be noted that references to "an" or "one" embodiment of the present invention in this disclosure are not necessarily to the same embodiment, but rather mean at least one. Also, for purposes of brevity and reducing the total number of figures, a given figure may be used to illustrate features of more than one embodiment of the present invention, and not all elements in a figure may be required for a given embodiment. [Figure 1] FIG. 1 illustrates an exemplary operating room environment with a surgical robotic system, in accordance with aspects of the subject technology. [Figure 2]1 is a schematic diagram illustrating one exemplary design of a robotic arm, tool drive, and cannula loaded with a robotic surgical tool, in accordance with aspects of the subject technology. FIG. [Figure 3A] 1A-1C are schematic diagrams illustrating exemplary tool drives with adjacent and non-adjacent loaded tools, respectively, in accordance with aspects of the subject technology; [Figure 3B] 1A-1C are schematic diagrams illustrating exemplary tool drives with adjacent and non-adjacent loaded tools, respectively, in accordance with aspects of the subject technology; [Figure 4A] FIG. 1 is a schematic diagram illustrating an end effector of an exemplary grasping instrument having a robotic wrist, a pair of opposing jaws, and a pulley and cable system for coupling the robotic wrist and the pair of jaws to an actuator of a tool drive, in accordance with aspects of the subject technology. [Figure 4B] FIG. 1 is a schematic diagram illustrating an end effector of an exemplary grasping instrument having a robotic wrist, a pair of opposing jaws, and a pulley and cable system for coupling the robotic wrist and the pair of jaws to an actuator of a tool drive, in accordance with aspects of the subject technology. [Figure 5] FIG. 1 is a block diagram of one embodiment of a surgical robotic system for machine learning model-based anomaly detection. [Figure 6] FIG. 10 is a block diagram of another embodiment of a surgical robotic system that uses machine learning models to detect anomalies. [Figure 7] An exemplary long-short-term memory (LSTM) cell for a machine learning model is shown. [Figure 8] 1 illustrates an example machine learning model using LSTM for anomaly detection. [Figure 9] FIG. 1 is a flowchart diagram and corresponding logic for a method of machine learning network-based anomaly detection in driving a surgical instrument, according to one embodiment. [Figure 10] 1 illustrates one embodiment of a cart-based robotic system configured for diagnostic and / or therapeutic use (e.g., bronchoscopy); and [Figure 11] 1 illustrates one embodiment of a table-based robotic system configured for a procedure (e.g., laparoscopy). DETAILED DESCRIPTION OF THE INVENTION

[0011] Machine learning models are used in predicting unexpected drive chain behavior in surgical instruments. For example, unexpected cable forces in a cable-driven tool cable are detected during teleoperation. An anomaly detection framework is used to predict cable forces based on historical trends, which then alerts the user to impending failures. As another example, predictions are used as a proxy for expected values ​​for a given encoder and / or load sensor. Models are machine-trained to learn the dynamics between (a) changes in encoder position and / or load and (b) the evolution of torque distribution in the articulation cable.

[0012] Based on the prediction, corrective action can be taken prior to failure, which may include driving the surgical instrument less aggressively to complete the current task, or replacing and / or repositioning the surgical instrument after an alert is issued.

[0013] The proposed framework is more robust in detecting anomalies, in contrast to other model-based techniques. With machine training, training data may be obtained from the actual operation of the instrument, accounting for unmodeled effects such as posture-dependent friction, cable elongation, and hysteresis. Furthermore, including data from multiple tools allows the machine learning model to generalize across multiple tools.

[0014] The following discussion first introduces an exemplary robotic surgical system (see FIGS. 1-4B). FIGS. 5-8 illustrate an embodiment for anomaly detection using an example or alternative robotic surgical system. FIG. 9 illustrates a method for anomaly detection. The embodiment of FIGS. 5-9 uses the application of machine learning models for faster and / or more accurate anomaly detection.

[0015] Figures 1-4B illustrate one embodiment of a cable-driven robotic surgical instrument and a robotic surgical system for using the instrument. Figures 10 and 11 illustrate other embodiments of the robotic surgical system. Any of these or other embodiments may be used in the anomaly detection system of Figures 5 and / or 6 and / or in the anomaly detection method of Figure 9. The illustrations of Figures 5, 6, and 9 may use the embodiment of Figures 1-4B, but may instead use the embodiment of Figure 10 or 11. Other cable-driven surgical instruments, robotic arms, and / or robotic surgical systems may be used.

[0016] In general, an end effector including a robotic wrist and / or one or more jaws may be coupled to an actuator through metal cables or wires. The wires may function in pairs, for example, where pulling on one wire applies an opposing force to the other wire, and thus the robotic wrist may be an antagonistic robotic wrist. While jaws are used in this example, other applicable surgical robotic instruments may also be used. Applicable surgical robotic instruments include graspers, forceps, scissors, needle holders, retractors, pliers, and cautery instruments, among others.

[0017] 1 is a diagram illustrating an exemplary operating room environment including a surgical robotic system 100 in accordance with aspects of the subject technology. As shown in FIG. 1, the surgical robotic system 100 includes a surgeon console box 120, a control tower 130, and one or more surgical robotic arms 112 located on a surgical robotic platform 110 (e.g., a table, bed, etc.), with surgical tools having end effectors attached to the distal ends of the robotic arms 112 for performing surgical procedures. While the robotic arms 112 are shown as table-mounted systems, in other configurations the robotic arms may be mounted to a cart, ceiling or sidewall, or another suitable support surface.

[0018] Generally, a user, such as a surgeon or other operator, can remotely control (e.g., teleoperate) the robotic arm 112 and / or surgical instrument using a user console box 120. The user console box 120 may be located in the same operating room as the robotic system 100, as shown in FIG. 1 . In other environments, the user console box 120 may be located in an adjacent or nearby room, or may be teleoperated from a remote location in another building, city, or country. The user console box 120 may include a seat 122, a foot-operated control 124, one or more handheld user interface devices 126, and at least one user display 128 configured to display, for example, a view of a surgical site within a patient. As shown in the exemplary user console box 120, a surgeon positioned in the seat 122 and viewing the user display 128 may operate the foot-operated control 124 and / or the handheld user interface device 126 to remotely control the robotic arm 112 and / or a surgical instrument mounted at the distal end of the arm.

[0019] In some variations, the user may also operate the surgical robotic system 100 in an "over the bed" (OTB) mode, in which the user is at the patient's side and simultaneously operates the robotic-driven tool / attached end effector and the manual laparoscopic tool (e.g., with a handheld user interface device 126 held in one hand). For example, the user's left hand may operate the handheld user interface device 126 to control the robotic surgical components, while the user's right hand may operate the manual laparoscopic tool. Thus, in these variations, the user may perform both robotically-assisted MIS and manual laparoscopic surgery on the patient.

[0020] During an exemplary procedure or surgery, the patient is sterilely prepped and draped to achieve anesthesia. Initial access to the surgical site may be performed manually with the robotic system 100 in a stowed or retracted configuration to facilitate access to the surgical site. Once access is complete, initial positioning or preparation of the robotic system may be performed. During the procedure, the surgeon in the user console box 120 may utilize the foot controls 124 and / or user interface devices 126 to operate the various end effectors and / or imaging system and perform the surgery. Manual assistance may also be provided at the procedure table by sterile, gowned personnel who may perform tasks including, but not limited to, retracting tissue or performing manual repositioning or tool changes involving one or more robotic arms 112. Non-sterile personnel may also be present to assist the surgeon at the user console box 120. When a procedure or surgery is completed, the robotic system 100 and / or the user console box 120 may be configured or set to facilitate one or more post-operative procedures, including, but not limited to, cleaning and / or sterilization of the robotic system 100 and / or entering or printing (whether electronic or hard copy) medical records, for example, via the user console box 120.

[0021] In some embodiments, communication between the robotic platform 110 and the user console box 120 may be via a control tower 130, which can translate user commands from the user console box 120 into robot control commands and transmit them to the robotic platform 110. The control tower 130 can also transmit status signals and feedback from the robotic platform 110 back to the user console 120. The connections between the robotic platform 110, the user console box 120, and the control tower 130 may be via wired and / or wireless connections, may be proprietary, and / or may be made using any of a variety of data communication protocols. Any wired connections may optionally be integrated into the floor and / or walls or ceiling of the operating room. The surgical robotic system 100 may provide video output to one or more displays, including displays in the operating room and remote displays accessible via the Internet or other network. The video output or feed may also be encrypted to ensure privacy, and all or portions of the video output may be stored on a server or electronic medical record system.

[0022] FIG. 2 is a schematic diagram illustrating one exemplary design of a robotic arm 112, a tool driver, and a cannula loaded with a robotic surgical tool, in accordance with aspects of the subject technology. The surgical robotic arm 112 is configured by design, shape, circuitry, and / or programming to hold and operate a surgical tool 220. As shown in FIG. 2, the exemplary surgical robotic arm 112 may include multiple links (e.g., link 204) and multiple actuation joint modules (e.g., joint 202) for actuating the multiple links relative to one another. The joint modules may include various types, such as pitch joints or roll joints, which may substantially constrain the movement of adjacent links relative to one another about a particular axis. The exemplary design of FIG. 2 also shows a tool driver 210 attached to the distal end of the robotic arm 112. The tool driver 210 may include a cannula 214 coupled to its end for receiving and guiding a surgical instrument 220 (e.g., an endoscope, stapler, grasper, etc.). The surgical instrument (or "tool") 220 may include an end effector 222 at the distal end of the tool. Multiple joint modules of the robotic arm 112 can be actuated to position and orient a tool drive 210 that actuates the end effector 222 for robotic surgery.

[0023] 3A and 3B are schematic diagrams illustrating exemplary tool drives with adjacent and non-adjacent loaded tools, respectively, in accordance with aspects of the present technology. As shown in FIGS. 3A and 3B, in one variation, the tool drive 210 may include an elongated base (or “stage”) 310 having a longitudinal track 312 and a tool carriage 320 slidably engaged with the longitudinal track 312. The stage 310 may be configured to couple to a distal end of a robotic arm such that articulation of the robotic arm positions and / or orients the tool drive 210 in space. Note that the tool carriage 320 may be configured to receive a tool base 352 of a tool 220, which may also include a tool shaft 354 extending from the tool base 352 through the cannula 214, with an end effector 222 (not shown) disposed at its distal end.

[0024] It should be noted that the tool carriage 320 may actuate the series of articulations of the end effector, such as via a cable system or conductors (the terms "cable" and "wire" are used interchangeably) operated and controlled by an actuation drive (the terms "actuator," "motor," and "drive" are used interchangeably). The tool carriage 320 may include actuation drives of different configurations. For example, a rotary shaft drive may include a motor with a hollow rotor and a planetary gear transmission disposed at least partially within the hollow rotor. The multiple rotary shaft drives may be arranged in any suitable manner. For example, the tool carriage 320 may include six rotary drives 322A-322F arranged in two rows extending longitudinally along the base, with the rotary drives slightly staggered to reduce the width of the carriage and increase the compactness of the tool drive. As clearly shown in FIG. 3B, rotary drives 322A, 322B, and 322C may be generally arranged in a first row, and rotary drives 322D, 322E, and 322F may be generally arranged in a second row slightly longitudinally offset from the first row.

[0025] 4A and 4B are schematic diagrams illustrating an exemplary grasping instrument end effector having a robotic wrist, a pair of opposing jaws, and a pulley and cable system for coupling the robotic wrist and pair of jaws to an actuator of a tool drive, in accordance with aspects of the subject technology. While the following tool model is described with reference to an exemplary surgical robotic grasping instrument, it should be noted that the proposed anomaly detection can be adapted to any tool including an end effector in which a cable or other drive is coupled to a tool shaft used to control the end effector. Similar tools include, but are not limited to, grasping instruments, grippers, forceps, needle holders, retractors, and cautery instruments. Any number of wrist joints, such as one, zero, two, or more than those shown in FIGS. 4A and 4B, may be used. Any number of corresponding cables 405, such as one, two, three, four, or more, may be used to control the end effector. In the example of FIGS. 4A and 4B, four cables 405 are used to control the end effector's three degrees of freedom.

[0026] 4A , a pair of opposing jaws 401A and 401B are movably coupled to a first yoke 402 of a robot wrist via an elongated axle 412 along a first axis 410. The first yoke 402 may be movably coupled to a second yoke 403 of a robot wrist via a second elongated axle 422 along a second axis 420. The pair of jaws 401A and 401B may each be coupled to or integrally formed with pulleys 415A and 415B, respectively, via the elongated axle 412, such that both jaws can rotate about axis 410. Pulleys 425A, 425B, 425C, and 425D are coupled to the elongated axle 422 and rotate about axis 420. Pulleys 425A, 425B, 425C, and 425D are arranged with a first set of pulleys 425B and 425C on one side of yoke 402 and a second set of pulleys 425A and 425D on the other side of yoke 402. Pulleys 425A and 425C are outer pulleys and pulleys 425B and 425D are inner pulleys. Similarly, a third set of pulleys 435A, 435B, 435C, and 435D are coupled to a third elongated axle 432 and rotate about an axis 430 that is parallel to axis 420.

[0027] The gripping instrument 220 can be actuated to move one or both of the jaws 401A and 401B in various ways about the axis 410. For example, the jaws 401A and 401B may open and close relative to one another. The jaws 401A and 401B may also be actuated to rotate together as a pair to provide yaw movement of the gripping instrument 220. Additionally, the first yoke 402, pulleys 415A and 415B, and jaws 401A and 401B may rotate about the axis 420 to provide pitch movement of the gripping instrument 220. These movements of the tool's robot wrist and / or jaws may be achieved by controlling four independent cables 405A-405D. 4A, cable 405A can begin (or terminate) on one side of pulley 415A and travel along pulleys 425A and 435A, while cable 405B is configured to terminate on the other side of pulley 415A and travel through pulleys 425B and 435B. Similarly, another pair of cables 405C and 405D can be coupled to jaw 401B. For example, cable 405C extends from one side of pulley 415B to pulleys 425C and 435C, while cable 405D travels through pulleys 425D and 435D and terminates on the other side of pulley 415B. The third set of pulleys 435A, 435B, 435C, and 435D are positioned to keep the cables 405A-405D fixed to the second set of pulleys 425A-425D and prevent the cables from slipping or sliding relative to the pulleys 425A-425D.

[0028] Controlling the movement of the grasping instrument via four independent cables 405 has several advantages. One advantage may be a reduction in the number of cables extending from the tool base 352 to the robot wrist compared to typical commercially available designs that use six cables 405 (or three cable loops with six cable ends). Having fewer cables 405 may reduce the tool size and complexity of the wrist assembly, which may be beneficial for minimally invasive surgical or non-surgical applications. Furthermore, having four independent cable loops instead of two or three cable loops not only allows for independent control of the tension of each cable 405 without the need to pre-tension the cables 405, but also allows for variable compliance of the wrist joint and improved sensitivity to external loads. Furthermore, the tension of each cable may be independently readjusted, further improving tool performance.

[0029] As shown in FIGS. 4A and 4B, the grasping device can be actuated to move jaws 401A and 401B in various ways, such as grip (e.g., jaws rotating independently about axis 410), yaw (e.g., jaws rotating together about axis 410), and pitch (e.g., jaws rotating about axis 420) (three degrees of freedom), by imparting motion to one or more of pulleys 415A, 415B, 425A, 425B, 425C, and 425D, thereby imparting motion to first yoke 402 and / or one or both of jaws 401A and 401B. Cables 405A-405D can be grouped into two antagonistic pairs, i.e., when one cable of the antagonistic pair is actuated or tensioned and the other cable is relaxed, the jaws rotate in one direction. On the other hand, when only the other cable is tensioned, the jaws rotate in the opposite direction.

[0030] For example, cables 405A and 405B are a first antagonistic pair for moving jaw 401A, and cables 405C and 405D are a second antagonistic pair for controlling jaw 401B. When cable 405A is tensioned (e.g., by at least one of rotary drives 322a-322f) while cable 405B is slack, jaw 401A closes (moves toward opposing jaw 401B). Conversely, when cable 405B is tensioned and cable 405A is slack, jaw 401A opens (moves away from opposing jaw 401B). Similarly, when cable 405C is tensioned, it closes jaw 401B (moves toward opposing jaw 401A), and cable 405D opens jaw 401B (moves away from opposing jaw 401A) while the other cable is slack. As another example, the gripping force between jaws 401A and 401B can be achieved by maintaining tension on both cables 405A and 405C after the jaws are closed (contacted with each other) (while cables 405B and 405D are relaxed).

[0031] If both cables of an antagonistic pair are tensioned simultaneously while both cables of the other pair are slack, pulley 415A or pulley 415B will not rotate. Instead, first yoke 402, along with jaws 401A and 401B, is caused to pitch about axis 420 by pulleys 415A and 415B. For example, if both cables of a pair 405A and 405B are tensioned simultaneously while pair 405C and 405D are slack, the jaws (together with yoke 402) will pitch out of the page. On the other hand, if both cables 405C and 405D are tensioned simultaneously and pair 405A and 405B are left slack, the jaws will pitch into the page.

[0032] 4B is a schematic diagram illustrating exemplary angle definitions for various movements of gripping instrument 220, in accordance with aspects of the subject technology. The angles are defined with reference to axes 410 and 420, as well as axis 452 of first yoke 402 and axis 453 of second yoke 403. For example, as shown in FIG. 4B, the angle between axis 452 and axis 453 (θ1) may represent the angle of rotation of yoke 402 about axis 420, which may also represent the pitch angle (θ ピッチ ) (whereas in FIG. 4A , the jaws remain in the reference position, i.e., there is no pitch movement, so axis 452 of yoke 402 is superimposed on axis 453 of yoke 403). In addition, angles (θ2) and (θ3) may represent the angles between each of jaws 401A and 401B and axis 452 of yoke 402 (as the origin), respectively. To distinguish between sides of axis 452, angles (θ2) and (θ3) may have different signs. For example, as illustrated in FIG. 4B , angle (θ2) is negative and angle θ(3) is positive.

[0033] The angular position and gripping force of the distal end effector of the robotic surgical instrument are controlled. The control system may include a feedback loop with position, torque, and / or velocity feedback from the actuators and force feedback measured on four conductors to provide the desired position and gripping force. Other sensors may be included. In one embodiment, each actuator and corresponding cable in the drive chain includes an encoder or other position sensor to determine the rotational position of the actuator and / or cable, and a load sensor (e.g., a torque sensor) to determine the load or force on the actuator and / or cable.

[0034] In some implementations, the actuator controllers may operate in a position plus feedforward current mode. For example, the position control may drive the distal end effector to a desired angular position in space based on position feedback, while the gripping force control provides additional feedforward current based on the gripping force measured by a four-conductor load cell to achieve the desired gripping force between the opposing members or jaws 401A-401B.

[0035] 10 illustrates another embodiment of a surgical robotic system, such as one configured for bronchoscopy, which may include a cable drive or other drive chain from an actuator or drive unit 1028 to a tool or instrument 1013. Robotic-enabled medical systems may be configured in a variety of ways depending on the particular procedure.

[0036] FIG. 10 illustrates one embodiment of a cart-based, robot-enabled system 1000 configured for diagnostic and / or therapeutic bronchoscopy. During bronchoscopy, the system 1000 may include a cart 1011 with one or more robotic arms 1012 for delivering a medical instrument, such as a steerable endoscope 1013, which may be a procedure-specific bronchoscope for bronchoscopy, to a natural orifice access point (i.e., the mouth of a patient positioned on a table in this example) for delivering diagnostic and / or therapeutic tools. As shown, the cart 1011 may be positioned adjacent to the patient's upper torso to provide access to the access point. Similarly, the robotic arms 1012 may be actuated to position a bronchoscope relative to the access point. The configuration in FIG. 10 may also be utilized when gastrointestinal (GI) procedures are performed using a gastroscope, which is a specialized endoscope for GI procedures.

[0037] Once the cart 1011 is properly positioned, the robotic arm 1012 can insert the steerable endoscope 1013 into the patient robotically, manually, or a combination thereof. As shown, the steerable endoscope 1013 may include at least two telescoping components, such as an inner leader section and an outer sheath section, each coupled to a separate instrument drive from a set of instrument drives 1028, each coupled to the distal end of a respective robotic arm 1012. This linear arrangement of the instrument drives 1028, which facilitates coaxial alignment of a portion of the leader with a portion of the sheath, creates a “virtual rail” 1029 that can be repositioned in space by manipulating one or more robotic arms 1012 to different angles and / or positions. The virtual rails described herein are not the physical structure of the system, but rather the arrangement of other structures. Translation of the instrument drive 1028 along the virtual rail 1029 nests a portion of the inner leader relative to a portion of the outer sheath, or advances or retracts the endoscope 1013 from the patient. The angle of the virtual rail 1029 can be adjusted, translated, or pivoted based on the clinical application or physician preference. For example, in a bronchoscopy, the angle and position of the illustrated virtual rail 1029 represents a compromise between providing the physician access to the endoscope 1013 and minimizing friction resulting from bending the endoscope 1013 into the patient's mouth.

[0038] After insertion, the endoscope 1013 can be directed downstream of the patient's trachea and lungs using precise commands from the robotic system until it reaches the target destination or surgical site. To enhance navigation through the patient's pulmonary network and / or reach a desired target, the endoscope 1013 can be manipulated to telescope a portion of the inner leader out of a portion of the outer sheath and to enhance articulation and increase bend radius. The use of a separate instrument driver 1028 also allows the leader and sheath portions to be driven independently of each other.

[0039] The system 1000 may also include a movable tower 1030 that may be connected to the cart 1011 via a support cable and provide support for controls, electronics, fluidics, optics, sensors, and / or power to the cart 1011. Placing such functionality in the tower 1030 allows the form factor of the cart 1011 to be smaller, allowing the surgeon and his or her staff to more easily adjust and / or reposition the cart 1011. Furthermore, the division of functionality between the cart / table and the support tower 1030 reduces clutter in the operating room and facilitates improved clinical workflow. The cart 1011 may be positioned near the patient, while the tower 1030 may be housed in a remote location so as not to get in the way during the procedure.

[0040] To support the robotic system described above, the tower 1030 may include computer-based control system component(s) that store computer program instructions in a non-transitory computer-readable storage medium, such as, for example, a persistent magnetic storage drive, a solid-state drive, or the like. Execution of these instructions, whether performed in the tower 1030 or the cart 1011, may control the entire system or its subsystem(s). For example, when executed by a processor in a computer system, the instructions may cause the robotic system components to actuate associated carriages and arm mounts, operate a robotic arm, and control a medical instrument. For example, in response to receiving control signals, motors in articulating joints of a robotic arm may position the arm in a particular pose.

[0041] The tower 1030 may also include pumps, flow meters, valve controls, and / or fluid access to provide controlled irrigation and aspiration capabilities for the system, which may be deployed through the endoscope 1013. The tower 1030 may include voltage and surge protection designed to provide filtered and protected power to the cart 1011, thereby avoiding the need to place power transformers and other auxiliary power components within the cart 1011, making the cart 1011 smaller and more mobile. The tower 1030 may also include support for sensors deployed throughout the robotic system 1000. Similarly, the tower 1030 may also include electronic subsystems for receiving and processing signals from deployed electromagnetic (EM) sensors. The tower 1030 may also be used to house and position EM field generators for detection by EM sensors within or on the medical instrument.

[0042] The tower 1030 may also include a console box 1031 in addition to other console boxes available to the rest of the system, such as a console box mounted on top of a cart. The console box 1031 may include a user interface and a display screen, such as a touch screen, for the physician operator. The console boxes in the system 1000 are generally designed to provide both pre-operative and real-time procedure information, such as robotic control and navigation and localization information for the endoscope 13. If the console box 1031 is not the only console box available to the physician, the console box 1031 may be used by a second operator, such as a nurse, to monitor the patient's health or vitals and the operation of the system 1000, and to provide procedure-specific data, such as navigation and localization information. In other embodiments, the console box 1031 is housed in a separate body from the tower 1030.

[0043] Embodiments of a robot-enabled medical system may also incorporate a patient table. Incorporation of a table reduces the amount of capital equipment in the operating room by removing carts and allows for better access to the patient. FIG. 11 illustrates an embodiment of such a robot-enabled system. The system 1136 includes a support structure or column 1137 for supporting a platform 1138 (shown as a "table" or "bed") across the floor. Similar to cart-based systems, the end effector of the robotic arm 1139 of the system 1136 includes an instrument drive designed to manipulate elongated medical instruments such as a bronchoscope. In practice, a C-arm for providing fluoroscopic imaging can be positioned above the patient's upper abdominal region by placing emitters and detectors around the table 38.

[0044] The column 1137 may include one or more carriages 1143, shown in the system 1136 as ring-shaped, which may serve as a base for one or more robotic arms 1139. The carriages 1143 may translate along a vertical column interface extending the length of the column 1137 to provide different vantage points from which the robotic arms 1139 may be positioned to reach the patient. The carriage(s) 1143 may rotate about the column 1137 using mechanical motors positioned within the column 1137, allowing the robotic arms 1139 to have access to multiple sides of the table 1138, such as both sides of the patient. In embodiments with multiple carriages 1143, the carriages 1143 may be individually positioned on the column 1137 and may translate and / or rotate independently of the other carriages. While the carriage 1143 need not surround the column 1137, or even be circular, the ring shape as shown facilitates rotation of the carriage 1143 around the column 1137 while maintaining structural balance. Rotation and translation of the carriage 1143 enable the system 1136 to align medical instruments, such as endoscopes and laparoscopes, with different access points on the patient. In other embodiments (not shown), the system 1136 may include a patient table or patient bed having an adjustable arm support in the form of a bar or rail extending alongside it. One or more robotic arms 1139 may be attached to the adjustable arm support (e.g., via a shoulder with an elbow articulation) that may be vertically adjustable. By providing vertical adjustment, the robotic arm 1139 may advantageously be stored compactly under the patient table or patient bed and then raised during a procedure.

[0045] The robotic arm 1139 may be mounted to the carriage 1143 via a set of arm mounts 1145 that include a series of articulating joints that may rotate independently and / or extend telescopically to provide additional configurability for the robotic arm 1139. Note that the arm mounts 1145 may be positioned on the carriage 1143 such that, when the carriage 1143 is appropriately rotated, the arm mounts 1145 may be positioned either on the same side of the table 1138, on opposite sides of the table 1138 (as shown in FIG. 11 ), or on adjacent sides of the table 1138 (not shown).

[0046] The column 1137 structurally provides support for the table 1138 and a path for vertical translation of the carriage 1143. Internally, the column 1137 may include a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the translation of the carriage 1143 based on the lead screw. The column 1137 may also transmit power and control signals to the carriage 1143 and a robotic arm 1139 mounted thereon.

[0047] In one embodiment, the robotic surgical system of FIG. 11 is used for laparoscopy. In laparoscopic procedures, minimally invasive instruments may be inserted into the patient's anatomy through small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instruments include an elongated, rigid member, such as a shaft, that is used to access the patient's anatomy. After distension of the patient's abdominal cavity, the instruments can be oriented to perform surgical or medical tasks, such as grasping, cutting, ablation, suturing, etc. In some embodiments, the instruments are not necessarily surgical in the sense of directly manipulating tissue, but may include viewing equipment, such as a laparoscope, for viewing procedures. FIG. 11 illustrates an embodiment of a robotic-compatible table-based system configured for laparoscopic procedures. As shown in FIG. 11 , the carriage 1143 of the system 1136 can be rotated and vertically adjusted to position a pair of robotic arms 1139 on either side of the table 1138 so that instruments 1159 can be positioned using arm mounts 1145 to pass through minimal incisions on either side of the patient and reach the patient's abdominal cavity. Laparoscopy may then be performed using one or more drives with the laparoscope as a tool or instrument.

[0048] FIG. 5 illustrates an exemplary surgical robotic system including a processor 540 interacting with a drive chain 500. The drive chain 500 may be, for example, the drive chain of FIGS. 3A-4B. Other drive chains using cables, gears, actuators, motors, pulleys, guides, and / or other force transmission devices may be used. The drive chain may be part of a surgical system for manual manipulation or guidance instead of a robotic system for teleoperation (e.g., a cable-driven laparoscopic system). In the example of FIG. 5, one or more cables 405 connect the actuator 322 to the tool 220 for the drive chain 500.

[0049] The surgical tool 220 is a grasping instrument having two jaws, but may be another type of surgical instrument. The surgical tool 220 connects to a number of actuators 322 by a number of cables 405, such as four cables 405 connecting four engaged actuators 322. The cables 405 enable actuation of the actuators 322 to move the surgical tool 220, such as opening and closing the jaws and / or rotating in pitch and / or yaw.

[0050] Sensors 510, 520, such as force sensors (e.g., load cells) and / or position sensors (e.g., encoders for absolute position), are used for control. A processor 540, using instructions in a non-transitory computer-readable storage medium, uses information from the sensors 510, 520 to control the operation of the actuator 322.

[0051] Encoder 520 senses the position of the actuator. Encoder 520 may sense angular position with or without a number of rotations (e.g., 0 to 360 degrees or 0 to N, where N is greater than 360 degrees). Linear position may be sensed instead. In one embodiment, encoder 520 is an absolute position sensor. Other position sensors and / or encoders may also be used.

[0052] The load sensor 510 is a strain gauge or other sensor for sensing a load or force. The force may be longitudinal and / or rotational. The force applied to the cable 405 and / or the tool 220 by the actuator 322 is sensed. In one embodiment, the load sensor 510 is positioned on an arm of the actuator 322. In another embodiment, the load sensor 510 is positioned on the cable 405. In yet another embodiment, the load sensor 510 is a current sensor that senses the current drawn by the actuator 322 to apply a force to the cable 405.

[0053] Other drive chains with the same or different sensors may be used. For example, only an encoder or other position sensor may be provided. As another example, only a load sensor, other torque sensor, and / or other force sensor may be provided. In yet other embodiments, a speed sensor may be provided instead of or in addition to the load sensor 510 and / or the encoder 520.

[0054] Processor 540 may be a general purpose processor, an application specific integrated circuit, a field programmable gate array, a digital signal processor, a controller, an artificial intelligence processor, a tensor processor, a graphics processing unit, a digital circuit, an analog circuit, a combination thereof, and / or any other now known or later developed processor for robotic control. Processor 540 may be configured with software, hardware, and / or firmware to detect anomalies during teleoperation.

[0055] Processor 540 is configured to detect anomalies in drive chain 500 by application of a machine learning model. The machine learning model is configured to receive past and / or current operating information and output a predicted operation. Processor 540 is configured to detect anomalies as the output of the machine learning model (e.g., a model trained to output a classification as normal or abnormal) and / or by comparing the output of the model to actual measured operation (e.g., the model predicts normal and an anomaly is detected when the measured operation is a threshold difference from the predicted operation).

[0056] Various sources of input may be used in the model, such as position, load (torque), and / or speed. Inputs may be for one drive chain 500 (e.g., a single cable), for a subset of the drive chains 500 for a tool 220, or for all of the drive chains 500 for a tool 220. Other inputs may include operating parameters such as the type of tool 220, the procedure being performed, the time associated with the procedure, and / or the stage of the procedure (e.g., suturing). Any combination of inputs may be used.

[0057] Various outputs may be generated by the machine learning model. For example, a cable force, such as a load or torque applied to the cable 405, may be predicted. As another example, the output may be a tool position, a speed, and / or a torque of the tool 220. In yet another example, the output may be a classification of whether or not there is an anomaly, the type of anomaly, and / or the severity of the anomaly. A level of uncertainty or certainty in the output by the machine learning model may also be output.

[0058] 6, a machine learning model 610 stored in memory 600 is configured by being trained to receive position 620 and / or load 622 and output cable force (load) 624. The machine learning model 610 outputs a predicted cable force in response to receiving or inputting position(s) 620 and load(s) 622. While this example is used herein, other inputs and / or outputs may be used based on training data previously used to train the machine learning model 610.

[0059] Any number of past inputs may be used to generate the current output(s). For example, a sequence of M positions 620 and loads 622 may be input to predict the current or next cable force 624, where M is an integer greater than or equal to 1 (e.g., M=4, 8, 16, 32, 50, or 64). The sequence is defined by a moving window of M inputs, where the window slides over time to select the M most recent inputs. In response to a given sequence of inputs, any number of current and future outputs may be predicted. For example, N outputs may be generated, where N is an integer greater than or equal to 1. If N=1, the output may be the output for the current or next time increment (e.g., cable force 624). If N=2 or greater, it provides a current prediction or even a positive prediction in the future.

[0060] The current output may be an expected output, such as an expected normal cable force 624. The machine learning model 610 was trained to output a normal output (e.g., cable force 624). In one embodiment, the processor 540 is configured to detect anomalies where the normal movement predicted by the machine learning model deviates from the actual movement of the surgical tool 220 by the robotic arm 112. The processor 540 is configured to detect anomalies by comparing the predicted cable force 624 to the actual cable force sensed by the load sensor 510 at the current or next time. A classification of whether abnormal or normal may instead be output directly by the model 610.

[0061] The machine learning model 610 was pre-trained using machine learning, an offline training phase in which the goal is to identify an optimal set of values ​​for the model's 610 learnable parameters that can be applied to many different inputs (i.e., sequences of actuator positions and / or loads over time). These machine-learned parameters can then be used during operation (a testing or application phase) to quickly predict the behavior (e.g., cable forces) of the drive chain 500. Once trained, the machine learning model 610 is used in an online processing phase in which a series of measurements are input, and the behavior of the drive chain 500 is output based on the model values ​​learned during the training phase.

[0062] Various types of machine learning models may be used, such as support vector machines, neural networks, Bayesian, or other. In one embodiment, the model is a neural network, such as a fully connected neural network or a convolutional neural network. Any architecture or layer structure for machine learning may be used. The architecture defines the structure, learnable parameters, and relationships between parameters. In one embodiment, the architecture of the model to be trained and the resulting trained model 610 is a recurrent neural network (RNN) 612. Any of a variety of recurrent neural networks may be used, such as those based on one or more long short-term memories (LSTMs) 614. The recurrent neural network 612 may alternatively be a transformer network.

[0063] Figure 7 shows one embodiment of an LSTM 614. An LSTM is an artificial RNN architecture used in the field of deep learning. An LSTM defines various weights (e.g., σ) that can be learned. An input x passes through neural network layers (e.g., σ and tanh) with operators x and + to generate a cell state h, where C is an input or output term from another LSTM 614. T is a time series index. Other LSTM configurations may be used. A transformer may be used instead of an LSTM.

[0064] FIG. 8 shows one embodiment of the RNN 612. The RNN 612 models time series data. In this embodiment, the dynamics of the drive chain operation are modeled using telemetry data. The sequence of LSTMs 614 is arranged in forward and backward chains or layers. A sequence of inputs x is input to the forward and backward chains or layers. The output y is generated from an activation layer fed from the LSTM 614. Other RNNs with different configurations may be used. Other RNNs without the LSTM 614 or a transformer may be used. Any number of hidden layers may be provided between the input layer and the output layer.

[0065] The model 610, such as a recurrent neural network 612, is machine-learned. Deep machine training is performed. Deep learning is used to train the model 610. Other machine learning methods may be used. Many (e.g., hundreds or thousands) of samples of inputs to the model and ground truth outputs are collected as training data. For example, data from testing (e.g., bench testing or manufacturer testing) is collected as training data. Inputs are measured over time and resulting outputs are measured over time during testing. Many sample sequences for a given type of surgical instrument 220 and / or drive chain 500 may be collected during a malfunction or other test of the drive chain 500 and / or surgical instrument 220.

[0066] Machine learning learns both the features of input data and the transformation of those features into a desired output. Backpropagation, RMSprop, ADAM, or another optimization is used to learn the values ​​of the learnable parameters of the model 610. When training is supervised, the difference (e.g., L1, L2, or mean squared error) between the estimated output and the ground truth output is minimized.

[0067] Once trained, the model 610 is applied during teleoperation. For example, the machine learning model 610 is used to predict normal operation. The training data used for training represents normal operation, and therefore the prediction is the output given the input, which occurs during operation without faults or interference. In an alternative embodiment, the machine learning model 610 is trained to predict normal and abnormal operation. The model 610 is trained to indicate the current class of operation (e.g., normal or abnormal). In other embodiments, the model is trained to identify when abnormal operation is likely to occur.

[0068] The machine learning model 610 is pre-trained and then used as trained. Fixed values ​​of the learned parameters are used for application. The learned values ​​and network architecture determine the output from the input. The same learned weights or values ​​are used during application in teleoperation. The model and values ​​of the learnable parameters do not change from patient to patient, at least over a given time (e.g., weeks, months, or years) or a given number of surgeries (e.g., tens or hundreds). These fixed values ​​and corresponding fixed models are applied to different patient input sequences continuously and / or by different processors. The model may be updated, such as by retraining or replacement, but does not learn new values ​​as part of application to a given patient.

[0069] Retraining may also be used. For example, the machine learning model 610 (e.g., RNN 612) is trained using training data from bench testing (i.e., non-patient data). Then, the same surgical instrument and / or the same type of surgical instrument and robotic arm are used in teleoperation with a patient. Data from the teleoperation with the patient is collected. Retraining may then be performed using this second set of data. The model 610 starts with previously learned values ​​for the parameters of the architecture, and then optimization is performed again using a second data set or both the original data set and the second data set. The result is a retrained model 610 ready for application. The values ​​of one or more of the learnable parameters may be the same or different due to the retraining.

[0070] In another example of retraining, the original training data is based on the type of surgical instrument. The model 610 is trained using data for that type of surgical instrument. The resulting trained model 610 may later be retrained using training data from a different type of surgical instrument. The retraining may use both data sets or may only use training data for the other type of surgical instrument. By using data from more than one instrument in the original training or retraining, one model 610 may be trained to predict different types of instruments.

[0071] 9 illustrates one embodiment of a method for anomaly detection in a surgical robotic system's instrument 220. A trained machine learning model 610 is used to rapidly predict the behavior of the drive chain 500 of the surgical instrument 220. This prediction provides, or is used to provide, detection of the anomaly.

[0072] The method is performed by the surgical robotic system of Figures 1, 2, or 3A and 3B or another surgical robotic system. In one embodiment, the surgical robotic system of Figures 5 and / or 6 performs the method. The method is performed on the surgical instrument of Figures 4A and 4B or another instrument. In the example of Figures 4A and 4B, four cables 405 control three degrees of freedom of the cable-driven instrument 220. Other numbers of cables 405 that are equal to or less than the number of degrees of freedom of the instrument 220 may be used. A programmed processor 540 (also referred to herein as processing logic) of a control unit or other controller performs the method during teleoperation. A non-transitory memory 600 can store instructions for the programmed processor 540 to detect anomalies.

[0073] These acts may be performed in the order shown or in a different order. For example, acts 900 and 910 may be performed simultaneously and / or repeatedly. Additional, different, or fewer acts may be provided. For example, act 930 may not be provided. As another example, processor 540 detects the type of surgical instrument 220 connected to robotic arm 112 and selects machine learning model 610 to use for anomaly detection based on the type. In yet another example, acts are provided for connecting surgical instrument 220 to robotic arm 112 and controlling homing and / or teleoperation.

[0074] Once the tool drive 210 is connected to the tool instrument 220 and any homing, calibration, and / or staging is complete, the robotic arm 122, drive chain 500, and surgical instrument 220 are used in teleoperation while at least a portion of the surgical instrument 220 is within a patient. A motor (e.g., actuator 322) in the tool drive 210 rotates. The motor rotation is performed under position control, although other control modes may be used. The surgeon controls the teleoperation. Input commands from the surgeon are converted into position (or other) control signals for the actuator 322.

[0075] In operation 900, the processor receives measurements from one or more sensors 510, 520. The sensors 510, 520 measure the movement of the drive chain 500 during teleoperation of the instrument 220 attached to the surgical robotic arm 112.

[0076] The surgical robotic system operates the cable-driven instrument 220 during surgery. For example, movement of the user interface device 126 and / or foot pedal 124 is translated into articulation commands using inverse kinematics or the like. The actuator 322 is moved and controlled. The cable-driven instrument 220 is moved during surgery. Alternatively or additionally, the jaws are opened and closed.

[0077] Periodically (e.g., every 1 / 10th of a second), one or more sensors 510, 520 sense the movement of the drive chain 500. Different sensors 510, 520 may sense at different frequencies.

[0078] In one embodiment, both the motor or actuator position and the load torque are sensed over time. Sensing occurs at the actuator 322, along the cable 405, and / or at other portions of the drive chain 500. For example, an encoder 520 senses the position of the actuator 322, and a load sensor 510 senses the load or torque on the actuator 322 and / or the cable 405. Various inputs may be used, such as position as commanded by a user, load resulting from teleoperation, velocity, or any combination thereof. Data (measurements) from a torque sensor (e.g., load sensor 510) alone can be used to detect anomalies, but may be sensitive to thresholds and result in false positives.

[0079] In operation 910, the processor 540 predicts the behavior of the drive chain 500 during teleoperation. The prediction is performed, at least in part, by the machine learning model 610. The machine learning model 610 quickly predicts behavior (i.e., with less processing than an algorithm) in response to input measurements. The machine learning model 610 is pre-trained on surgical instruments 220 of the same type, and therefore can accurately predict the behavior of a given surgical instrument 220 using measurements from that instrument's 220 behavior. Each type of tool 220 is consistent. For example, friction characteristics remain relatively the same for a given type of instrument 220.

[0080] In one embodiment, the prediction is a prediction of normal operation. Given inputs, the machine learning model 610 predicts the operation that would occur without interference or damage along the drive chain 500 (e.g., damage to the actuator 322, cable 405, and / or implement 220). For example, a series of travel positions and loads for a given drive chain 500 or all drive chains 500 connected to the same implement 220 are input to the model 610, which outputs a load, torque, or another cable force for the next time increment. The predicted cable force is a prediction of what operation would be like if normal. The dynamics between the input positions and / or loads and the output cable forces are modeled through training, allowing the trained model to make predictions.

[0081] The machine learning model 610 is an RNN 612, although other neural networks or models may be used. For example, the model 610 is an RNN 612 that uses one or more LSTMs 614 and / or transformers.

[0082] The model 610 is trained based on simulation, bench testing, or both. The training data is collected from testing or simulation, not from use on patients. Alternatively, the training data is collected from use on patients. In one embodiment, an initial model 610 is trained and used as trained based on simulation and / or bench test data. This data is readily available from initial testing of the tool during the manufacturing process. The extensive data collected during testing is leveraged to build surrogate models specific to each tool type. The model 610 can then be retrained using data or examples collected during use on patients. Trained models 610 are developed for each tool type, but these models 610 can be further refined over time as more data becomes available. Similarly, the model 610 may be trained for one type of instrument 220. Data collected from testing and / or use on a patient of a different type of instrument 220 (e.g., the original is a forceps and the other type is a scalpel) is then used to retrain the model 610 for use with this other type of instrument 220.

[0083] Measurements are continuously received in operation 900. A moving window of the M most recent measurements defines the measurements for input to the model. For each time increment, model 600 predicts one or more values ​​for behavior for future and / or current time. Operations 900 and 910 repeat as the time window moves to define measurements in a first-in, first-out manner.

[0084] In operation 920, the processor 540 detects an anomaly. The anomaly may be detected based on the model 610 outputting that an anomaly has occurred in a measurement. In one embodiment, the anomaly is detected using the output. The model 610 outputs normal operation, such as a predicted cable force. This predicted cable force for normal or undisturbed, undamaged operation is compared to the actual cable force measured for the predicted time increment. A comparison of the predicted operation output by the machine learning model 610 to the actual operation of the drive chain 500 indicates whether an anomaly has occurred. The actual operation is normal if the predicted and actual are within a threshold difference of each other. An anomaly has occurred if the actual value deviates from the expected value by a threshold amount.

[0085] Because the machine learning model 610 was trained using training data for normal operation and / or to detect anomalies regardless of the source of the anomaly, the anomaly detection works regardless of which source the anomaly occurs. Deviations from normal may be due to one or more of a variety of issues. The anomaly is detected without relying on detecting the anomaly as being due to a specific one of a variety of issues. A comparison between the predicted output and the actual output may be used to evaluate unexpected events, such as a cable break or blockage, higher than expected friction within the tool 220, and / or an unexpected external load on the wrist. Anomalies are detected regardless of their source. In other embodiments, the model 610 is trained to indicate the type of anomaly, and / or the processor 540 determines the type of anomaly.

[0086] Anomalies may be detected based on predictions from a given time. Alternatively, deviations over a window of actual time (period) from predictions are combined (e.g., summed or averaged) or used together. Anomalies are detected when the comparison for each time exceeds a threshold difference over the period, or when the combination exceeds a threshold. For example, the cable forces for the next "n" time steps are predicted for each of the time steps. If the cumulative error between the predicted and actual cable forces for a specified time window is greater than a predetermined threshold, a warning is issued to the user.

[0087] In operation 930, the processor 540 outputs an indication of the detection of the anomaly. The processor 540 outputs to memory, a display, a robotic system controller, or a report. The processor 540 generates an indication such as text, a warning light, a flag, a graphic, or repair instructions. For example, a user interface may display a warning of the detected anomaly so that the user can prepare for an operational disruption. As another example, the controller operates the robotic arm 112 and / or the surgical instrument 220 differently (e.g., more slowly or with less force) due to the detection of the anomaly. In another example, commands are issued to test, repair, or replace the drive chain 500 and / or the instrument 220.

[0088] In an exemplary embodiment, the RNN 612 of Figure 8 is trained and used. The input training data is normalized and scaled, for example, torque telemetry and encoder position are normalized and scaled. The model is trained using the following hyperparameters: time step 50, learning rate: 1e-5, optimizer: ADAM, loss function: mean squared error, batch size: 32, epochs: 500 with early stopping (patience = 30), network architecture: 2-layer LSTM with ELU activation and 4 units (see Figure 8), training-validation split: 0.8 / 0.2, input: [{enc(t-50), tau(t-50)}, {enc(t-49), tau(t-49)}...{enc(t), 0}], output: [tau(t)] (enc is the encoder and tau is the torque or load). Test cases: data collected on different tools for four scenarios: normal run, fault pitch, fault yaw, and fault jaw. Training data is collected for random movements in the articulation degrees of freedom on the entire system for run, collected on different tools simultaneously. Data was collected at 200 Hz, resulting in 68,331 data points. For preprocessing, the post-homing torque was subtracted from the data, the torque and encoder position were normalized, and the normalized data was used for training. The final output is the network output of model 610 plus the initial torque distribution. Other training values ​​and / or configurations may be used.

[0089] In the test, a normal run results in no anomaly being detected. In the case of a pitch fault, an anomaly is detected. Cables 1, 2 and 5, 6 are detected as having an incorrect torque distribution over the specified time interval. In the case of a yaw fault, an anomaly is detected. Cables 1, 6 and 2, 5 are detected as having an incorrect torque distribution over the specified time interval. For a jaw fault, an anomaly is detected in the region where the cable torque changes abruptly. Cables 1, 5 and 2, 6 are detected as having an incorrect torque distribution over the specified time interval. In the case of tools with different friction profiles, the network generalizes to friction variations between tools, and therefore no anomaly is detected. With an external load, an anomaly is detected. The number of detections is a function of the threshold. Thresholding is performed on the error between the predicted output and the actual output, not the actual output itself.

[0090] In other embodiments, different types of inputs are used. Images are used as inputs rather than position, torque, and / or velocity. For example, a camera (e.g., an endoscope) provides images as feedback to estimate the position of the tool tip relative to commands issued by the actuator and / or user. The model 610 is trained to indicate the position and / or generate images indicative of the position (e.g., using a generative neural network). The actual and predicted are compared to identify anomalies. The error or difference of the estimate from the actual may be used to assess whether the tool is responding as expected.

[0091] The above description of illustrated embodiments of the present invention, including what is described in the Summary below, is not intended to be exhaustive or to limit the invention to the precise form disclosed. While specific embodiments and examples of the present invention have been described herein for illustrative purposes, those skilled in the art will recognize that various modifications are possible within the scope of the present invention. These modifications can be made to the present invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed herein. Rather, the scope of the present invention will be determined entirely by the following claims, which are to be construed in accordance with established principles of claim interpretation.

[0092] [Embodiment] (1) A method for anomaly detection in an instrument of a surgical system, the method comprising: receiving measurements from a sensor, the sensor sensing a drive chain of the tool; predicting a first movement of the drive chain during use of the tool, the predicting being performed by a machine learning model in response to input of the measurements; and detecting the anomaly based on a comparison of the predicted first movement output by the machine learning model and an actual movement of the drive chain; and outputting an indication of the detection of the anomaly. (2) The method of embodiment 1, wherein predicting includes predicting the first operation as a normal operation. (3) The method of embodiment 2, wherein detecting includes detecting the anomaly as a deviation from the normal operation, such that the deviation is due to one or more of a variety of issues, without relying on detecting the anomaly as being due to a particular one of a variety of issues. (4) The method of embodiment 1, wherein the machine learning model is trained based on simulation, bench testing, or both simulation and bench testing, and further comprising retraining the machine learning model based on examples from use in patients. (5) The method of embodiment 1, wherein the instrument is of a first type and the machine learning model has been trained on an instrument of a second type different from the first type and retrained on the first type.

[0093] (6) The method of embodiment 1, wherein receiving includes receiving the measurements as motor position and motor load torque over time for a motor of the drive chain. (7) The method of embodiment 6, wherein the sensor includes an encoder configured to measure the motor position and a load sensor configured to measure the load torque. (8) The method of embodiment 1, wherein predicting includes predicting the first movement as a cable force on a cable of the drive chain. (9) The method of embodiment 1, further comprising repeating the receiving and the predicting over time, and wherein detecting the anomaly comprises detecting when the comparison exceeds a threshold difference over a period of time. (10) The method of embodiment 1, wherein predicting includes predicting using the machine learning model including a recurrent neural network.

[0094] (11) The method of embodiment 10, wherein predicting comprises predicting using the recurrent neural network including long-short-term memory. (12) The method of embodiment 1, wherein receiving includes receiving the measurement value from the sensor while the sensor is sensing in the drive chain for teleoperation of the instrument attached to a robotic arm, and predicting includes predicting during teleoperation. (13) A surgical robot system for anomaly detection, the surgical robot system comprising: a surgical tool connected by a first number of cables to a respective number of actuators, the surgical tools connected such that actuation of the actuators moves the surgical tools; a first sensor configured to sense a position of the actuator; and a processor configured to detect the anomaly by applying a machine learning network, the machine learning network configured to receive the position and output a predicted cable force. (14) The surgical robot system of embodiment 13, further comprising a second sensor configured to sense a load on the actuator, wherein the machine learning network is configured to output the predicted cable force in response to receiving the position and the load. (15) The surgical robot system of claim 14, wherein the processor is configured to detect the anomaly by comparing the predicted cable force with an actual cable force.

[0095] (16) A surgical robot system as described in embodiment 13, wherein the machine learning network includes a recurrent neural network having long-short-term memory. (17) The surgical robot system of embodiment 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the first set being based on non-patient data and the second set being based on patient data. (18) The surgical robot system of embodiment 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the second set being based on a tool type of the surgical tool, and the second set being based on a different type of tool. (19) A surgical robotic system for anomaly detection, the surgical robotic system comprising: a robotic arm configured to hold and operate a surgical tool; a processor configured to detect the anomaly by applying a machine learning model, the machine learning model configured to receive past motion information and output predicted motion. (20) The surgical robot system of embodiment 19, wherein the machine learning model includes a neural network configured by training to output the predicted operation as normal operation, and the processor is configured to detect the anomaly when the predicted normal operation deviates from the actual operation of the surgical tool by the robot arm.

[0096] (21) The surgical robot system of claim 19, further comprising a load sensor configured to detect a load on an actuator and an encoder configured to detect a position of the actuator, wherein the past motion information includes a sequence of the positions and the loads, and the predicted motion includes a cable force.

Claims

1. 1. A method for anomaly detection in an instrument of a surgical system, the method comprising: receiving measurements from a sensor, the sensor sensing a drive chain of the tool; predicting a first movement of the drive chain during use of the tool, the predicting being performed by a machine learning model in response to input of the measurements; detecting the anomaly based on a comparison of the predicted first movement output by the machine learning model and an actual movement of the drive chain; and outputting an indication of the detection of the anomaly.

2. The method of claim 1 , wherein predicting comprises predicting the first behavior as normal behavior.

3. 3. The method of claim 2, wherein detecting comprises detecting the anomaly as a deviation from the normal operation, such that the deviation is due to one or more of a variety of problems, without relying on detecting the anomaly as due to a particular one of a variety of problems.

4. 10. The method of claim 1, wherein the machine learning model is trained based on simulation, bench testing, or both simulation and bench testing, and further comprising retraining the machine learning model based on examples from use in patients.

5. 2. The method of claim 1, wherein the instrument is of a first type and the machine learning model has been trained on an instrument of a second type different from the first type and retrained on the first type.

6. The method of claim 1 , wherein receiving comprises receiving the measurements as motor position and motor load torque over time for a motor of the drive chain.

7. The method of claim 6 , wherein the sensors include an encoder configured to measure the motor position and a load sensor configured to measure the load torque.

8. The method of claim 1 , wherein predicting comprises predicting the first movement as a cable force on a cable of the drive chain.

9. 10. The method of claim 1, further comprising repeating the receiving and the predicting over time, and wherein detecting the anomaly comprises detecting when the comparison exceeds a threshold difference over a period of time.

10. The method of claim 1 , wherein predicting comprises predicting using the machine learning model comprising a recurrent neural network.

11. The method of claim 10 , wherein predicting comprises predicting using the recurrent neural network that includes long short-term memory.

12. 10. The method of claim 1, wherein receiving comprises receiving the measurements from the sensor while the sensor senses in the drive chain for teleoperation of the instrument attached to a robotic arm, and wherein predicting comprises predicting during teleoperation.

13. 1. A surgical robotic system for anomaly detection, the surgical robotic system comprising: a surgical tool connected by a first number of cables to a respective number of actuators, the surgical tools connected such that actuation of the actuators moves the surgical tools; a first sensor configured to sense a position of the actuator; and a processor configured to detect the anomaly by applying a machine learning network, the machine learning network configured to receive the position and output a predicted cable force.

14. 14. The surgical robotic system of claim 13, further comprising a second sensor configured to sense a load on the actuator, wherein the machine learning network is configured to output the predicted cable force in response to receiving the position and the load.

15. The surgical robotic system of claim 14 , wherein the processor is configured to detect the anomaly by comparing the predicted cable force with an actual cable force.

16. The surgical robotic system of claim 13 , wherein the machine learning network comprises a recurrent neural network with long short-term memory.

17. 14. The surgical robotic system of claim 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the first set based on non-patient data and the second set based on patient data.

18. 14. The surgical robotic system of claim 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the second set based on a tool type of the surgical tool, and the second set based on a different type of tool.

19. 1. A surgical robotic system for anomaly detection, the surgical robotic system comprising: a robotic arm configured to hold and operate a surgical tool; a processor configured to detect the anomaly by applying a machine learning model, the machine learning model configured to receive past motion information and output predicted motion.

20. 20. The surgical robotic system of claim 19, wherein the machine learning model includes a neural network configured by training to output the predicted movement as normal movement, and the processor is configured to detect the anomaly when the normal movement as predicted deviates from an actual movement of the surgical tool by the robotic arm.

21. 20. The surgical robotic system of claim 19, further comprising: a load sensor configured to detect a load on an actuator; and an encoder configured to detect a position of the actuator, wherein the past motion information includes a sequence of the positions and the loads, and the predicted motion includes a cable force.

Images