Abnormality detection in instrument in surgical robotic system

Abstract

A machine learning model is used to detect anomalies in operation of an instrument drive chain of a surgical robotic system. For example, normal operation (e.g., cable force) based on operation information (e.g., location and / or load) is predicted by the machine learning model, and then the prediction is compared to actual operation. The comparison enables early detection of anomalies based on a combination of machine learning prediction and corresponding historical operations.

Description

Technical Field

[0001] Embodiments relate to detecting anomalies in instruments used in teleoperation by a surgical robotic system. Background Art

[0002] Minimally invasive surgery (MIS) such as laparoscopic surgery involves techniques designed to reduce tissue damage during surgical procedures. MIS can be performed using a robotic system that includes one or more robotic arms for manipulating surgical tools based on commands from a teleoperator. In a robotic MIS system, it may be desirable to establish and maintain high positional accuracy of the surgical instruments supported by the robotic arms.

[0003] Surgical instruments for robotic arms may share similar designs, for example, a tool may have an end effector that includes a robotic wrist and one or more jaws. The end effector may include tools for grasping, cutting, suturing, and other surgical tasks. A cable system couples the end effector to an actuator in the tool drive that can drive multi-axis motion (e.g., pitch and yaw) of the end effector. For example, four actuators using four cables drive an end effector having a robotic wrist with a pair of jaws.

[0004] Anomalies may occur, such as due to wear of the surgical instrument or drive train or due to interference. Unexpected forces may be generated due to factors such as the presence of debris, increased friction due to cable wear, unexpected external loads, etc. Such events may cause the instrument cable to break or impede tool tip motion and adversely affect the surgeon's ability to perform the procedure. Early response or replacement is required to avoid problems with patient use. Summary of the invention

[0005] By way of introduction, the preferred embodiments described below include methods, systems, instructions, and computer readable media for anomaly detection in instruments of surgical systems. A machine learning model is used to detect anomalies in the operation of an instrument drive train. For example, normal operation (e.g., cable force) based on operational information (e.g., position and / or load) is predicted by the machine learning model, and the prediction is then compared to actual operation. The comparison enables early detection of anomalies based on a combination of the machine learning prediction and the corresponding historical operation.

[0006] In a first aspect, a method for anomaly detection in an instrument of a surgical system is provided. A measurement is received from a sensor. The sensor senses at a drive chain of the instrument. During use of the instrument, a first operation of the drive chain is predicted by a machine learning model in response to the measured input. An anomaly is detected based on a comparison of the predicted first operation output by the machine learning model with an actual operation of the drive chain. An indication of a detection of the anomaly is output.

[0007] In a second aspect, a surgical robotic system for anomaly detection is provided. A surgical tool is connected by a first number of cables to a corresponding number of actuators. The surgical tool is connected so that actuation of the actuators moves the surgical tool. A first sensor is configured to sense a position of the actuator. A processor is configured to detect anomalies by applying a machine learning network. The machine learning network is configured to receive the position and output a predicted cable force.

[0008] In a third aspect, a surgical robot system for anomaly detection is provided. The robotic arm is configured to hold and operate a surgical tool. The processor is configured to detect anomalies by applying a machine learning model. The machine learning model is configured to receive past operation information and output a predicted operation.

[0009] The present invention is defined by the following claims, and nothing in this section should be taken as limiting those claims.Other aspects and advantages of the present invention are discussed below in conjunction with the preferred embodiments, and may be subsequently claimed independently or in combination. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Embodiments of the present invention are shown in the figures of the accompanying drawings by way of example and not by way of limitation, wherein like reference numerals indicate like elements. It should be noted that references to "one" or "an" embodiment of the present invention in this disclosure are not necessarily the same embodiment, and that they refer to at least one. In addition, for the sake of brevity and to reduce the total number of drawings, a given drawing may be used to illustrate features of more than one embodiment of the present invention, and not all elements in the drawings may be required for a given embodiment.

[0011] Figure 1 is a diagram illustrating an example operating room environment with a surgical robotic system in accordance with aspects of the subject technology;

[0012] Figure 2 is a schematic diagram illustrating one exemplary design of a robotic arm, a tool drive, and a cannula loaded with a robotic surgical tool in accordance with aspects of the subject technology;

[0013] Figure 3A and Figure 3B are schematic diagrams illustrating exemplary tool driving devices with and without adjacent loaded tools, respectively, in accordance with aspects of the subject technology;

[0014] Figure 4A and Figure 4Bis an end effector illustrating an exemplary gripper according to aspects of the subject technology, the end effector having a robotic wrist, a pair of opposing jaws, and a pulley and cable system for coupling the robotic wrist and the pair of jaws to an actuator of a tool drive;

[0015] Figure 5 is a block diagram of an embodiment of a surgical robotic system for anomaly detection based on a machine learning model;

[0016] Figure 6 is a block diagram of another embodiment of a surgical robotic system using a machine learning model to detect anomalies;

[0017] Figure 7 An example long short-term memory (LSTM) cell for a machine learning model is illustrated;

[0018] Figure 8 Illustrated an example machine learning model for anomaly detection using LSTM;

[0019] Fig. 9 is a flow chart and corresponding logic of a method for machine learning network-based detection of anomalies when driving a surgical instrument, according to one embodiment;

[0020] Fig.10 An embodiment of a cart-based robotic system arranged for diagnostic and / or therapeutic use (e.g., bronchoscopy) is illustrated; and

[0021] Fig.11 Embodiments of a table-based robotic system configured for a procedure (eg, laparoscopic surgery) are illustrated. DETAILED DESCRIPTION

[0022] A machine learning model is used to predict unexpected drive train operations of surgical instruments. For example, unexpected cable forces in the cable of a cable-driven tool are detected during teleoperation. An anomaly detection framework is used to predict cable forces based on past trends, and the prediction notifies the user of an impending failure. As another example, the prediction is used as a proxy for expected values ​​given an encoder and / or load sensor. The model is machine trained to learn the dynamics between (a) changes in encoder position and / or load and (b) the evolution of torque distribution in the articulation cable.

[0023] Based on this prediction, corrective action can be taken before a failure occurs. Corrective action may include driving the surgical instrument less aggressively to complete the current task, or replacing and / or repositioning the surgical instrument once an alert is raised.

[0024] Compared with other model-based techniques, the proposed framework is more robust in detecting anomalies. With machine training, the training data is obtained from the actual operation of the instrument and non-modeled effects such as posture-related friction, cable stretch, and hysteresis can be accounted for. In addition, including data from multiple tools enables the machine learning model to generalize across multiple tools.

[0025] The following discussion first introduces an example robotic surgical system (see Figures 1 to 4B ). Figures 5 to 8 Embodiments of anomaly detection using an example or another robotic surgical system are shown. Fig. 9 A method for anomaly detection is shown. Figures 5 to 9 Embodiments of the invention use application of machine learning models for faster and / or accurate anomaly detection.

[0026] Figures 1 to 4B One example of a cable-driven robotic surgical instrument and a robotic surgical system for using the instrument is illustrated. Fig.10 and Fig.11 Other examples of robotic surgical systems are illustrated. Any of these or other examples are used Figure 5 and / or Figure 6 in anomaly detection systems and / or for Fig. 9 in the anomaly detection method. Figure 5 , Figure 6 and Fig. 9 The description can be used Figures 1 to 4B but could alternatively be used Fig.10 or Fig.11 Other cable-driven surgical instruments, robotic arms, and / or robotic surgical systems may be used.

[0027] Generally, an end effector including a robotic wrist and / or one or more jaws may be coupled to an actuator via a metal cable or wire. The wires may, for example, work in pairs, where pulling one wire exerts an opposing force on the other wire of the pair, so the robotic wrist may be an opposing robotic wrist. Jaws are used in this example, but other applied surgical robotic instruments may also be used. Applicable surgical robotic instruments include graspers, forceps, scissors, needle drivers, retractors, forceps, and cautery instruments, among others.

[0028] Figure 1 is a diagram illustrating an example operating room environment having a surgical robotic system 100 in accordance with aspects of the subject technology. Figure 1As shown, the surgical robotic system 100 has a surgeon's console 120, a control tower 130, and one or more surgical robotic arms 112 located at a surgical robotic platform 110 (e.g., a table or bed, etc.), wherein surgical tools with end effectors are attached to the distal ends of the robotic arms 112 for performing surgical procedures. The robotic arms 112 are shown as a table-mounted system, but in other configurations, the robotic arms may be mounted in a cart, ceiling or sidewall, or other suitable support surface.

[0029] Generally, a user (such as a surgeon or other operator) may remotely manipulate the robotic arm 112 and / or surgical instruments (e.g., teleoperation) using the user console 120. The user console 120 may be located in the same operating room as the robotic system 100, such as in FIG. Figure 1 As shown in the exemplary user console 120, a surgeon located in the seat 122 and viewing the user display 128 can manipulate the foot control 124 and / or the handheld user interface device 126 to remotely control the robotic arm 112 and / or the surgical instruments mounted to the distal end of the arm.

[0030] In some slightly different scenarios, the user may also operate the surgical robotic system 100 in an "over the bed" (OTB) mode, in which the user is at the side of the patient and manipulates both the robotically driven tools / end effectors attached to the patient (e.g., holding the handheld user interface device 126 with one hand) and manual laparoscopic tools. For example, the user's left hand may manipulate the handheld user interface device 126 to control the robotic surgical components, while the user's right hand may manipulate the manual laparoscopic tools. Thus, in these different scenarios, the user may perform both robotically assisted MIS and manually perform laparoscopic surgery on the patient.

[0031] During the exemplary procedure or surgical operation, the patient is prepared and covered in a sterile manner to achieve anesthesia. The initial approach to the surgical site can be performed manually using the robotic system 100 in a stowed configuration or a retracted configuration to help approach the surgical site. Once the approach is completed, the initial positioning and / or preparation of the robotic system can be performed. During the procedure, the surgeon in the user console 120 can manipulate various end effectors and / or imaging systems to perform surgical operations using foot controls 124 and / or user interface devices 126. Manual assistance can also be provided at the operating table by personnel wearing sterile work clothes, and the tasks that the personnel can perform include but are not limited to pulling tissue, or performing manual repositioning or tool replacement involving one or more robotic arms 112. Non-sterile personnel may also be present to assist the surgeon at the user console 120. When the procedure or surgical operation is completed, the robotic system 100 and / or the user console 120 may be configured or set to a state that facilitates one or more post-operative procedures, including but not limited to cleaning and / or disinfection of the robotic system 100, and / or medical record input or printout, whether electronic or paper, such as via the user console 120.

[0032] In some aspects, the communication between the robot platform 110 and the user console 120 can be through the control tower 130, which can convert user commands from the user console 120 into robot control commands and send them to the robot platform 110. The control tower 130 can also send status and feedback from the robot platform 110 back to the user console 120. The connection between the robot platform 110, the user console 120 and the control tower 130 can be a wired connection and / or a wireless connection, and can be proprietary and / or implemented using any of a variety of data communication protocols. Any wired connection can optionally be built into the floor and / or wall or ceiling of the operating room. The surgical robot system 100 can provide video output to one or more displays, including displays in the operating room and remote displays accessed via the Internet or other networks. The video output or feed can also be encrypted to ensure privacy, and all or part of the video output can be saved to a server or electronic health record system.

[0033] Figure 2 1 is a schematic diagram illustrating one exemplary design of a robotic arm 112, a tool drive, and a cannula loaded with a robotic surgical tool according to aspects of the subject technology. The surgical robotic arm 112 is configured by design, shape, circuitry, and / or programming to hold and operate a surgical tool 220. Figure 2As shown, the example surgical robot arm 112 may include a plurality of links (e.g., links 204) and a plurality of actuated joint modules (e.g., joints 202) for actuating the plurality of links relative to each other. The joint modules may include various joint types, such as pitch joints or roll joints, which may substantially constrain the movement of adjacent links about certain axes relative to other axes. Figure 2 Also shown in the exemplary design of FIG. 1 is a tool drive 210 attached to the distal end of the robotic arm 112. The tool drive 210 may include a cannula 214 coupled to its end to receive and guide a surgical instrument 220 (e.g., an endoscope, a stapler, a gripper, etc.). The surgical instrument (or "tool") 220 includes an end effector 222 located at the distal end of the tool. The multiple joint modules of the robotic arm 112 may be actuated to position and orient the tool drive 210, which actuates the end effector 222 to perform robotic surgery.

[0034] Figure 3A and Figure 3B 2 is a schematic diagram illustrating an exemplary tool driving device with and without an adjacent loaded tool, respectively, according to aspects of the subject technology. Figure 3A and Figure 3B As shown, in one variation, the tool drive 210 may include an elongated base (or "tray") 310 having a longitudinal track 312 and a tool bracket 320 that is slidably engaged with the longitudinal track 312. The tray 310 may be configured to be coupled to a distal end of a robotic arm such that articulation of the robotic arm positions and / or orients the tool drive 210 in an appropriate position. Additionally, the tool bracket 320 may be configured to receive a tool base 352 of a tool 220, which may also include a tool shaft 354 extending from the tool base 352 and passing through the cannula 214, wherein the end effector 222 (not shown) is disposed at the distal end.

[0035] In addition, the tool bracket 320 may actuate a set of joint movements of the end effector, such as by a cable system or wire (the terms "cable" and "wire" are used interchangeably) manipulated and controlled by an actuation drive (the terms "actuator," "motor," and "drive" are used interchangeably). The tool bracket 320 may include different configurations with an actuation drive. For example, a rotary axis drive may include a motor having a hollow rotor and a planetary gear transmission at least partially disposed within the hollow rotor. Multiple rotary axis drives may be arranged in any suitable manner. For example, the tool bracket 320 may include six rotary drives 322A-322F, which are arranged in two rows, extending longitudinally along the base, and slightly staggered to reduce the width of the rack and increase the compact nature of the tool drive. As Figure 3BAs best shown, the rotational drive devices 322A, 322B, and 322C may be generally arranged in a first row, while the rotational drive devices 322D, 322E, and 322F may be generally arranged in a second row that is slightly longitudinally offset from the first row.

[0036] Figure 4A and Figure 4B is an end effector illustrating an exemplary gripper according to aspects of the subject technology, the end effector having a robotic wrist, a pair of opposing jaws, and a pulley and cable system for coupling the robotic wrist and the pair of jaws to an actuator of a tool drive. It is noted that although the following tool model is described with reference to an exemplary surgical robotic gripper, the proposed anomaly detection is applicable to any tool that includes an end effector coupled to a tool shaft, wherein a cable or other drive is used to control the end effector. Similar tools include, but are not limited to, grippers, clamps, forceps, needle drivers, retractors, and cautery instruments. Any number of wrist joints may be used, such as Figure 4A and Figure 4B One wrist joint is shown, but zero, two, or more wrist joints may be used. Any number of corresponding cables 405 for controlling the end effector may be used, such as one, two, three, four, or more. Figure 4A and Figure 4B In the example of FIG. 4 , four cables 405 are used to control the three degrees of freedom of the end effector.

[0037] like Figure 4A As shown, a pair of opposing jaws 401A and 401B are movably coupled to a first yoke 402 of a robot wrist via an extension shaft 412 along a first axis 410. The first yoke 402 may be movably coupled to a second yoke 403 of a robot wrist via a second extension shaft 422 along a second axis 420. The pair of jaws 401A and 401B may each be coupled or integrally formed with pulleys 415A and 415B, respectively, via the extension shaft 412, so that both jaws may rotate about the axis 410. Pulleys 425A, 425B, 425C, and 425D are coupled to the extension shaft 422 and rotate about the axis 420. The pulleys 425A, 425B, 425C, and 425D are arranged as a first set of pulleys 425B and 425C on one side of the yoke 402 and a second set of pulleys 425A and 425D on the other side of the yoke 402. Pulleys 425A and 425C are outer pulleys and pulleys 425B and 425D are inner pulleys. Similarly, a third set of pulleys 435A, 435B, 435C and 435D are coupled to a third extension shaft 432 and rotate about an axis 430 parallel to axis 420 .

[0038] The gripper 220 can be actuated to move one or both of the jaws 401A and 401B about the axis 410 in various ways. For example, the jaws 401A and 401B can be opened and closed relative to each other. The jaws 401A and 401B can also be actuated to rotate together as a pair to provide yaw motion of the gripper 220. In addition, the first yoke 402, the pulleys 415A and 415B, and the jaws 401A and 401B can rotate about the axis 420 to provide pitch motion of the gripper 220. These movements of the jaws of the robot wrist and / or tool can be achieved by controlling four independent cables 405A-405D. As shown in FIG. Figure 4A As shown, cable 405A may start (or terminate) from one side of pulley 415A and be routed along pulleys 425A and 435A, and cable 405B is configured to terminate at the other side of pulley 415A and be routed through pulleys 425B and 435B. Similarly, another pair of cables 405C and 405D may be coupled to jaw 401B. For example, cable 405C extends from one side of pulley 415B to pulleys 425C and 435C; and cable 405D is routed through pulleys 425D and 435D and terminates at the other side of pulley 415B. The third set of pulleys 435A, 435B, 435C, and 435D are arranged in a manner so as to keep cables 405A-405D attached to the second set of pulleys 425A-425D and prevent the cables from slipping or sliding relative to pulleys 425A-425D.

[0039] Controlling the movement of the gripper via four independent cables 405 has several advantages. One advantage of using four independent cables can be the reduction in the number of cables extending from the tool base 352 to the robot wrist compared to typical commercial designs using six cables 405 (or three cable loops with six cable ends). A smaller number of cables 405 can reduce tool size and the complexity of the wrist assembly, which can be beneficial for minimally invasive surgical procedures or non-surgical applications. In addition, instead of two or three cable loops, the arrangement of four independent cable loops not only allows independent control of the tension on each cable 405 without the need to pre-tension the cables 405, but also achieves variable compliance in the wrist joint and increased sensitivity to external loads. In addition, the tension on each cable can be readjusted independently, which can further increase tool performance.

[0040] like Figure 4A and Figure 4BAs shown, the gripper can be actuated to move the jaws 401A and 401B in various ways, such as gripping (e.g., the jaws rotate independently about axis 410) (three degrees of freedom), yaw (e.g., the jaws rotate together about axis 410), and pitch (e.g., the jaws rotate about axis 420) by imparting motion to one or more of the pulleys 415A, 415B, 425A, 425B, 425C, and 425D to thereby impart motion to the first yoke 402 and / or one or both of the jaws 401A and 401B. The cables 405A-405D can be grouped into two antagonistic pairs, i.e., when one cable in the antagonistic pair is actuated or tensioned and the other cable is released, the jaws will rotate in one direction. When only the other cable is tensioned, the jaws will rotate in the opposite direction.

[0041] For example, cables 405A and 405B are a first antagonistic pair for moving jaw 401A, and cables 405C and 405D are a second antagonistic pair for controlling jaw 401B. When cable 405A is tensioned (e.g., by rotating at least one of drive devices 322a-322f) and cable 405B is released, jaw 401A closes (moves toward opposing jaw 401B). On the other hand, when cable 405B is tensioned and cable 405A is released, jaw 401A opens (moves away from opposing jaw 401B). Similarly, when tensioned, cable 405C closes jaw 401B (moves toward opposing jaw 401A), and cable 405D opens jaw 401B (moves away from opposing jaw 401A), while the other cable is released. As another example, the clamping force between jaws 401A and 401B may be achieved by continuing to tension cables 405A and 405C (while loosening cables 405B and 405D) after the jaws are closed (contacting each other).

[0042] In the case where the two cables of the antagonistic pair are simultaneously tensioned and the two cables of the other pair are released, the pulley 415A or the pulley 415B does not rotate. Instead, the first yoke 402 together with the jaws 401A and 401B are imparted by the pulleys 415A and 415B to pitch about the axis 420. For example, when the pair of cables 405A and 405B are simultaneously tensioned and the pair of cables 405C and 405D are released, the jaws (together with the yoke 402) pitch out of the plane of the paper. However, when the two cables 405C and 405D are simultaneously tensioned and the pair 405A and 405B remain released, the jaws pitch into the plane of the paper.

[0043] Figure 4B2 is a schematic diagram illustrating exemplary angle definitions for various motions of the gripper 220 according to aspects of the subject technology. The angles are defined with reference to the axes 410 and 420 and the axis 452 of the first yoke 402 and the axis 453 of the second yoke 403. For example, Figure 4B As shown, the angle (θ1) between the axis 452 and the axis 453 may represent the rotation angle of the yoke 402 around the axis 420, which may also be defined as the pitch angle (θ 俯仰 )(And in Figure 4A , axis 452 of yoke 402 is superimposed on axis 453 of yoke 403 because the jaws remain in the reference position, i.e., there is no pitch motion). In addition, angles (θ2) and (θ3) may represent the angle between each of jaws 401A and 401B and axis 452 of yoke 402 (as the origin), respectively. In order to distinguish the sides of axis 452, angles (θ2) and (θ3) may take different signs. For example, Figure 4B As shown, angle (θ2) is negative and angle (θ3) is positive.

[0044] The angular position and clamping force of the distal end effector of the robotic surgical instrument are controlled. The control system may include a feedback loop involving position, torque and / or velocity feedback from the actuator and force feedback measured on the four wires to achieve the desired position and clamping force. Other sensors may be included. In one embodiment, each actuator and corresponding cable of the drive chain includes an encoder or other position sensor for determining the rotational position of the actuator and / or cable and a load sensor (e.g., a torque sensor) for determining the load or force on the actuator and / or cable.

[0045] In some implementations, the actuator controller can operate in a position plus feed-forward current mode. For example, the position controller can drive the distal end effector to a desired angular position in space based on position feedback, while the clamping force controller provides additional feed-forward current to achieve a desired clamping force between opposing members or jaws 401A-401B based on the clamping force measured by the load cells on the four wires.

[0046] Fig.10 Another embodiment of a surgical robotic system is shown, such as one arranged for use with a bronchoscope. This embodiment may include a cable drive train or other drive train from an actuator or driver 1028 to a tool or instrument 1013. The robotic-enabled medical system may be configured in a variety of ways, depending on the particular procedure.

[0047] Fig.10An embodiment of a cart-based, robotically enabled system 1000 arranged for diagnostic and / or therapeutic bronchoscopy is illustrated. During bronchoscopy, system 1000 may include a cart 1011 having one or more robotic arms 1012 to deliver medical instruments such as a steerable endoscope 1013 (which may be a procedure-specific bronchoscope for bronchoscopy) to a natural orifice entry point (i.e., the oral cavity of a patient positioned on a table in this example) to deliver the diagnostic and / or therapeutic tool. As shown, cart 1011 may be positioned near the patient's upper torso to provide access to the entry point. Similarly, robotic arms 1012 may be actuated to position the bronchoscope relative to the access point. When performing gastrointestinal (GI) procedures using a gastroscope (a specialized endoscope for GI procedures), the bronchoscope may also be utilized. Fig.10 The arrangement in .

[0048] Once the cart 1011 is correctly positioned, the robotic arm 1012 can insert the steerable endoscope 1013 into the patient's body robotically, manually, or in a combination thereof. As shown, the steerable endoscope 1013 may include at least two telescopic parts, such as an inner guide portion and an outer sheath portion, each of which is coupled to a separate instrument driver from a set of instrument drivers 1028, each of which is coupled to the distal end of a separate robotic arm 1012. This linear arrangement of instrument drivers 1028 that facilitates coaxial alignment of the guide portion with the sheath portion creates a "virtual track" 1029 that can be repositioned in space by manipulating one or more robotic arms 1012 to different angles and / or positions. The virtual track described herein is not any physical structure of the system, but rather an arrangement of other structures. Translation of the instrument driver 1028 along the virtual track 1029 causes the inner guide portion to telescope relative to the outer sheath portion, or to advance or retract the endoscope 1013 from the patient. The angle of the virtual track 1029 can be adjusted, translated, and pivoted based on clinical application or physician preference. For example, in bronchoscopy, the angle and position of the virtual track 1029 as shown represent a compromise between providing the physician with access to the endoscope 1013 while minimizing friction caused by the bending of the endoscope 1013 into the patient's mouth.

[0049] After insertion, the endoscope 1013 can be guided down the patient's trachea and lungs using precise commands from the robotic system until the target destination or surgical site is reached. To enhance navigation through the patient's pulmonary network and / or to reach the desired target, the endoscope 1013 can be manipulated to telescopically extend the inner guide portion from the outer sheath portion to obtain enhanced articulation and a larger bending radius. The use of separate instrument drivers 1028 also allows the guide portion and the sheath portion to be driven independently of each other.

[0050] The system 1000 may also include a mobile tower 1030 that may be connected to the cart 1011 via a support cable to provide control, electronic, fluid, optical, sensor and / or power support to the cart 1011. Placing such functionality in the tower 1030 allows for a smaller form factor cart 1011 that may be more easily adjusted and / or repositioned by the operating physician and his / her staff. Additionally, dividing functionality between the cart / table and the support tower 1030 reduces operating room clutter and facilitates improved clinical workflow. While the cart 11 may be positioned close to the patient, the tower 1030 may be stowed in a remote location to be out of the way during a procedure.

[0051] To support the robotic system described above, the tower 1030 may include components of a computer-based control system that stores computer program instructions, for example, in a non-transitory computer-readable storage medium such as a permanent magnetic storage drive, a solid-state drive, or the like. Whether the execution occurs in the tower 1030 or in the cart 1011, the execution of these instructions may control the entire system or a subsystem thereof. For example, when executed by a processor of a computer system, the instructions may cause the components of the robotic system to actuate the associated brackets and arm mounts, actuate the robotic arm, and control the medical device. For example, in response to receiving a control signal, a motor in a joint of the robotic arm may position the arm into a particular posture.

[0052] The tower 1030 may also include pumps, flow meters, valve controllers, and / or fluid pathways to provide controlled irrigation and aspiration capabilities to systems that may be deployed through the endoscope 1013. The tower 1030 may include voltage and surge protectors designed to provide filtered and protected power to the cart 11, thereby avoiding the need to place power transformers and other auxiliary power components in the cart 1011, resulting in a smaller, more mobile cart 1011. The tower 1030 may also include support equipment for sensors deployed throughout the robotic system 1000. Similarly, the tower 1030 may also include electronic subsystems for receiving and processing signals received from deployed electromagnetic (EM) sensors. The tower 1030 may also be used to house and position EM field generators for detection by EM sensors in or on medical devices.

[0053] In addition to other consoles available in the rest of the system (e.g., consoles mounted on top of carts), tower 1030 may also include console 1031. Console 1031 may include a user interface and display screen, such as a touch screen, for a physician operator. The console in system 1000 is typically designed to provide both preoperative information and real-time information for robotic control and procedures, such as navigation and positioning information for endoscope 13. When console 1031 is not the only console available to the physician, it may be used by a second operator (such as a nurse) to monitor the health status or vital signs of the patient and the operation of system 1000, as well as to provide procedure-specific data, such as navigation and positioning information. In other embodiments, console 1031 is housed in a body separate from tower 1030.

[0054] Embodiments of the robotic-enabled medical system may also incorporate a patient table. Incorporating a table reduces the amount of capital equipment within the operating room by removing a cart, which allows for greater access to the patient. Fig.11 1136 includes a support structure or column 1137 for supporting a platform 1138 (shown as a "table" or "bed") on a floor. Much like a cart-based system, the end effector of a robotic arm 1139 of system 1136 includes an instrument driver designed to manipulate an elongated medical instrument, such as a bronchoscope. In practice, a C-arm for providing fluoroscopic imaging can be positioned over the upper abdominal region of a patient by placing emitters and detectors around table 38.

[0055] The column 1137 may include one or more brackets 1143, shown as annular in the system 1136, on which one or more robotic arms 1139 may be based. The brackets 1143 may translate along a vertical column interface extending along the length of the column 1137 to provide different vantage points from which the robotic arms 1139 may be positioned to reach the patient. The brackets 1143 may rotate around the column 1137 using a mechanical motor positioned within the column 1137 to allow the robotic arms 1139 to access multiple sides of the table 1138, such as both sides of the patient. In an embodiment with multiple brackets 1143, the brackets 1143 may be individually positioned on the column 1137 and may translate and / or rotate independently of the other brackets. Although the brackets 1143 do not need to be around the column 1137 or even circular, the annular shape as shown facilitates the rotation of the brackets 1143 around the column 1137 while maintaining structural balance. Rotation and translation of the bracket 1143 allows the system 1136 to align medical instruments (such as endoscopes and laparoscopes) into different entry points on the patient. In other embodiments (not shown), the system 1136 may include a patient table or bed with an adjustable arm support in the form of a rod or track extending beside the patient table or bed. One or more robotic arms 1139 (e.g., via a shoulder with an elbow joint) may be attached to the adjustable arm support, which can be adjusted vertically. By providing vertical adjustment, the robotic arm 1139 is advantageously able to be compactly received under the patient table or bed and then raised during the procedure.

[0056] The robotic arm 1139 may be mounted on the carriage 1143 via a set of arm mounts 1145 including a series of joints that may be individually rotated and / or telescopically extended to provide additional configurability to the robotic arm 1139. Additionally, the arm mounts 1145 may be positioned on the carriage 1143 such that when the carriage 1143 is appropriately rotated, the arm mounts 1145 may be positioned on the same side of the table 1138, on the opposite side of the table 1138 (e.g., Fig.11 ) or on an adjacent side of stage 1138 (not shown).

[0057] Column 1137 provides structural support for stage 1138 and provides a path for vertical translation of carriage 1143. Internally, column 1137 may be equipped with a lead screw for guiding the vertical translation of the carriage, and a motor for mechanizing the lead screw-based translation of carriage 1143. Column 1137 may also transmit power and control signals to carriage 1143 and robotic arm 1139 mounted thereon.

[0058] In one embodiment, Fig.11A robotic surgical system is used to perform laparoscopy. In a laparoscopy procedure, minimally invasive instruments may be inserted into the patient's anatomical structure through one or more small incisions in the patient's abdominal wall. In some embodiments, the minimally invasive instrument includes an elongated rigid member, such as a shaft, for accessing the anatomical structure within the patient. After the patient's abdominal cavity is inflated, the instrument may be guided to perform surgical or medical tasks, such as grasping, resection, ablation, suturing, etc. In some embodiments, the instrument may include a scope, such as a laparoscope for viewing tasks (not necessarily surgical in the sense of operating directly on tissue). Fig.11 Embodiments of a robotically enabled table-based system configured for laparoscopic procedures are illustrated. Fig.11 As shown, the carriage 1143 of the system 1136 can be rotated and adjusted vertically to position the pair of robotic arms 1139 on opposite sides of the table 1138 so that instruments 1159 can be positioned through minimal incisions on both sides of the patient to reach his / her abdominal cavity using arm mounts 1145. Laparoscopy can then be performed using one or more actuators with a laparoscope as a tool or instrument.

[0059] Figure 5 An example surgical robotic system is shown that includes a processor 540 that interacts with a drive chain 500. For example, the drive chain 500 is FIG. 3A to FIG. 4B Other drive chains using cables, gears, actuators, motors, pulleys, guides, and / or other force transfer devices may be used. The drive chain may be part of a surgical system for manual manipulation or guidance rather than a teleoperated robotic system, such as a cable-driven laparoscopic system. Figure 5 In the example of FIG. 5 , one or more cables 405 connect the actuator 322 to the tool 220 for driving the chain 500 .

[0060] The surgical tool 220 is a gripper with two jaws, but may be another type of surgical instrument. The surgical tool 220 is connected by a plurality of cables 405 to a corresponding number of actuators 322, such as four cables 405 connected to four engaged actuators 322. The cables 405 allow actuation of the actuators 322 to move the surgical tool 220, such as to open and close the jaws and / or to rotate in pitch and / or yaw.

[0061] Sensors 510, 520 such as force sensors (e.g., load cells) and / or position sensors (e.g., encoders for absolute position) are used for control. The processor 540 using instructions in a non-transitory computer readable storage medium uses information from the sensors 510, 520 to control the operation of the actuator 322.

[0062] The encoder 520 senses the position of the actuator. The encoder 520 can sense angular position with or without multiple rotations (e.g., 0 degrees-360 degrees or 0 degrees-N degrees, where N is greater than 360 degrees). Alternatively, linear position can be sensed. In one embodiment, the encoder 520 is an absolute position sensor. Other position sensors and / or encoders can be used.

[0063] The load sensor 510 is a strain gauge or other sensor for sensing a load or force. The force may be longitudinal and / or rotational. The force applied to the cable 405 and / or tool 220 by the actuator 322 is sensed. In one embodiment, the load sensor 510 is positioned on the arm of the actuator 322. In another embodiment, the load sensor 510 is positioned on the cable 405. In yet another embodiment, the load sensor 510 is a current sensor for sensing the current drawn by the actuator 322 to apply force to the cable 405.

[0064] Other drive trains with the same or different sensors may be used. For example, only an encoder or another position sensor may be provided. As another example, only a load sensor, other torque sensors, and / or other force sensors may be provided. In yet other embodiments, a speed sensor may be provided in place of or in addition to the load sensor 510 and / or encoder 520.

[0065] Processor 540 is a general purpose processor, an application specific integrated circuit, a field programmable gate array, a digital signal processor, a controller, an artificial intelligence processor, a tensor processor, a graphics processing unit, a digital circuit, an analog circuit, a combination thereof, and / or other now known or later developed processor for robot control. Processor 540 is configured by software, hardware, and / or firmware to detect anomalies during teleoperation.

[0066] The processor 540 is configured to detect anomalies in the drive train 500 by applying a machine learning model. The machine learning model is configured to receive past and / or current operation information and output a predicted operation. The processor 540 is configured to detect anomalies as an output of a machine learning model (e.g., a model trained to output a classification as normal or abnormal) and / or is configured to detect anomalies by comparing the output of the model with the actual measured operation (e.g., the model predicts normal, and anomalies are detected when the measured operation is a threshold difference from the predicted operation).

[0067] Various input sources may be used for the model, such as position, load (torque), and / or velocity. Inputs may be used for one drive chain 500 (e.g., a single cable), a subset of drive chains 500 for a tool 220, or all drive chains 500 for a tool 220. Other inputs may include operating parameters, such as the type of tool 220, the surgical procedure being performed, the time relative to the surgical procedure, and / or the stage of the operation (e.g., suturing). Any combination of inputs may be used.

[0068] Various outputs may be generated by the machine learning model. For example, predicting cable forces, such as load or torque applied to or by the cable 405. As another example, outputting tool position, tool 220 speed, and / or torque. In yet another example, the output is a classification of abnormal or non-abnormal, the type of abnormality, and / or the magnitude of the abnormality. The level of uncertainty or certainty in the output of the machine learning model may be output.

[0069] exist Figure 6 In an example embodiment, a machine learning model 610 stored in memory 600 is configured through training to receive a position 620 and / or a load 622 to output a cable force (load) 624. The machine learning model 610 outputs a predicted cable force in response to receiving or inputting the position 620 and the load 622. This example is used herein, but other inputs and / or outputs may be used based on training data previously used to train the machine learning model 610.

[0070] Any number of past inputs may be used to generate the current output. For example, a sequence of M positions 620 and loads 622 is input to predict the current or next cable force 624. M is an integer of 1 or greater (e.g., M=4, 8, 16, 32, 50, or 64). The sequence is defined by a moving window of M inputs, where the window slides over time to select the M most recent inputs. Any number of current and future outputs may be predicted in response to a given sequence of inputs. For example, N outputs are generated, where N is an integer of 1 or greater. In the case of N=1, the output may be the output of the current or next time increment (e.g., cable force 624). N=2 or greater provides current or next plus even further into the future predictions.

[0071] The current output may be an expected output, such as an expected normal cable force 624. The machine learning model 610 is trained to output a normal output (e.g., cable force 624). In one embodiment, the processor 540 is configured to detect anomalies where the normal operation predicted by the machine learning model deviates from the actual operation of the surgical tool 220 by the robotic arm 112. The processor 540 is configured to detect anomalies by comparing the predicted cable force 624 with the actual cable force sensed by the load sensor 510 at the current or next time. An anomaly or a classification of normal or not may alternatively be output directly by the model 610.

[0072] Machine learning model 610 is previously trained using machine learning. Machine learning is an offline training phase where the goal is to identify an optimal set of values ​​for learnable parameters of model 610 that can be applied to many different inputs (i.e., actuator positions and / or load sequences over time). These machine learning parameters can then be used during operation (test or application phase) to quickly predict the operation of drive train 500 (e.g., cable forces). Once learned, machine learning model 610 is used in an online processing phase where a measurement sequence is input and the operation of drive train 500 is output based on the model values ​​learned during the training phase.

[0073] Various types of machine learning models can be used, such as support vector machines, neural networks, Bayesian or another type of machine learning model. In one embodiment, the model is a neural network, such as a fully connected neural network (e.g., a convolutional neural network). Any architecture or layer structure for machine learning can be used. The architecture defines the structure, learnable parameters, and the relationship between the parameters. In one embodiment, the architecture of the model to be trained and the resulting training model 610 is a recursive neural network (RNN) 612. Any of a variety of recursive neural networks can be used, such as based on one or more long short-term memories (LSTM) 614. The recursive neural network 612 can alternatively be a transformer network.

[0074] Figure 7 An example LSTM 614 is shown. LSTM is an artificial RNN architecture used in the field of deep learning. LSTM defines various weights (e.g., σ) that can be learned. Input x passes through neural network layers (e.g., σ and tanh), using operators x and + to generate a cell state h, where C is an input or output term from other LSTM 614. T is a time index. Other LSTM structures can be used. Transformers can be used instead of LSTM.

[0075] Figure 8An example RNN 612 is shown. The RNN 612 models time series data. In this example, the dynamics of the drive chain operation are modeled using telemetry data. The sequence of LSTM 614 is arranged as forward and backward chains or layers. The input sequence x is input to the forward chain and backward chain or layer. The output y is generated by the activation layer fed from the LSTM 614. Other RNNs with different arrangements can be used. Other RNNs without LSTM 614 or transformers can be used. Any number of hidden layers can be provided between the input layer and the output layer.

[0076] The model 610 (such as a recurrent neural network 612) is machine trained. Deep machine training is performed. Deep learning is used to train the model 610. Other machine learning may be used. Many (e.g., hundreds or thousands) samples of the inputs to the model and the true outputs are collected as training data. For example, data from testing (e.g., bench or manufacturer testing) is collected as training data. The inputs are measured over time, and the resulting outputs are measured over time during testing. Many sample sequences of a given type of surgical instrument 220 and / or drive chain 500 may be collected during failure or other testing of the drive chain 500 and / or surgical instrument 220.

[0077] Machine training learns both the features of the input data and the transformation of these features to the desired output. Back propagation, RMSprop, ADAM, or another optimization is used to learn the values ​​of the learnable parameters of the model 610. In the case of supervised training, the difference (e.g., L1, L2, or mean square error) between the estimated output and the true output is minimized.

[0078] Once trained, the model 610 is applied during teleoperation. For example, the machine learning model 610 is used to predict normal operation. The training data used for training represents normal operation, so that the prediction is the output that will occur during operation without defects or interference given the input. In an alternative embodiment, the machine learning model 610 is trained to predict normal and abnormal operation. The model 610 is trained to indicate the current operation category (e.g., normal or abnormal). In other embodiments, the model is trained to identify when abnormal operation is likely to occur.

[0079] The machine learning model 610 is previously trained and then used after training. Fixed values ​​of the learning parameters are used for application. The learned values ​​and network architecture determine the output from the input. During application in teleoperation, the same learning weights or values ​​are used. The model and values ​​of the learnable parameters do not change from one patient to the next at least within a given time (e.g., weeks, months, or years) or a given number of operations (e.g., tens or hundreds). These fixed values ​​and corresponding fixed models are applied sequentially and / or by different processors to input sequences for different patients. As part of application to a given patient, the model can be updated (such as retrained) or replaced but without learning new values.

[0080] Retraining may be used. For example, training data from bench testing (i.e., non-patient data) is used to train a machine learning model 610 (e.g., RNN 612). The same surgical instrument and / or type of surgical instrument and robotic arm is then used in teleoperation with a patient. Data from teleoperation associated with the patient is collected. This second set of data can then be used for retraining. The model 610 starts with previously learned values ​​for the parameters of the architecture, and then optimization is performed again using the second set of data or both the original data and the second set of data. The result is a retrained model 610 ready for application. Due to the retraining, one or more values ​​of the learnable parameters may be the same or different.

[0081] In another example for retraining, the original training data is based on the type of surgical instrument. The model 610 is trained using data from that type of surgical instrument. The resulting trained model 610 can later be retrained using training data from a different type of surgical instrument. The retraining can use both sets of data or only use training for the other type of surgical instrument. By using data from more than one instrument in the original training or retraining, one model 610 can be trained to predict different types of instruments.

[0082] Fig. 9 One embodiment of a method for anomaly detection in an instrument 220 of a surgical robotic system is shown. A trained machine learning model 610 is used to quickly predict the operation of a drive train 500 of a surgical instrument 220. The prediction provides or is used to provide detection of anomalies.

[0083] This method is Figure 1 , Figure 2 or Figure 3A and Figure 3B In one embodiment, Figure 5 and / or Figure 6 The method is performed by a surgical robot system. Figure 4A and Figure 4BThe method is performed using a surgical instrument or another instrument. Figure 4A and Figure 4B In the example of FIG. 4 , four cables 405 control three degrees of freedom for the cable-driven instrument 220. Other numbers of cables 405 having an equal or fewer number of degrees of freedom than the instrument 220 may be used. A programmed processor 540 (also referred to herein as processing logic) of a control unit or other controller performs the method during teleoperation. The non-transitory memory 600 may store instructions for programming the processor 540 to detect anomalies.

[0084] The actions are performed in the order shown or in a different order. For example, actions 900 and 910 are performed simultaneously and / or repeatedly. Additional, different, or fewer actions may be provided. For example, action 930 is not provided. As another example, the processor 540 detects the type of surgical instrument 220 connected to the robotic arm 112 and selects the machine learning model 610 for anomaly detection based on the type. In yet another example, actions are provided for connecting the surgical instrument 220 to the robotic arm 112, homing, and / or controlling teleoperation.

[0085] Once the tool drive 210 is connected to the tool instrument 220 and any homing, calibration and / or staging is completed, the robotic arm 122, drive chain 500 and surgical instrument 220 are used for teleoperation while at least a portion of the surgical instrument 220 is within the patient. The motor (e.g., actuator 322) of the tool drive 210 rotates. The rotation of the motor is performed under position control, but other control modes may be used. The surgeon controls teleoperation. Input commands from the surgeon are converted into position (or other) control signals for the actuator 322.

[0086] In act 900, the processor receives measurements from one or more sensors 510, 520. The sensors 510, 520 measure operation of the drive chain 500 during teleoperation of an instrument 220 mounted on the surgical robotic arm 112.

[0087] The surgical robotic system operates the cable-driven instrument 220 in a surgical procedure. For example, the motion of the user interface device 126 and / or the foot pedal 124 is converted into joint commands, such as using inverse kinematics. The actuator 322 is moved and controlled. The cable-driven instrument 220 moves in a surgical procedure. Alternatively or additionally, the jaws open or close.

[0088] Periodically (eg, every tenth of a second), one or more sensors 510, 520 sense the operation of the drive train 500. Different sensors 510, 520 may sense at different frequencies.

[0089] In one embodiment, motor or actuator position and load torque are sensed over time. Sensing is performed at the actuator 322, along the cable 405, and / or at other parts of the drive chain 500. For example, the encoder 520 senses the position of the actuator 322, and the load sensor 510 senses the load or torque on the actuator 322 and / or the cable 405. Various inputs can be used, such as the position commanded by the user, the load generated by teleoperation, the speed, or any combination thereof. Data (measurements) from a torque sensor (e.g., load sensor 510) can be used alone to detect anomalies, but may be sensitive to thresholds and lead to false alarms.

[0090] In action 910, the processor 540 predicts the operation of the drive chain 500 during teleoperation. The prediction is performed at least in part by the machine learning model 610. The machine learning model 610 predicts the operation quickly (i.e., with less processing than an algorithm) in response to measured inputs. The machine learning model 610 is previously trained on the same type of surgical instrument 220, so measurements from the operation of a given surgical instrument 220 can be used to accurately predict the operation of that instrument 220. Each type of tool 220 is consistent. For example, for a given type of instrument 220, the friction characteristics remain relatively the same.

[0091] In one embodiment, the prediction is normal operation. Given an input, the machine learning model 610 predicts the operation that will occur without interference or damage along the drive chain 500 (e.g., damage to the actuator 322, the cable 405, and / or the instrument 220). For example, a series of forward positions and loads for a given drive chain 500 or for all drive chains 500 connected to the same instrument 220 are input to the model 610, which outputs a load, torque, or another cable force for the next time increment. The predicted cable force is a prediction as if the operation is normal. The dynamics between the input position and / or load and the output cable force are modeled by training, thereby allowing the training model to make predictions.

[0092] The machine learning model 610 is an RNN 612, but other neural networks or models may be used. For example, the model 610 is an RNN 612 using one or more LSTMs 614 and / or transformers.

[0093] Model 610 is trained based on simulation, bench testing or both. Training data is collected from testing or simulation, rather than use in patients. Alternatively, training data is collected from use in patients. In one embodiment, initial model 610 is trained based on simulation and / or bench testing data and used as a trained model. This data can be easily obtained from the initial testing of the tool during the manufacturing process. A large amount of data collected during testing is used to build a surrogate model dedicated to each tool type. The data or examples collected during use in patients can then be used to retrain model 610. Training models 610 are deployed for each tool type, and as more data is available, these models 610 can be further refined over time. Similarly, model 610 may have been trained for one type of instrument 220. Data collected from testing and / or using different types of instruments 220 in patients (e.g., the original is clamps and other types are scalpels) are then used to retrain model 610 for use with the other types of instruments 220.

[0094] In act 900, measurements are continuously received. A moving window of M most recent measurements defines the measurements for input to the model. For each time increment, the model 600 predicts one or more values ​​of the operation at the future and / or current time. Acts 900 and 910 are repeated as the time window moves to define the measurements in a first-in, first-out operation.

[0095] In action 920, the processor 540 detects an anomaly. The anomaly may be detected based on that an anomaly in the model 610 output measurement has occurred. In one embodiment, the output is used to detect the anomaly. The model 610 outputs normal operation, such as an expected cable force. The expected cable force for normal or unimpeded and undamaged operation is compared to the actual cable force measured for the predicted time increment. The predicted operation output by the machine learning model 610 is compared to the actual operation of the drive chain 500 to indicate whether an anomaly has occurred. In the case where the expected and actual are within a threshold difference of each other, then the actual operation is normal. In the case where the actual deviates from the expected threshold amount, then an anomaly has occurred.

[0096] Since the machine learning model 610 is trained using training data for normal operation and / or detecting anomalies regardless of the source of the anomaly, the anomaly detection operation occurs regardless of which anomaly source occurs. Deviation from normal may be due to one or more of a variety of problems. Anomalies are detected without relying on detecting the anomaly as being due to a specific one of a variety of problems. The comparison between the predicted output and the actual output can be used to evaluate unexpected events, such as cable breakage or blockage, higher than expected friction in the tool 220 and / or unexpected external loads at the wrist. Regardless of the source, anomalies are detected. In other embodiments, the model 610 is trained to indicate the type of anomaly, and / or the processor 540 determines the type of anomaly.

[0097] Anomalies can be detected based on predictions from a given time. Alternatively, the deviations of the actual and predicted over a time window (period) are combined (e.g., summed or averaged) or used together. Anomalies are detected if each comparison exceeds a threshold difference within the period or if the combination exceeds a threshold. For example, the cable force is predicted for the next "n" time steps for each time step. If the cumulative error between the predicted cable force and the actual cable force within a specified time window is greater than a predetermined threshold, an alert is issued to the user.

[0098] In action 930, the processor 540 outputs an indication of the detection of the abnormality. The processor 540 outputs to a memory, a display, a robot system controller, or a report. The processor 540 generates the indication as text, a warning light, a sign, a graphic, or a remedial instruction. For example, the user interface displays an alert of the abnormality detection so that the user can plan the operation to be blocked. As another example, due to the detection of the abnormality, the controller causes the robot arm 112 and / or the surgical instrument 220 to operate differently (e.g., more slowly or with less force). In another example, an instruction to test, fix, or replace the drive chain 500 and / or the instrument 220 is issued.

[0099] In an exemplary embodiment, training and using Figure 8 RNN 612. The input training data is normalized and scaled, such as normalizing and scaling torque telemetry and encoder position. The model is trained using the following hyperparameters: time step: 50, learning rate: 1e-5, optimizer: ADAM, loss function: mean squared error, batch size: 32, training epochs: 500 (early stopping, tolerance epochs = 30), network architecture: 2-layer LSTM with ELU activation and 4 units (see Figure 8), Training-validation split: 0.8 / 0.2, Input: [{enc(t-50),tau(t-50)},{enc(t-49),tau(t-49)}.....{enc(t),0}], Output: [tau(t)] (where enc is the encoder and tau is the torque or load), Test cases: Data collected on different tools for 4 scenarios: normal operation, pitch failure, yaw failure, and jaw failure. For runs collected on different tools at the same time, training data for random motion in the joint motion degrees of freedom on the entire system was collected. Data was collected at 200Hz, resulting in 68,331 data points. For preprocessing: the torque after homing was subtracted from the data, the torque and encoder position were normalized, and the normalized data was used for training. The final output is the network output of model 610 plus the initial torque distribution. Other training values ​​and / or arrangements may be used.

[0100] In the test, the results of normal operation have no anomalies detected. For pitch faults, anomalies are detected. Cables 1, 2 and 5, 6 are detected as having wrong torque distribution within a defined time interval. For yaw faults, anomalies are detected. Cables 1, 6 and 2, 5 are detected as having wrong torque distribution within a defined time interval. For jaw faults, anomalies are detected in the area where the cable torque changes sharply. Cables 1, 5 and 2, 6 are detected as having wrong torque distribution within a defined time interval. For tools with different friction curves, the network generalizes the friction changes between the tools, so no anomalies are detected. With external loads, anomalies are detected. The number of detections is a function of the threshold. Thresholding is performed on the error between the predicted output and the actual output, not on the actual output itself.

[0101] In other embodiments, different types of inputs are used. Images are used instead of position, torque, and / or velocity as inputs. For example, a camera (e.g., an endoscope) provides images as feedback to estimate the position of the tool tip relative to commands issued by an actuator and / or a user. Model 610 is trained to indicate position and / or generate images showing position (e.g., using a generative neural network). Actual is compared to prediction to identify any anomalies. Errors or differences between the estimate and actual can be used to assess whether the tool is responding as expected.

[0102] The above description of the exemplary embodiments of the present invention, including the content described below in the abstract of the specification, is not intended to be exhaustive or to limit the present invention to the specific form disclosed. Although specific embodiments and examples of the present invention are described herein for illustrative purposes, various modifications within the scope of the present invention are possible as will be appreciated by those skilled in the relevant art. These modifications can be made to the present invention in accordance with the above detailed description. The terms used in the following claims should not be understood to limit the present invention to the specific embodiments disclosed in the specification. On the contrary, the scope of the present invention is fully determined by the following claims, which will be understood based on the content established by the interpretation of the claims.

Claims

1. A method for anomaly detection in an instrument of a surgical system, the method comprising: receiving measurements from a sensor that senses at a drive train of the instrument; predicting a first operation of the drive train during use of the instrument, the prediction being performed by a machine learning model in response to the measured input; detecting the anomaly based on a comparison of a predicted first operation output by the machine learning model and an actual operation of the drive train; as well as A detection indication of the abnormality is outputted. The method of claim 1 , wherein predicting comprises predicting the first operation as a normal operation.

3. The method of claim 2, wherein detecting comprises detecting the anomaly as a deviation from the normal operation such that the deviation is due to one or more of a variety of problems independent of detecting the anomaly as due to a particular one of the various problems.

4. The method of claim 1 , wherein the machine learning model is trained based on simulation, bench testing, or both simulation and bench testing, and the method further comprises retraining the machine learning model based on examples from use in patients.

5. The method of claim 1, wherein the instrument is of a first type, and wherein the machine learning model is trained for an instrument of a second type different from the first type and is retrained for the first type. 6 . The method of claim 1 , wherein receiving comprises receiving the measurements of motor position and motor load torque over time as a motor of the drive train. 7 . The method of claim 6 , wherein the sensor comprises an encoder configured to measure the motor position and a load sensor configured to measure the load torque. 8 . The method of claim 1 , wherein predicting comprises predicting the first operation as a cable force for a cable of the drive train.

9. The method of claim 1, further comprising repeating the receiving and the predicting over time, and wherein detecting the anomaly comprises detecting where the comparison exceeds a threshold difference within a period.

10. The method of claim 1, wherein predicting comprises predicting using the machine learning model comprising a recurrent neural network.

11. The method of claim 10, wherein predicting comprises predicting using the recurrent neural network including long short-term memory.

12. The method of claim 1, wherein receiving comprises receiving the measurement from the sensor, wherein the sensor senses at the drive chain for teleoperation of the instrument when mounted to a robotic arm, and wherein predicting comprises predicting during teleoperation.

13. A surgical robot system for abnormality detection, the surgical robot system comprising: a surgical tool connected by a first number of cables to a corresponding number of actuators, the surgical tool being connected such that actuation of the actuators moves the surgical tool; a first sensor configured to sense a position of the actuator; A processor is configured to detect the anomaly by applying a machine learning network configured to receive the location and output a predicted cable force.

14. The surgical robotic system of claim 13, further comprising a second sensor configured to sense a load of the actuator, and wherein the machine learning network is configured to output the predicted cable force in response to receiving the position and the load.

15. The surgical robotic system of claim 14, wherein the processor is configured to detect the anomaly by comparing the predicted cable force to an actual cable force.

16. The surgical robotic system of claim 13, wherein the machine learning network comprises a recurrent neural network with long short-term memory.

17. The surgical robotic system of claim 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the first set being based on non-patient data and the second set being based on patient data.

18. The surgical robotic system of claim 13, wherein the machine learning network is trained on a first set of data and retrained on a second set of data, the second set being based on a tool type of the surgical tool, and the second set being based on a different tool type.

19. A surgical robot system for abnormality detection, the surgical robot system comprising: a robotic arm configured to hold and manipulate surgical tools; and A processor configured to detect the anomaly by applying a machine learning model configured to receive past operation information and output a predicted operation.

20. The surgical robotic system of claim 19, wherein the machine learning model comprises a neural network configured through training to output the predicted operation as a normal operation, and wherein the processor is configured to detect the anomaly when the predicted normal operation deviates from an actual operation of the surgical tool performed by the robotic arm.

21. The surgical robotic system of claim 19, further comprising a load sensor configured to detect a load on an actuator and an encoder configured to detect a position of the actuator, wherein the past operation information comprises a sequence of the positions and the loads, and wherein the predicted operation comprises a cable force.

Images