Automatic selection of collaborative robot control parameters based on tool and user interaction forces

Abstract

A system includes a robotic arm having an instrument interface, a force / torque sensor to sense a force at the instrument interface, a robot controller to control the robotic arm and to control robot control parameters, and a system controller. The system controller receives temporal force / torque data, wherein the temporal force / torque data is representative of the force at the instrument interface over time during a collaborative procedure with a user, analyzes the temporal force / torque data to determine a current intent of the user and / or a state of the collaborative procedure, and causes the robot controller to control the robotic arm in a control mode, the control mode being predefined for the determined current intent of the user or state of the collaborative procedure, wherein the control mode determines the robot control parameters.

Description

Technical Field

[0001] This invention relates to robots, and more particularly to collaborative robots that can be used, for example, in a surgical operating room, and methods for operating such collaborative robots. Background Technology

[0002] Collaborative robots are robots that work in the same space as humans, often interacting directly with them (e.g., through force control). An example of such a collaborative robot is one that includes an end effector, used to hold or guide a tool as it is manipulated by a human to perform a task. Collaborative robots are generally considered safe and do not require dedicated safety barriers. As more people accept such collaborative robots, they expect them to exhibit greater intelligence and automation.

[0003] Collaborative robots should possess advanced perception capabilities to provide intuitive assistance in protocol-heavy workflows, such as those in the operating room. However, compared to humans, robots suffer from very poor context awareness due to limited sensory modalities, quality, and bandwidth feedback. For these robots to truly collaborate, they need some ability to autonomously modify their behavior based on environmental states, the task at hand, and / or user intent.

[0004] Therefore, it is desirable to provide collaborative robots and methods for operating collaborative robots. In particular, it is desirable to provide collaborative robots and methods for operating collaborative robots that can provide automatic selection of one or more robot control parameters based on environmental conditions, the task at hand, and / or user intentions. Summary of the Invention

[0005] In one aspect of the invention, a system includes: a robotic arm having one or more control degrees of freedom, wherein the robotic arm includes a machine interface; at least one force / torque sensor configured to sense a force at the machine interface; a robot controller configured to control the robotic arm to move the machine interface to a determined position and control at least one robot control parameter; and a system controller. The system controller is configured to: receive time-varying force / torque data, wherein the time-varying force / torque data represents the force at the machine interface sensed by the at least one force / torque sensor over time during a collaborative process with a user; analyze the time-varying force / torque data to determine at least one of the user's current intention and the state of the collaborative process; and cause the robot controller to control the robotic arm in a control mode predefined for the determined current intention of the user or the state of the collaborative process, wherein the control mode determines at least one robot control parameter.

[0006] In some embodiments, the instrument interface includes a tool guide configured to connect to a tool interface that can be manipulated by the user during the collaborative process, and the force applied by the user to the instrument interface includes at least one of the following: (1) a force indirectly applied to the tool guide during user manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

[0007] In some embodiments, the system controller is configured to apply the time force / torque data to a neural network to determine the user's current intent or the state of the collaborative process.

[0008] In some embodiments, the neural network is configured to determine when the user drills with the tool based on the time-force / torque data, and is also configured to determine when the user hammers with the tool based on the time-force / torque data.

[0009] In some embodiments, the at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction.

[0010] In some embodiments, when the neural network determines, based on the time-force / torque data, that the user is hammering with the tool, the neural network further determines whether the tool is hammering through bone or through tissue. When the tool is determined to be hammering through tissue, the control mode is a first stiffness mode, wherein the robot controller controls the tool guide to have a first stiffness. And when the tool is determined to be hammering through bone, the control mode is a second stiffness mode, wherein the robot controller controls the tool guide to have a second stiffness, wherein the second stiffness is less than the first stiffness.

[0011] In some embodiments, the system provides an alert to the user when the system changes the control mode.

[0012] In some embodiments, the system controller is further configured to receive auxiliary data, which includes at least one of the following: video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data, and the system controller is further configured to determine the user's current intention or the state of the collaborative process based on the time force / torque data and the auxiliary data.

[0013] In another aspect of the invention, a method for operating a robotic arm having one or more control degrees of freedom is provided, wherein the robotic arm includes a machine interface. The method includes: receiving time-varying force / torque data, wherein the time-varying force / torque data represents forces at the machine interface sensed over time by a force / torque sensor during a collaborative process with a user; analyzing the time-varying force / torque data to determine at least one of the user's current intention and the state of the collaborative process; and controlling the robotic arm in a control mode predefined for the determined current intention of the user or the state of the collaborative process, wherein the control mode determines at least one robot control parameter.

[0014] In some embodiments, the instrument interface includes a tool guide configured to connect to a tool interface operable by the user during the collaborative process, wherein the force / torque sensor measures at least one of the following: (1) a force indirectly applied to the tool guide by the user during the user's manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

[0015] In some embodiments, analyzing the time-force / torque data to determine at least one of the user's current intention and the state of the collaboration process includes applying the time-force / torque data to a neural network to determine the user's current intention or the state of the collaboration process.

[0016] In some embodiments, the neural network determines when the user drills with the tool based on the time-force / torque data, and also determines when the user hammers with the tool based on the time-force / torque data.

[0017] In some embodiments, the at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction.

[0018] In some embodiments, when the neural network determines, based on the time-force / torque data, that the user is hammering with the tool, the neural network further determines whether the tool is hammering through bone or through tissue. When the tool is determined to be hammering through tissue, the control mode is a first stiffness mode, wherein the tool guide has a first stiffness. And when the tool is determined to be hammering through bone, the control mode is a second stiffness mode, wherein the tool guide has a second stiffness, wherein the second stiffness is less than the first stiffness.

[0019] In some embodiments, the method further includes providing an alert to the user when the control mode changes.

[0020] In some embodiments, the method further includes: receiving auxiliary data, the auxiliary data including at least one of the following: video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data; and determining the user's current intention or the state of the collaboration process based on the time force / torque data and the auxiliary data.

[0021] In another aspect of the invention, a processing system for controlling a robotic arm is provided, the robotic arm having one or more control degrees of freedom, wherein the robotic arm includes a machine interface. The processing system includes: a processor; and a memory storing instructions. When executed by the processor, the instructions cause the processor to: receive time-force / torque data, wherein the time-force / torque data represents forces at the machine interface at any time during a collaborative process with a user; analyze the time-force / torque data to determine at least one of the user's current intention and the state of the collaborative process; and cause the robotic arm to be controlled in a control mode predefined for the determined current intention of the user or the state of the collaborative process, wherein the control mode sets at least one robot control parameter.

[0022] In some embodiments, the instrument interface includes a tool guide configured to connect to a tool interface that can be manipulated by the user during the collaborative process, and the force includes at least one of the following: (1) a force indirectly applied to the tool guide by the user during the user's manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

[0023] In some embodiments, the instructions further cause the processor to analyze the time-force / torque data to identify commands provided by the user to the system, thereby instructing the system to switch the control mode to a predefined mode.

[0024] In some embodiments, the at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction. Attached Figure Description

[0025] Figure 1 The illustration shows an example of a surgical operating room where a surgeon works with a collaborative robot during a simulated spinal fusion procedure.

[0026] Figure 2 An example embodiment of a collaborative robot tool guide with force sensing is illustrated.

[0027] Figure 3 An example embodiment of a collaborative robot is illustrated.

[0028] Figure 4 The illustration shows a block diagram of an example embodiment of a processor and associated memory according to an embodiment of the present disclosure.

[0029] Figure 5 The figure illustrates the force / torque curves for hammering and drilling at different stages of a collaborative surgical intervention.

[0030] Figure 6 The illustration shows an example of an arrangement for classifying events during a collaborative process based on robot data, which is based on force / torque data and robot state detected by mapping.

[0031] Figure 7 The illustration shows a first example embodiment of a control flow for automatically switching the control mode of a collaborative robot based on force / torque state detection performed by the collaborative robot.

[0032] Figure 8 The illustration shows a second example embodiment of a control flow for automatically switching the control mode of a collaborative robot based on force / torque state detection performed by the collaborative robot.

[0033] Figure 9 The illustration shows a flowchart of an example embodiment of a method for controlling a collaborative robot based on force / torque state detection performed by the collaborative robot. Detailed Implementation

[0034] The invention will now be described more fully with reference to the accompanying drawings, in which preferred embodiments of the invention are illustrated. However, the invention may be embodied in different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided as illustrative examples of the invention.

[0035] In particular, various systems have been described in the context of robot-guided surgery (e.g., spinal fusion surgery) to illustrate the principles of the invention. However, it should be understood that this is to illustrate specific examples of collaborative robots and methods of operating collaborative robots. More broadly, various aspects of collaborative robots and methods of operating collaborative robots disclosed herein can be applied to a wide range of other contexts and settings. Therefore, the invention is to be understood as defined by the claims, and not limited to the details of the specific embodiments described herein, unless such details are recited in the claims themselves.

[0036] In this article, when something is referred to as “approximately” or “about” a certain value, it means within 10% of that value.

[0037] Figure 1 An example of a surgical operating room 100 is illustrated, in which a surgeon 10 operates with a collaborative robot 110 in a simulated robot-guided spinal fusion surgical procedure. Figure 1 The diagram shows a robot controller 120, a surgical navigation display 130, a camera 140, and a cone-beam computed tomography (CBCT) device, which assist surgeon 10 in performing a robot-guided spinal fusion surgical procedure. Here, robot 110 assists surgeon 10 in precisely creating holes within the pedicles (vertebral segments) along a planned trajectory. After creating holes in multiple pedicles using a needle or drill, surgeon 10 places screws within these navigation holes and secures adjacent screws with rods to fuse multiple vertebrae in the desired configuration.

[0038] Currently, robot behavior or patterns are manually altered by the robot user or another human assistant, which is inefficient in terms of overall time and latency and disrupts workflows. In some cases, human operators may not even be aware that it is beneficial to change robot patterns in a timely manner. Such changes can include, for example, altering robot compliance based on the type of task being performed (e.g., drilling versus hammering), or changing the safety zone (tool angle / position) based on the type of tissue the instrument is traversing.

[0039] Current methods for managing this situation include threshold-based event / state detection, but these methods lack sufficient robustness and specificity to discern the complexity and veracity of signals associated with specific events or states within a process. The same applies to Fourier space analysis techniques. Furthermore, pattern changes must be intuitive and transparent, thus requiring communication of the chosen type of behavior.

[0040] To address some or all of these needs, the inventors have conceived of collaborative robots and control methods for collaborative robots that utilize force sensing of tool interaction forces to automatically modify robot behavior based on the robot's state and relevant dynamic force information sensed at the tool interface.

[0041] Figure 2 An example embodiment of a collaborative robot 110 and an associated tool guide 30 with force sensing is illustrated. Figure 2As shown, the collaborative robot 110 includes a robotic arm 111 having an instrument interface including a tool guide 30 disposed at the end effector 113 of the robotic arm 111. Here, the tool guide 30 may have a cylindrical shape, and a tool or instrument 20 (e.g., a drill, needle, etc.) with a handle 22 passes through an opening in the tool guide 30 for use by a surgeon during a surgical procedure (e.g., a spinal fusion surgical procedure).

[0042] The collaborative robot 110 also includes a force / torque sensor 112, which senses forces applied by the user to or on the tool guide 30 during operation, for example, in Figure 1 The forces indirectly applied by the surgeon 10 during the spinal fusion surgical procedure illustrated, while manipulating the tool 20 within the tool guide 30, and / or forces that may be directly applied to the tool guide 30 as commands by the user or surgeon 10, are discussed in more detail below. In some cases, force / torque sensors may also sense forces from the robot's environment and / or forces generated by the tool or instrument 20. An example of a suitable force / torque sensor 112 is the Nano25 force / torque sensor—a six-axis transducer from ATI Industrial Automation.

[0043] Advantageously, the collaborative robot 110 can be directly controlled by a user (e.g., surgeon 10) pushing the tool guide 30. Surgeon 10 can use handover control (also known as "force control" or "admittance control") to adjust the position of the collaborative robot 110. The collaborative robot 110 can also be used as an intelligent tool guide, thereby precisely moving the cylindrical tool guide 40 to a planned position and posture or attitude for a planned trajectory, and maintaining that position when surgeon 10 engages an instrument or tool 20 (e.g., a needle) inside the tool guide 30 with the pedicle by hammering or drilling.

[0044] As described in more detail below, the admittance control method (using signals from force / torque sensor 112) also allows for independent adjustment of the compliance of the collaborative robot 110 (more specifically, the compliance of end effector 113 and tool guide 30) in each degree of freedom (DOF), for example, being very stiff in Cartesian rotations but very compliant in Cartesian translations.

[0045] Figure 3 A more general example embodiment of the collaborative robot 110 is illustrated.

[0046] The collaborative robot 110 includes a robot body 114 and a robot arm 111 extending from the robot body 114. The robot arm 111 has a tool interface including a tool guide 30, which is held by an end effector 113 disposed at the end of the robot arm 111. The end effector 113 may include a gripping mechanism for grasping and holding the tool guide 30. Figure 3 A tool 20 is shown passing through an opening in a cylindrical tool guide 30, and the tool 20 has a handle 22 that a user (e.g., a surgeon) can manipulate to perform a desired collaborative process.

[0047] The collaborative robot 110 also includes a robot controller 120 and a system controller 300. The robot controller 120 may include one or more processors, memories, actuators, motors, etc., for realizing the movement of the collaborative robot 110, particularly the movement and orientation of the machine interface including the tool guide 30. Figure 3 As shown, the system controller 300 may include one or more processors 310 and (one or more) associated memory 320.

[0048] In some embodiments, robot controller 120 may be integrated with robot body 140. In other embodiments, some or all components of robot controller 120 may be provided separately from robot body 140, for example, provided as a laptop computer or other device that may include a display and graphical user interface. In some embodiments, system controller 300 may be integrated with robot body 140. In other embodiments, some or all components of system controller 300 may be provided separately from robot body 140. In some embodiments, one or more processors or memories of system controller 300 may be shared with robot controller 120. Many different partitions and configurations of robot body 140, robot controller 120, and system controller 300 are contemplated.

[0049] The robot controller 120 and the system controller 300 are described in more detail below.

[0050] The robotic arm 111 may have one or more joints, each of which may have up to six degrees of freedom—for example, translation along any combination of mutually orthogonal x, y, and z axes, and rotation about the x, y, and z axes (also known as yaw, pitch, and roll). Alternatively, some or all of the joints of the robotic arm 111 may have fewer than six degrees of freedom. Movement of the joints in any or all degrees of freedom can be performed in response to control signals provided by the robot controller 120. In some embodiments, motors, actuators, and / or other mechanisms for controlling one or more joints of the robotic arm 111 may be included in the robot controller 120.

[0051] The collaborative robot 110 also includes a force / torque sensor 112, which senses forces applied to or at the instrument interface, such as forces applied to the tool 20 by the tool 20 disposed within the tool guide 30 when the surgeon 10 is manipulating the tool 20 during a spinal fusion surgical procedure. Figure 1 As shown. In some embodiments, the collaborative robot may include a plurality of force / torque sensors 112.

[0052] The robot controller 120 may control the robot 110 in part in response to one or more control signals received from the system controller 300, as described in more detail below. Conversely, the system controller 300 may output one or more control signals to the robot controller 120 in response to one or more signals received from the force / torque sensor 112. Specifically, the system controller 300 receives time-varying force / torque data, which represents the force applied over time to or at the instrument interface including the tool guide 30, and is sensed by the force / torque sensor 112 during a collaborative process performed by a user. As described below, the system 300 may be configured to: interpret the signals(s) from the force / torque sensor 112 to determine the user's intent and / or commands to the collaborative robot 110, and control the collaborative robot 110 to perform actions according to the user's intent or commands (as represented by the force / torque sensed by the force / torque sensor 112).

[0053] Figure 4 A block diagram of an example embodiment of a processor 400 and an associated memory 450 according to an embodiment of the present disclosure is illustrated.

[0054] Processor 400 may be used to implement one or more processors described herein, for example, Figure 3The processor 310 is shown. The processor 400 can be any suitable processor type, including but not limited to microprocessors, microcontrollers, digital signal processors (DSPs), field-programmable arrays (FPGAs) (wherein the FPGA has been programmed to form a processor), graphics processing units (GPUs), application-specific integrated circuits (ASICs) (wherein the ASIC has been designed to form a processor), or combinations thereof.

[0055] Processor 400 may include one or more cores 402. Core 402 may include one or more arithmetic logic units (ALUs) 404. In some embodiments, in addition to or in place of ALU 404, core 402 may include a floating-point logic unit (FPLU) 406 and / or a digital signal processing unit (DSPU) 408.

[0056] Processor 400 may include one or more registers 412 communicatively coupled to core 402. Registers 412 may be implemented using dedicated logic gates (e.g., flip-flops) and / or any memory technology. In some embodiments, registers 412 may be implemented using static memory. Registers 412 may provide data, instructions, and addresses to core 402.

[0057] In some embodiments, processor 400 may include one or more levels of cache memory 410 communicatively coupled to core 402. Cache memory 410 may provide computer-readable instructions to core 402 for execution. Cache memory 410 may provide data for core 402 to process. In some embodiments, computer-readable instructions may be provided to cache memory 410 from local memory (e.g., local memory attached to external bus 416). Cache memory 410 may be implemented using any suitable cache memory type, such as metal-oxide-semiconductor (MOS) memory, e.g., static random access memory (SRAM), dynamic random access memory, and / or any other suitable memory technology.

[0058] Processor 400 may include controller 414, which can control other processors and / or components included in the system (e.g., Figure 3 The force / torque sensor 112 in the system provides input to the processor 400 and / or from the processor 400 to other processors and / or components included in the system (e.g., Figure 3The output of the robot controller 120 is specified. The controller 414 can control the data paths in the ALU 404, FPLU 406, and / or DSPU 408. The controller 414 can be implemented as one or more state machines, data paths, and / or dedicated control logic units. The gates of the controller 414 can be implemented as stand-alone gates, FPGAs, ASICs, or any other suitable technology.

[0059] Register 412 and cache memory 410 can communicate with controller 414 and core 402 via internal connections 420A, 420B, 420C and 420D. The internal connections can be implemented as a bus, multiplexer, crossbar switch and / or any other suitable connection technology.

[0060] Inputs and outputs to processor 400 may be provided via bus 416, which may include one or more wires. Bus 416 may be communicatively coupled to one or more components of processor 400, such as controller 414, cache memory 410, and / or register 412. Bus 416 may be coupled to one or more components of the system, such as the previously mentioned robot controller 120.

[0061] Bus 416 may be coupled to one or more external memories. The external memories may include read-only memory (ROM) 432. ROM 432 may be a mask ROM, electronically programmable read-only memory (EPROM), or any other suitable technology. One or more external memories may include random access memory (RAM) 433. RAM 433 may be static RAM, battery-backed static RAM, dynamic RAM (DRAM), or any other suitable technology. One or more external memories may include electrically erasable programmable read-only memory (EEPROM) 435. One or more external memories may include flash memory 434. One or more external memories may include magnetic storage devices, such as a disk 436. In some embodiments, the external memories may be included in a system such as robot 110.

[0062] A collaborative robot 110, including a force / torque sensor 112, is used to measure force / torque (FT) in a static tool guide holding mode during drilling and hammering of the pedicle, a common but extremely difficult task in spinal fusion.

[0063] Figure 5 The illustration shows the force / torque distribution 500 for hammering and drilling at different stages of collaborative surgical intervention.

[0064] Figure 5A first force / torque trace 510 is shown, representing the force applied as a function of time to or at the instrument interface including the tool guide 30 when a surgeon is performing a hammering operation with a tool 20 disposed within the tool guide 20. The first torque trace 510 includes two distinct and identifiable temporal force / torque patterns: a first temporal pattern 512 corresponding to the force / torque applied to the instrument interface including the tool guide 30 during a hammering operation or procedure penetrating soft tissue, and a second temporal pattern 514 corresponding to the force / torque applied to the instrument interface including the tool guide 30 during a hammering operation or procedure penetrating bone. A second torque trace 520 shows a temporal pattern corresponding to the force / torque applied to the instrument interface including the tool guide 30 during a drilling operation or procedure penetrating bone.

[0065] Figure 5 The force / torque traces depict the distinct differences in the temporal patterns of the force / torque applied to the instrument interface, including the tool guide 30, between hammering and drilling (and even between tissue types that interact with the instrument or tool).

[0066] As discussed in more detail below, the system controller 300 can be configured to recognize patterns in these different temporal force / torque data, thereby clarifying what operation the user (e.g., surgeon 10) is performing. The temporal force / torque data can be supplemented by knowledge of the ordered operations to be performed during a particular surgical procedure (e.g., the ordered operations the surgeon should perform during a robot-guided spinal fusion surgical procedure). For example, it can be anticipated that after a puncture of tissue, there will be a puncture into bone, followed by drilling into the bone, etc. Such knowledge can be stored in memory associated with the system controller 300, and the system controller's processor can access this knowledge when the system controller 300 controls the operation of the collaborative robot 110 during a collaborative surgical procedure.

[0067] A system and method for detecting intervention status (or user intent) during a collaborative process employs a recurrent neural network to consider a time series of force / torque measurement data as well as the current state of the collaborative robot 110 (e.g., the velocity of the collaborative robot 110 at the same location as the force / torque measurement data resolved at, for example, at tool guide 30). This type of network can be trained with data collected from multiple trials to improve its performance.

[0068] Figure 6 The illustration shows an example of an arrangement for classifying events during a collaborative process based on robot data, which is based on force / torque data and robot state detected by mapping.

[0069] Figure 6 A neural network 600 is illustrated. The neural network 600 receives a robot state sequence 602 of the collaborative robot 110 and temporal force / torque data 604 as input. The temporal force / torque data 604 represents the forces applied over time to the instrument interface, including the tool guide 30, during the collaborative process. For example, the forces applied to the tool guide 30 by the tool 20 disposed within the tool guide 30 when the tool 20 is manipulated by a user (e.g., surgeon 10). The robot state sequence 602 is, for example, a time series of robot states during the collaborative process, from previous operation to the present. In response to the temporal force / torque data 604 and the robot state sequence 602, the neural network 600 outputs the current robot state of the collaborative robot 110 from a set of possible robot states 610 of the collaborative robot 110.

[0070] Each possible robot state 610, in turn, corresponds to one or more control modes of the collaborative robot 110. For example, as Figure 6 As shown, when the neural network 600 determines that the current robot state of the collaborative robot 110 is a "hammering soft tissue" state, it causes the collaborative robot 110 to operate in a "high stiffness control mode" 620A. Conversely, when the neural network 600 determines that the current robot state of the collaborative robot 110 is a "hammering internal skeleton" state, it causes the collaborative robot 110 to operate in a "low stiffness control mode" 620B. In some cases, the relative stiffness can be reversed to maintain the planned trajectory, regardless of the geometry of the anatomical structure.

[0071] In some embodiments, the neural network 600 can be implemented in the system controller 300 by the processor 310 running a computer program defined by instructions stored in the memory 320. Other embodiments utilizing various combinations of hardware and / or firmware and / or software are also possible.

[0072] exist Figure 6 In the example, at any given time, the collaborative robot 110 can operate in any of the six defined robot states (including hammering soft tissue, hammering internal bone, drilling cortex, drilling cancellous tissue, unknown, and inactive). Other robot states 610 are also possible, such as a moving state, a state where scraping is detected, a retracted state, a tool-guided state with inserted instruments, a state where a gesture is detected (indicating the expectation of specific user control), etc., depending on the collaborative process in which the collaborative robot 110 participates.

[0073] Advantageously, each robot state 610 can be defined as a Cartesian velocity resolved at the instrument interface including the tool guide 30 (at the end of the end effector 113). The velocity can be calculated based on the joint encoder values ​​and the forward kinematics model of the collaborative robot 110, as well as the robot's Jacobian matrix, which correlates the joint velocity with the linear and angular velocities of the end effector 113.

[0074] The robot state 610 may also include, for example: the speed from an external tracking system (optical tracker, electromagnetic tracker); the torque on the motor of the robot arm 111; the type of tool 20 used; the tool's state (drilling on or off, position tracking, etc.); data from the accelerometer on the tool 20 or the collaborative robot 110, etc.

[0075] Force / torque data can represent force / torque over one or more degrees of freedom. Generally, force / torque can be resolved at one of several convenient locations. Advantageously, force / torque can be resolved at the tool guide 30 at the end of the end effector 113. Generally, preprocessing of the time force / torque data 604 is not required, except for basic noise reduction and processing to resolve force / torque at a specific location (e.g., at the tool guide 30 at the end of the end effector 113).

[0076] Advantageously, robot state detection methods can (e.g., via neural network 600) use data-driven models to continuously classify the robot state (also called type) of a process or intervention based on short-term data (i.e., temporal force / torque data 604) output from force / torque sensor 112 and robot state 610. Many potential model architectures can be used to classify time-series data with multiple inputs. One useful model is the Long Short-Term Memory (LSTM) network because it is more stable than a typical Recurrent Neural Network (RNN). Examples of other possible networks include echo-state networks, convolutional neural networks (CNNs), convolutional LSTMs, and handcrafted networks using multilayer perceptrons, decision trees, logistic regression, etc.

[0077] Beneficially, the data stream input to the neural network 600 (which may include force / torque data (604), robot state (602), and current control mode (see below) Figure 7Preprocessing (interpolation, downsampling, upsampling, etc.) is performed on each data point to ensure that each data point has the same time period (typically the time period of the highest frequency data stream, which is typically from the time data of the force / torque sensor 112) (e.g., a 1 kHz or 1 ms time period). In some embodiments, the time sliding window of the neural network 600 can be set to approximately three (3) seconds long to capture typical events (e.g., drilling, reaming, pushing, pulling, twisting, and hammering (e.g., hammering with a typical interval of approximately one (1) second)) while being short enough to respond to a given task. Each window shift (forward) is a new sample to be classified, and during the training phase, it has an associated robot state label. Smaller windows may be discarded.

[0078] In some embodiments, a single input sample may contain 12 features for each of N time steps (e.g., 36K individual features over 3 seconds at 1 kHz sampling), with each feature consisting of: 3 features for force, 3 features for torque, 3 features for XYZ robot linear velocity, and 3 features for robot angular velocity. In some embodiments, the output of the LSTM is the probability of each robot state for a given input window sample. In some embodiments, the model has two LSTM hidden layers, followed by a dropout layer (to reduce overfitting), a dense fully connected layer with a common modified linear unit (“ReLU”) activation function, and an output layer with a normalized exponential function (“softmax”) activation. LSTM and ReLU are common building blocks of deep learning models that can be understood by those skilled in the art. The loss function is class cross-entropy and can be optimized using the Adaptive Learning Rate Optimization Algorithm (Adam) optimizer. The Adam optimizer is a widely used optimizer for deep learning models, as described in, for example, "Adam: A method for stochastic optimization" by Diederik P. Kingma et al. (3rd International Conference on Learning Representations, San Diego, 2015).

[0079] Beneficially, during training, care is employed to balance examples of all the different robot states expected, especially those rarely experienced (e.g., drilling) compared to the most common (or one or more) robot states (e.g., inactivity). This reduces the bias toward common robot states. These techniques may include undersampling of the most common robot states and / or oversampling of rare robot states in the training sequence. Additionally, a cost-based classifier is used to penalize misclassifications of robot states of interest while reducing the cost of correctly classifying (or one or more) common robot states. This is particularly useful in cases where rare events occur within a time window (e.g., three hammer blows of inactivity vs. drilling for three seconds).

[0080] Figure 7 and Figure 8 The illustration shows two different examples of control mode switching algorithms used for collaborative robots such as the collaborative robot 110.

[0081] Figure 7 A first example embodiment of a control flow 700 for automatically switching control modes of a collaborative robot 110 based on force / torque state detection performed by the collaborative robot 110 is illustrated. The control flow 700 can be implemented by a system controller 300, and more specifically by the processor 310 of the system controller 100. The control flow 700 employs a single model (and a corresponding single neural network 600) with a current mode input 606 to detect the robot state 610 during the current control mode 620. The neural network 600 implicitly takes into account the context of the current control mode in the robot state detection.

[0082] Initially, in operation 702, the system controller 300 selects a start control mode, either in response to direct input from a user (e.g., surgeon 10) or as a pre-programmed initial control mode for the collaborative robot 110 (which can be determined for a specific collaborative process).

[0083] In operation 704, system controller 300 sets a current control mode 706 for collaborative robot 110, initially as a start control mode. System controller 300 may provide one or more signals to robot controller 120 to indicate the current control mode 706 and / or cause robot controller 120 to control collaborative robot 110 (specifically robot arm 111) according to the current control mode 706. By setting the current mode (examples of which have been described above), system controller 300 may control one or more robot control parameters, including, for example, the amount of stiffness exhibited by tool guide 30 against forces applied to tool guide 30 in one or more of up to six degrees of freedom (e.g., in at least one direction).

[0084] In some embodiments, in addition to the stiffness presented at the tool guide 30, the system controller 300 may also control other robot control parameters, such as position constraints (trajectory constraints), dwell time (at a specific location), acceleration of the robot arm 111, vibration, drilling speed (on / off) of the tool 20, maximum speed and minimum speed, etc.

[0085] The control flow 700 employs a robot state detection network 750 to determine or detect the robot state 610 of the collaborative robot 110. As described above, the state detection network 750 includes a neural network 600 that receives a sequence of robot states 602 of the collaborative robot 110, time force / torque data 604, and the current control mode 606 as its inputs, and selects the robot state 610 from a variety of possible robot states of the collaborative robot 110 in response to these inputs.

[0086] Operation 712 maps the robot state 610 detected by the robot state detection network 750 to the mapping control mode 620 of the collaborative robot 110.

[0087] Operation 714 determines whether the mapped control mode 620 is the same as the current control mode 706 of the collaborative robot 110. If so, the current control mode 706 remains unchanged. If not, the current control mode 706 should be changed or switched to the mapped control mode 620.

[0088] In some embodiments, during operation 716, system controller 300 may alert the user to the fact that system controller 300 has pending control mode switching requests. System controller 300 may request the user (e.g., surgeon 10) to acknowledge or approve the control mode switching request. In some embodiments, operation 716 may be run only for some specific processes, while operation 716 may be skipped for others.

[0089] In some embodiments, the control mode switching request for operation 716 may be presented to the user via a user interface associated with system controller 300, such as visually presenting the request via a display device (e.g., display device 130) or audibly (e.g., verbally) via a speaker.

[0090] In those embodiments or processes that execute the control mode switching request in operation 716, then in operation 718, the system controller determines whether the user (e.g., surgeon 10) acknowledges or approves the control mode switching request. The user or surgeon can acknowledge or approve (or conversely, deny or disapprove) the control mode switching request in any of a variety of ways. Examples include:

[0091] Users or surgeons can respond by clicking the pedals or buttons on the user interface to confirm the change in control mode.

[0092] • Voice recognition can be used to confirm user approval or acceptance of changes to control modes.

[0093] • Users can approve or accept changes in control modes via gestures / body postures, which can be detected using a vision or depth tracking camera and provided to the system controller 300 as supplementary or auxiliary data input.

[0094] In some cases, control mode switching without confirmation may be acceptable, and these cases may be mixed with some that require user input. Therefore, operations 716 and 718 can be optional. In these cases, a simple auditory effect indicating to the user or surgeon which mode has been entered is sufficient. If this is not the desired mode, the user or surgeon can cancel or stop the robot's movement. It is beneficial to use audiovisual means (e.g., digital displays, LEDs on the robot, voice feedback describing what the system is sensing and changing) to clearly communicate the currently detected robot state and control mode to the user or surgeon.

[0095] If the control mode switching request is not approved, the current control mode 706 will remain.

[0096] On the other hand, if the control mode switching request is approved, or if operations 716 and 718 are omitted, operation 704 is repeated to set the mapping mode 620 to the new current control mode 706 of the collaborative robot 110. The new current control mode 706 is provided to the input 606 of the neural network 600 and is provided to the robot controller 120 as one or more output signals.

[0097] Figure 8 The illustration shows a second example embodiment of a control flow 800 for automatically switching the control mode of a collaborative robot based on force / torque state detection performed by the collaborative robot. The control flow 800 can be implemented by a system controller 300, and more specifically by a processor 310 of the system controller 300.

[0098] For the sake of brevity, the descriptions of the same operations and process paths in control mode 800 as those in control mode 700 will not be repeated.

[0099] In contrast to control flow 700, control flow 800 employs multiple robot state detection networks 850A, 850B, 850C, etc., to detect robot state, with one robot state detection network for each control mode selected during a control mode switching event. Each of the robot state detection networks 850A, 850B, 850C, etc., implements a corresponding model for robot state detection and outputs the corresponding detected robot state. In control flow 800, during operation 855, the detected robot state output from one of the multiple models (and its corresponding neural network 600) is explicitly selected for each control mode.

[0100] In some embodiments, a handcrafted state machine layer can be added to prevent false positives and false negatives, time filters can be added, and process planning or longer-term state transitions can be considered. For example, in the case of pedicle drilling, a physician is unlikely to hammer the drill bit after drilling has been performed, and a higher-level state machine can be included in the control flow to detect such inconsistencies. Errors regarding improper use of the collaborative robot 110 or failure to follow procedures can be communicated.

[0101] Figure 9 A flowchart illustrating an exemplary embodiment of method 900 is shown, which controls a collaborative robot (e.g., collaborative robot 110) based on force / torque state detection performed by the collaborative robot 110 during a process or intervention.

[0102] In operation 910, a user (e.g., surgeon 10) manipulates an instrument or tool (e.g., tool 20) that applies force to an instrument interface (e.g., tool guide 30) or other parts of the robotic arm 111 in a collaborative procedure (e.g., a spinal fusion surgical procedure).

[0103] In operation 920, one or more force / torque sensors 112 sense forces applied to the instrument interface or other parts of the robot arm 111 (e.g., forces at the tool guide 30).

[0104] In operation 930, the processor 310 of the system controller 300 receives time-dependent force / torque data 604 generated from one or more force / torque sensors 112.

[0105] In operation 940, processor 310 analyzes time force / torque data 604 to determine the user's current intent and / or the state of one or more robots during the collaborative process.

[0106] In operation 950, system controller 400 determines the control mode of collaborative robot 110 based on the user's current intent and / or the current robot state and / or (one or more) past robot states during the collaborative process.

[0107] In operation 960, the system controller 300 notifies the user of the determined control mode to which the collaborative robot should be set, and waits for user confirmation before setting or changing the current control mode to the determined control mode.

[0108] In operation 970, the system controller 300 sets the control mode of the collaborative robot 110.

[0109] In operation 980, system controller 300 sets one or more robot control parameters based on the current control mode. These one or more robot control parameters can control the amount of stiffness exhibited by tool guide 30, for example, in one or more of up to six degrees of freedom. In some embodiments, in addition to the stiffness exhibited at tool guide 30, system controller 300 can also control other operating parameters, such as position constraints (trajectory constraints), dwell time (at a specific location), etc.

[0110] Many variations of the above embodiments can be conceived.

[0111] For example, in the basic case described above, the force / torque sensor 112 is located between the main robot body 114 and the tool guide 30. However, in some embodiments, the force / torque sensor 112 may be located near the tool guide 30 or integrated into the robot body 114. Advantageously, six-degree-of-freedom force / torque sensing technology can be employed. Torque measurements at the joints of the robot arm 111 can also provide basic information about the force / torque resolved at the tool guide 30. The force can be resolved at the tool guide 30 or at the estimated or measured location at the tip of the tool 20.

[0112] In some robot / sensor configurations, the system controller 300 can distinguish user-applied input forces / torques based on forces / torques exerted on the instrument by the environment (e.g., force / torque sensors integrated on the instrument tip and another force / torque sensor on the tool guide). For example, environmental forces (e.g., tissue pushing on the tool) can be a primary source of feedback information for data models to determine whether the tool is passing through soft tissue or bone. That is, environmental forces include the results of anatomical structures responding to stimuli provided by the user and the robot through the tool.

[0113] In some embodiments, the system controller 300 may consider different inputs for detecting the robot's state during a collaborative process or intervention. Examples of such inputs may include:

[0114] Frequency domain of force / torque data

[0115] Frequency domain of velocity / acceleration data

[0116] Current robot status

[0117] • Estimated location of the target

[0118] • Types of processes

[0119] • Estimated bone type

[0120] • Estimated tissue type at the tip of the instrument (from navigation)

[0121] Robot stiffness

[0122] • Robot control mode

[0123] • Computed tomography data

[0124] Magnetic resonance imaging data

[0125] Each of these data inputs has a different behavior and can be considered in accordance with the content that the collaborative robot 110 expects to focus on.

[0126] In some embodiments, the system controller 300 may receive supplemental or auxiliary data input to help identify the background (search space) and improve robot state detection. Such data may include one or more of the following: video data, diagnostic data, image data, audio data, surgical plan data, time data, robot vibration data, etc. The system controller 300 may be configured to determine the current intent or state of the user (surgeon)'s collaborative process based on time-force / torque data and auxiliary data.

[0127] In some embodiments, a user (e.g., surgeon 10) may also apply force / torque to the collaborative robot 110 in a very specific manner to engage a particular control mode. For example, the system controller 300 of the collaborative robot 110 may be configured to recognize when the user applies a circumferential force to the tool guide 30 (either via an instrument or tool 209 in the tool guide 20 or by applying force directly to the tool guide 30), and in response, the system controller 300 may place the collaborative robot 110 into a typical force control mode (e.g., an admittance controller, which allows an operator to move the robot by applying force to it in a desired direction). In other words, the processor of the system controller 300 may be configured to analyze the time force / torque data 604 to identify commands provided by the user to the system controller 300, thereby instructing the system controller 300 to switch the control mode of the collaborative robot 110 to a predefined control mode. Some other examples of specific pressure actions by the user that can be interpreted as control mode commands may include:

[0128] • The user or surgeon presses up and down 3 times – panning mode only.

[0129] • The user presses up twice to make a circular motion – Insert only mode.

[0130] • Users apply pressure in a specific order (e.g., left, right, up, down) – then select the next planned trajectory.

[0131] Many other examples can be used for specific commands corresponding to the control mode.

[0132] In some embodiments, vibration sensing of the robot itself (e.g., via accelerometers) can supplement or replace force / torque sensing as described above. Events such as hammering and drilling cause vibrations in the robot's structure, which can be detected remotely from the robot's tool actuators and used in the same manner as described above.

[0133] Various embodiments can combine the above variations.

[0134] While preferred embodiments have been disclosed in detail herein, many other variations are possible within the spirit and scope of the invention. Such variations will become apparent to those skilled in the art upon examination of the specification, drawings, and claims herein. Therefore, the invention is not limited except within the scope of the claims.

Claims

1. A robot system, comprising: A robotic arm having one or more degrees of freedom for control, wherein the robotic arm includes a machine interface; At least one force / torque sensor is configured to sense force at the instrument interface; A robot controller configured to control the robot arm to move the instrument interface to a defined position and control at least one robot control parameter; and The system controller is configured as follows: Receive time-dependent force / torque data, wherein the time-dependent force / torque data represents the force at the device interface sensed over time by the at least one force / torque sensor during a collaborative process with the user. Analyze the time-force / torque data to determine at least one of the user's current intent and the state of the collaboration process, and The robot controller controls the robot arm in a control mode that is predefined for the current intent of the user or the state of the collaborative process, wherein the control mode determines the at least one robot control parameter. The instrument interface includes a tool guide configured to connect to a tool interface that can be manipulated by the user during the collaborative process, and wherein the force includes at least one of the following: (1) a force indirectly applied to the tool guide during user manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

2. The robot system according to claim 1, wherein, The system controller is configured to apply the time force / torque data to a neural network to determine the user's current intent or the state of the collaboration process.

3. The robot system according to claim 2, wherein, The neural network is configured to determine when the user drills with the tool based on the time-force / torque data, and is also configured to determine when the user hammers with the tool based on the time-force / torque data.

4. The robot system according to claim 3, wherein, The at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction.

5. The robot system according to claim 4, wherein, When the neural network determines that the user is hammering with the tool based on the time-force / torque data, the neural network further determines whether the tool is hammering through bone or through tissue. When the tool is determined to be hammering through tissue, the control mode is a first stiffness mode, where the robot controller controls the tool guide to have a first stiffness. And when the tool is determined to be hammering through bone, the control mode is a second stiffness mode, where the robot controller controls the tool guide to have a second stiffness, where the second stiffness is less than the first stiffness.

6. The robot system according to claim 1, wherein, When the robot system changes the control mode, the robot system provides an alarm to the user.

7. The robot system according to claim 1, wherein, The system controller is also configured to receive auxiliary data, which includes at least one of the following: video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data, and the system controller is also configured to determine the user's current intention or the state of the collaboration process based on the time force / torque data and the auxiliary data.

8. A computer program product comprising instructions, which, when executed by a computer, cause the computer to perform a method of operating a robotic arm, the robotic arm having one or more degrees of freedom, wherein, The robotic arm includes a machine interface, and the method includes: Receive time-dependent force / torque data, wherein the time-dependent force / torque data represents the force at the device interface at any given time, sensed by at least one force / torque sensor during a collaborative process with the user. Analyze the time-force / torque data to determine at least one of the user's current intent and the state of the collaboration process, and The robotic arm is controlled by a control mode that is predefined for the current intent of the user or the state of the collaborative process, wherein the control mode determines at least one robot control parameter. The instrument interface includes a tool guide configured to connect to the tool interface, which can be manipulated by the user during the collaborative process, and wherein the force / torque sensor measures at least one of the following: (1) a force indirectly applied to the tool guide by the user during the user's manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

9. The computer program product according to claim 8, wherein, Analyzing the time-force / torque data to determine at least one of the user's current intention and the state of the collaboration process includes applying the time-force / torque data to a neural network to determine the user's current intention or the state of the collaboration process.

10. The computer program product according to claim 9, wherein, The neural network determines when the user drills with the tool based on the time-force / torque data, and also determines when the user hammers with the tool based on the time-force / torque data.

11. The computer program product according to claim 10, wherein, The at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction.

12. The computer program product according to claim 11, wherein, When the neural network determines that the user is hammering with the tool based on the time force / torque data, the neural network further determines whether the tool is hammering through bone or through tissue. When the tool is determined to be hammering through tissue, the control mode is a first stiffness mode, where the tool guide has a first stiffness. And when the tool is determined to be hammering through bone, the control mode is a second stiffness mode, where the tool guide has a second stiffness, where the second stiffness is less than the first stiffness.

13. The computer program product according to claim 8, wherein, The method further includes providing an alert to the user when the control mode changes.

14. The computer program product according to claim 8, wherein, The method further includes: Receive auxiliary data, said auxiliary data including at least one of the following: video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data; and The user's current intention or the state of the collaboration process is determined based on the time force / torque data and the auxiliary data.

15. A processing system for controlling a robotic arm, the robotic arm having one or more control degrees of freedom, wherein, The robotic arm includes a machine interface, and the processing system includes: Processor; and A memory containing instructions that, when executed by the processor, cause the processor to: Receive time-force / torque data, wherein the time-force / torque data represents the force at the device interface at any time during the collaborative process with the user. Analyze the time-force / torque data to determine at least one of the user's current intent and the state of the collaboration process, and The robot arm is controlled in a control mode that is predefined for the current intent of the user or the state of the collaborative process, wherein the control mode sets at least one robot control parameter. The instrument interface includes a tool guide configured to connect to a tool interface that can be manipulated by the user during the collaborative process, and wherein the force includes at least one of the following: (1) a force indirectly applied to the tool guide by the user during the user's manipulation of the tool; (2) a force directly applied to the tool guide by the user; (3) a force from the robot's environment; and (4) a force generated by the tool.

16. The processing system according to claim 15, wherein, The instructions also cause the processor to analyze the time-force / torque data to identify commands provided by the user to the processing system, thereby instructing the processing system to switch the control mode to a predefined mode.

17. The processing system according to claim 15, wherein, The at least one robot control parameter controls the stiffness of the tool guide against the force applied in at least one direction.

Images