Automatic selection of collaborative robot control parameters based on tool and user interaction.

Abstract

The system includes a robotic arm having an instrument interface, a force / torque sensor for sensing forces at the instrument interface, a robot controller for controlling the robotic arm and controlling robot control parameters, and a system controller, wherein the system controller receives temporal force / torque data representative of forces at the instrument interface over time during a collaborative procedure with a user, analyzes the temporal force / torque data to determine a current user intention and / or collaborative procedure state, and causes the robot controller to control the robotic arm in a control mode predefined for the determined current user intention or collaborative procedure state, the control mode determining the robot control parameters.

Description

Technical Field

[0001] The present invention relates to robots, and more particularly to collaborative robots that can be used, for example, in an operating room, and methods of operating such collaborative robots.

Background Art

[0002] A collaborative robot is a robot that operates in the same space as a human and often interacts directly with the human, for example, via force control. Examples of such collaborative robots are robots that include an end effector for holding a tool or a tool guide for a tool, while the human operates the tool to achieve a task. Collaborative robots are generally considered to be safe and do not require special safety barriers. With the increasing acceptance of such collaborative robots, humans expect more intelligent and automatic behavior from these collaborative robots.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Collaborative robots should advance perception to provide intuitive assistance in protocol-heavy workflows such as operating rooms. However, compared to humans, robots have very poor context perception due to limited sensing modalities, quality, and bandwidth feedback. To truly make these robots collaborative, they require some ability to autonomously change their behavior based on the state of the environment, the current task, and / or the user's intent. Therefore, it is desirable to provide a collaborative robot and a method of operating a collaborative robot. In particular, it is desirable to provide such a collaborative robot and a method of operating a collaborative robot that can provide automatic selection of one or more robot control parameters based on the state of the environment, the task at hand, and / or the user's intent.

Means for Solving the Problems

[0004] ] In one aspect of the present invention, the system comprises a robot arm having control of one or more degrees of freedom, including an instrument interface; at least one force / torque sensor configured to sense forces at the instrument interface; a robot controller configured to control the robot arm to move the instrument interface to a determined position and to control at least one robot control parameter; and a system controller. The system controller receives temporal force / torque data, which represents forces at the instrument interface over time as sensed by at least one force / torque sensor during a collaborative procedure with a user; analyzes the temporal force / torque data to determine at least one of the user's current intention and the state of the collaborative procedure; and causes the robot controller to control the robot arm in a predefined control mode for the user's determined current intention or the state of the collaborative procedure, the control mode being configured to determine at least one robot control parameter.

[0005] In some embodiments, the instrument interface has a tool guide configured to interact with a tool that can be operated by the user during a collaborative procedure, and the forces applied to the instrument interface by the user include at least one of: (1) forces applied indirectly to the tool guide during user operation of the tool; (2) forces applied directly to the tool guide by the user; (3) forces from the robot's environment; and (4) forces generated by the tool.

[0006] In some embodiments, the system controller is configured to apply temporal force / torque data to a neural network to determine the user's current intention or the state of a collaborative action.

[0007] In some embodiments, the neural network is configured to determine from temporal force / torque data when the user is drilling with the tool, and further configured to determine from temporal force / torque data when the user is striking with the tool.

[0008] In some embodiments, at least one robot control parameter controls the stiffness of the tool guide against a force applied in at least one direction.

[0009] In some embodiments, when the neural network determines from temporal force / torque data that the user is hammering the tool, the neural network further determines whether the tool is hammering through bone or through tissue, and when it is determined that the tool is hammering through soft tissue, the control mode is a first stiffness mode that controls the tool guide to have a first stiffness, and when it is determined that the tool is hammering through bone, the control mode is a second stiffness mode that controls the tool guide to have a second stiffness, the second stiffness being less than the first stiffness.

[0010] In some embodiments, the system provides a warning to the user when the system changes its control mode.

[0011] In some embodiments, the system controller is further configured to receive auxiliary data having at least one of video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data, and is further configured to determine the user's current intention or the state of the collaborative procedure based on temporal force / torque data and auxiliary data.

[0012] In another aspect of the present invention, a method is provided for operating a robotic arm having control of one or more degrees of freedom, the robotic arm including an instrument interface. The method comprises the steps of: receiving temporal force / torque data, wherein the temporal force / torque data represents temporal force / torque data at the instrument interface, sensed by force / torque sensors during a collaborative procedure with a user; analyzing the temporal force / torque data to determine at least one of the user's current intention and the state of the collaborative procedure; and controlling the robotic arm in a predefined control mode for the determined user's current intention or state of the collaborative procedure, the control mode determining at least one robotic control parameter.

[0013] In some embodiments, the instrument interface has a tool guide configured to interface with a tool that can be operated by the user during collaborative treatment, and a force / torque sensor measures at least one of the following: (1) a force indirectly applied to the tool guide by the user during user operation of the tool, (2) a force directly applied to the tool guide by the user, (3) a force from the robot's environment, and (4) a force generated by the tool.

[0014] In some embodiments, analyzing temporal force / torque data to determine at least one of the user's current intent and the state of a collaborative action involves applying the temporal force / torque data to a neural network to determine the user's current intent or the state of a collaborative action.

[0015] In some embodiments, the neural network determines from temporal force / torque data when the user is drilling with the tool, and further determines from temporal force / torque data when the user is hammering with the tool.

[0016] In some embodiments, at least one robot control parameter controls a given stiffness of the tool guide against a force applied in at least one direction.

[0017] In some embodiments, when the neural network determines from temporal force / torque data that the user is hammering with the tool, the neural network further determines whether the tool is hammering through bone or through tissue, and when it is determined that the tool is hammering through tissue, the control mode is a first stiffness mode in which the tool guide has a first stiffness, and when it is determined that the tool is hammering through bone, the control mode is a second stiffness mode in which the tool guide has a second stiffness, the second stiffness being less than the first stiffness.

[0018] In some embodiments, the method further includes the step of providing a warning to the user when the control mode is changed.

[0019] In some embodiments, the method further comprises the steps of receiving auxiliary data having at least one of video data, image data, audio data, surgical planning data, diagnostic planning data, and robot vibration data, and determining the user's current intention or the state of the collaborative procedure based on temporal force / torque data and the auxiliary data.

[0020] In yet another aspect of the present invention, a processing system is provided for controlling a robotic arm having control of one or more degrees of freedom, the robotic arm including an instrument interface. The processing system includes a processor and a memory having instructions stored therein. When executed by the processor, an instruction causes the processor to receive temporal force / torque data, the temporal force / torque data representing the force over time at the instrument interface during a collaborative procedure with a user, to analyze the temporal force / torque data to determine at least one of the user's current intention and the state of the collaborative procedure, to control the robotic arm in a predefined control mode for the determined user's current intention or state of the collaborative procedure, the control mode setting at least one robot control parameter.

[0021] In some embodiments, the instrument interface has a tool guide configured to interface with a tool that can be operated by the user during collaborative treatment, and the force has at least one of the following: (1) a force indirectly applied to the tool guide by the user during user operation of the tool, (2) a force directly applied to the tool guide by the user, (3) a force from the robot's environment, and (4) a force generated by the tool.

[0022] In some embodiments, the instruction further causes the processor to analyze temporal force / torque data to identify commands provided to the system by the user in order to instruct the system to switch the control mode to a predetermined mode.

[0023] In some embodiments, at least one robot control parameter controls a given stiffness of the tool guide against a force applied in at least one direction. [Brief explanation of the drawing]

[0024] [Figure 1] This example shows a surgical operating room where a surgeon uses a collaborative robot to perform a simulated spinal fusion procedure. [Figure 2] Shows one exemplary embodiment of a collaborative robot tool guide with force sensing. [Figure 3] Shows an exemplary embodiment of a collaborative robot. [Figure 4] It is a block diagram showing an exemplary embodiment of a processor and related memory according to an embodiment of the present disclosure. [Figure 5] Shows force / torque profiles for hammering and drilling at different stages of a collaborative surgical intervention. [Figure 6] Shows an example of a configuration for classifying events during a collaborative procedure based on force / torque data and robot data mapping the detected robot state. [Figure 7] Shows a first exemplary embodiment of a control flow for automatically switching the control mode of a collaborative robot based on force / torque state detection by the collaborative robot. [Figure 8] Shows a second exemplary embodiment of a control flow for automatically switching the control mode of a collaborative robot based on force / torque state detection by the collaborative robot. [Figure 9] Shows a flowchart of an exemplary embodiment of a method for controlling a collaborative robot based on force / torque state detection by the collaborative robot.

Best Mode for Carrying Out the Invention

[0025] The present invention will now be described more fully hereinafter with reference to the accompanying drawings in which preferred embodiments of the invention are shown. However, the invention may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as illustrative examples of the teachings of the present invention.

[0026] In particular, various systems are described in the context of robot-guided surgery, such as spinal fusion surgery, in order to illustrate the principles of the present invention. However, it will be understood that this is for the purpose of illustrating specific examples of collaborative robots and methods for operating collaborative robots. More broadly, embodiments of collaborative robots and methods for operating collaborative robots as disclosed herein may be applied in a variety of other situations and settings. Accordingly, it should be understood that the present invention is defined by the claims and is not limited by the details of the specific embodiments described herein unless those details are described in the claims themselves.

[0027] Here, when something is said to be "approximately" or "about" a particular value, this means it is within 10% of that value.

[0028] Figure 1 shows an example of an operating room 100 in which a surgeon 10 performs a simulated robot-guided spinal fusion surgical procedure using a collaborative robot 110. Also shown in Figure 1 are a robot controller 120, a surgical navigation display 130, a camera 140, and a cone-beam computed tomography (CBCT) system that assist the surgeon 10 during the robot-guided spinal fusion surgical procedure. Here, the robot 110 is used to assist the surgeon 10 in the precise creation of holes inside the pedicles (sections of the vertebrae) along a planned trajectory. After holes have been created in multiple pedicles using a needle or drill, the surgeon 10 places screws inside these pilot holes and uses rods to fix adjacent screws in order to fuse multiple vertebrae in the desired configuration.

[0029] Currently, robot behavior or modes are manually changed by the robot user or another human assistant, which is inefficient in terms of total time and delay, and disrupts the workflow. In some cases, the human operator may not even be aware that the robot mode should be changed in time for useful purposes. Such changes may include, for example, changing robot compliance based on the type of task being performed, e.g., drilling versus hammering, or changing the safety zone (tool angle / position) based on the type of tissue the instrument is passing through.

[0030] Current approaches to managing this situation include threshold-based event / state detection, but these are not robust enough and specific to identify the complexity and severity of the signals associated with specific events or states of the procedure. The same applies to Fourier space analysis techniques. Furthermore, mode changes must be intuitive and transparent, and therefore need to communicate the type of behavior being selected.

[0031] To address some or all of these needs, the inventors have conceived of a collaborative robot and a method for controlling the collaborative robot that utilizes force sensing of tool interaction forces to automatically modify the robot's behavior based on the robot's state and relevant dynamic force information sensed by the tool interface.

[0032] Figure 2 shows one exemplary embodiment of a collaborative robot 110 and an associated tool guide 30 with force sensing. As shown in Figure 2, the collaborative robot 110 includes a robotic arm 111, and the instrument interface has a tool guide 30 located on the end effector 113 of the robotic arm 111. Here, the tool guide 30 may have a cylindrical shape, and a tool or instrument 20 (e.g., a drill, needle, etc.) having a handle 22 passes through an opening in the tool guide 30 used by a surgeon during a surgical procedure (e.g., spinal fusion surgery).

[0033] The collaborative robot 110 also includes a force / torque sensor 112 that senses forces applied by the user to or in the tool guide 30 during operation, forces indirectly applied by the surgeon 10 while manipulating the tool 20 in the tool guide 30 during a spinal fixation surgical procedure as shown in Figure 1, and / or forces that may be applied directly to the tool guide 30 by the user or surgeon 10 in the form of commands, as will be discussed in more detail below. In some cases, the force / torque sensor may also sense forces from the robot's environment and / or forces generated by the tool or instrument 20. An example of a suitable force / torque sensor 112 is the Nano25 force / torque sensor, a 6-axis transducer from ATI Industrial Automation.

[0034] Beneficially, the collaborative robot 110 can be directly controlled by a user (e.g., a surgeon 10) pushing the tool guide 30. The surgeon 10 can adjust the position of the collaborative robot 110 using hand-over-hand control (also known as “force control” or “admittance control”). The collaborative robot 110 can also function as a smart tool guide, precisely moving the cylindrical tool guide 40 to a planned position and orientation or pose for a planned trajectory, and holding that position while the surgeon 10 engages the instrument or tool 20 (e.g., a needle) inside the tool guide 30 with the pedicle by either hammering or drilling.

[0035] As will be described in more detail below, the admittance control method (using signals from the force / torque sensor 112) also allows for the adjustment of the compliance of the collaborative robot 110, more specifically the end effector 113 and the tool guide 30, independently in each degree of freedom (DOF), for example, to be very rigid in Cartesian rotation but compliant in Cartesian translation.

[0036] Figure 3 shows a more general exemplary embodiment of the collaborative robot 110.

[0037] The collaborative robot 110 includes a robot body 114 and a robot arm 111 extending from the robot body 114, and the instrument interface has a tool guide 30 held by an end effector 113 located at the end of the robot arm 111. The end effector 113 may have a gripping mechanism for gripping and holding the tool guide 30. Figure 3 shows a tool 20 having a handle 22 that passes through an opening in the cylindrical tool guide 30 and can be operated by a user (e.g., a surgeon) to perform a desired collaborative procedure.

[0038] The collaborative robot 110 also includes a robot controller 120 and a system controller 300. The robot controller 120 may have one or more processors, memory, actuators, motors, etc., for bringing the collaborative robot 110 to move, in particular, the movement and orientation of the instrument interface having a tool guide 30. As shown in Figure 3, the system controller 300 may have one or more processors 310 and associated memory 320.

[0039] In some embodiments, the robot controller 120 may be integrated with the robot body 140. In other embodiments, some or all components of the robot controller 120 may be provided separately from the robot body 140, for example, as a laptop computer or other device which may include a display and a graphical user interface. In some embodiments, the system controller 300 may be integrated with the robot body 140. In other embodiments, some or all components of the system controller 300 may be provided separately from the robot body 140. In some embodiments, one or more processors or memories of the system controller 300 may be shared with the robot controller 120. Many different divisions and configurations of the robot body 140, robot controller 120, and system controller 300 are envisioned.

[0040] The robot controller 120 and the system controller 300 are described in more detail below.

[0041] The robot arm 111 may have one or more joints, each having up to six degrees of freedom, for example, translation along any combination of mutually orthogonal x, y, and z axes, and rotation around the x, y, and z axes (also called yaw, pitch, and roll). On the other hand, some or all of the joints of the robot arm 111 may have fewer than six degrees of freedom. Movement of any or all of the joints in any degree of freedom may be performed in response to control signals provided by the robot controller 120. In some embodiments, the robot controller 120 may include motors, actuators, and / or other mechanisms for controlling one or more joints of the robot arm 111.

[0042] The collaborative robot 110 further includes a force / torque sensor 112 that senses forces applied to or in the instrument interface, for example, the force applied to the tool guide 30 by the tool 20 positioned within the tool guide 30 while the tool 20 is being manipulated by the surgeon 10 during a spinal fixation surgical procedure, as shown in Figure 1. In some embodiments, the collaborative robot may have multiple force / torque sensors 112.

[0043] The robot controller 120 may control the robot 110 in part in response to one or more control signals received from the system controller 300, as will be described in more detail below. The system controller 300 may then output one or more control signals to the robot controller 120 in response to one or more signals received from the force / torque sensor 112. In particular, the system controller 300 receives temporal force / torque data, which represents the force applied over time to or in the tool interface having the tool guide 30, and which is sensed by the force / torque sensor 112 during collaborative action by the user. As will be described below, the system 300 may be configured to interpret the signals from the force / torque sensor 112 to confirm the user's intent and / or command of the collaborative robot 110 and to control the collaborative robot 110 to operate according to the user's intent and / or command, as represented by the force / torque sensed by the force / torque sensor 112.

[0044] Figure 4 is a block diagram showing an exemplary embodiment of a processor 400 and associated memory 450 according to an embodiment of the present disclosure.

[0045] The processor 400 may be used to implement one or more processors described herein, for example, the processor 310 shown in Figure 3. The processor 400 may be any suitable processor type, including, but not limited to, a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA) programmed to form a processor, a graphical processing unit (GPU), an application-specific circuit (ASIC) designed to form a processor, or a combination thereof.

[0046] The processor 400 may include one or more cores 402. A core 402 may include one or more arithmetic logic units (ALUs) 404. In some embodiments, the core 402 may include, in addition to or instead of, a floating-point logic unit (FPLU) 406 and / or a digital signal processing unit (DSPU) 408.

[0047] The processor 400 may include one or more registers 412 that are communicatively coupled to the core 402. The registers 412 may be implemented using dedicated logic gate circuits (e.g., flip-flops) and / or any memory technology. In some embodiments, the registers 412 may be implemented using static memory. The registers 412 may provide data, instructions, and addresses to the core 402.

[0048] In some embodiments, the processor 400 may include one or more levels of cache memory 410 communicably coupled to the core 402. The cache memory 410 may provide computer-readable instructions to the core 402 for execution. The cache memory 410 may provide data for processing by the core 402. In some embodiments, computer-readable instructions may be provided to the cache memory 410 by local memory, for example, local memory attached to an external bus 416. The cache memory 410 may be implemented using any suitable cache memory type, such as metal-oxide-semiconductor (MOS) memory such as static random-access memory (SRAM), dynamic random-access memory (DRAM), and / or any other suitable memory technology.

[0049] The processor 400 may include a controller 414 that can control inputs to the processor 400 from other processors and / or components included in the system (e.g., the force / torque sensor 112 in Figure 3) and / or outputs from the processor 400 to other processors and / or components included in the system (e.g., the robot controller 120 in Figure 3). The controller 414 can control data paths in the ALU 404, FPLU 406, and / or DSPU 408. The controller 414 may be implemented as one or more state machines, data paths, and / or dedicated control logic. The gates of the controller 414 may be implemented as standalone gates, FPGAs, ASICs, or any other suitable technology.

[0050] The registers 412 and cache 410 can communicate with the controller 414 and core 402 via internal connections 420A, 420B, 420C, and 420D. The internal connections may be implemented as buses, multiplexers, crossbar switches, and / or any other suitable connection techniques.

[0051] Inputs and outputs for the processor 400 may be provided via a bus 416 which may include one or more conductive wires. The bus 416 may be communicatively coupled to one or more components of the processor 400, such as a controller 414, a cache 410, and / or registers 412. The bus 416 may be coupled to one or more components of the system, such as the robot controller 120 described above.

[0052] Bus 416 may be coupled to one or more external memories. The external memory may include read-only memory (ROM) 432. ROM 432 may be a mask ROM, an electronically programmable read-only memory (EPROM), or any other suitable technology. The external memory may include random access memory (RAM) 433. RAM 433 may be static RAM, a battery-backed static RAM, a dynamic RAM (DRAM), or any other suitable technology. The external memory may include electronically erasable programmable read-only memory (EEPROM) 435. The external memory may include flash memory 434. The external memory may include a magnetic storage device such as a disk 436. In some embodiments, the external memory may be included in a system such as a robot 110.

[0053] The collaborative robot 110, including the force / torque sensor 112, was used to measure force / torque (FT) in a static tool guide holding mode during pedicle drilling and hammering, a common but extremely difficult task in spinal immobilization.

[0054] Figure 5 shows the force / torque profiles 500 for hammering and puncture at different stages of collaborative surgical intervention.

[0055] Figure 5 shows a first force / torque trace 510 that expresses, as a function of time, the force applied to or in the instrument interface having the tool guide 30 when a surgeon is performing a hammering operation using a tool 20 positioned within the tool guide 20. The first torque trace 510 includes two distinct, recognizable temporal force / torque patterns, which include a first temporal pattern 512 corresponding to the force / torque applied to the instrument interface having the tool guide 30 during the hammering motion or process through soft tissue, and a second temporal pattern 514 corresponding to the force / torque applied to the instrument interface having the tool guide 30 during the hammering motion or process into bone. The second torque trace 520 shows a temporal pattern corresponding to the force / torque applied to the instrument interface having the tool guide 30 during the drilling motion or process into bone.

[0056] The force / torque trace in Figure 5 shows clear differences in the perceived or measured temporal patterns of force / torque applied to the instrument interface with the tool guide 30 between hammering and drilling, and even between different types of tissue in which the instrument or tool is interacting.

[0057] As will be described in more detail below, it is possible to configure the system controller 300 to recognize these different patterns of temporal force / torque data and thereby confirm the actions being performed by the user (e.g., surgeon 10). Temporal force / torque data may be supplemented by knowledge of the sequence of actions expected to be performed during a particular surgical procedure, for example, the sequence of actions expected to be performed by the surgeon during a robot-guided spinal fixation procedure. For example, a hammering action through tissue may be expected to be followed by hammering into bone and then drilling into bone. Such knowledge may be stored in memory associated with the system controller 300 and can be accessed by the system controller's processor while the system controller 300 controls the operation of the collaborative robot 110 during a collaborative surgical procedure.

[0058] One system and method for detecting the state of intervention (or user intent) during a collaborative procedure uses an iterative neural network to consider time-series force / torque measurement data along with the current state of the collaborative robot 110, such as the velocity of the collaborative robot 110, which is decomposed at the same location as the force / torque measurement data (e.g., decomposed in the tool guide 30). This type of network can be trained using data collected from multiple trials to improve its performance.

[0059] Figure 6 shows an example of a device for classifying events during collaborative procedures based on force / torque data and robot data that maps the detected robot state.

[0060] Figure 6 shows a neural network 600 that receives as input a robot state sequence 602 for the collaborative robot 110 and temporal force / torque data 604 representing the force applied to the instrument interface having a tool guide 30, for example, the force applied to the tool guide 30 by the tool 20 positioned within the tool guide 30 while the tool 20 is being manipulated over time by a user (e.g., surgeon 10) during a collaborative procedure. The robot state sequence 602 is a temporal sequence of robot states in which the collaborative robot 110 has operated up to the present, for example, during a collaborative procedure. In response to the temporal force / torque data 604 and the robot state sequence 602, the neural network 600 outputs the current robot state for the collaborative robot 110 from a set 610 of possible robot states for the collaborative robot 110.

[0061] Next, each of the possible robot states 610 corresponds to one or more control modes for the collaborative robot 110. For example, as shown in Figure 6, if the neural network 600 confirms that the current robot state of the collaborative robot 110 is "hammering soft tissue", it causes the collaborative robot 110 to operate in "mode enabling high stiffness control" 620A. In contrast, if the neural network 600 confirms that the current robot state of the collaborative robot 110 is "hammering in bone", it causes the collaborative robot 110 to operate in "mode enabling low stiffness control" 620B. In some cases, relative stiffness may be reversed to maintain a planned trajectory regardless of the geometric shape of the anatomical structure.

[0062] In some embodiments, the neural network 600 may be implemented in the system controller 300 by a processor 310 that executes a computer program defined by instructions stored in memory 320. Other embodiments are also possible, implemented by various combinations of hardware and / or firmware and / or software.

[0063] In the example in Figure 6, at any given time, the collaborative robot 110 may operate in any one of six defined robot states, including soft tissue hammering, intra-bone hammering, cortical bone perforation, cancellous bone perforation, unknown, and no activity. Depending on the collaborative procedure in which the collaborative robot 110 is engaged, other robot states 610 are possible, such as moving, skive detection, retraction, inserting an instrument into a tool guide, and gesture detection (indicating that specific user control is desired).

[0064] Beneficially, each robot state 610 can be defined as a Cartesian velocity decomposed at the instrument interface having a tool guide 30 (at the end of the end effector 113). The velocity can be calculated from the joint encoder value, a forward kinematic model for the collaborative robot 110, and the robot's Jacobian, which relates the joint velocity to the linear and angular velocities of the end effector 113.

[0065] The robot state 610 may also include, for example, speed from an external tracking system (optical tracker, electromagnetic tracker), motor torque for the robot arm 111, type of tool 20 used, tool status (drill on or off, position tracking, etc.), data from an accelerometer on the tool 20 or the collaborative robot 110, etc.

[0066] Force / torque data may represent force / torque in one or more degrees of freedom. Generally, force / torque can be decomposed at one of several different convenient locations. Beneficially, force / torque can be decomposed at the tool guide 30 at the end of the end effector 113. Generally, temporal force / torque data 604 does not need to be preprocessed beyond basic noise reduction and decomposition of force / torque at a specific location (e.g., the tool guide 30 at the end of the end effector 113).

[0067] Beneficial robot state detection methods may use a data-driven model (e.g., via a neural network 600) to sequentially classify robot states of treatment or intervention (also known as classes) based on short-term data output from a force / torque sensor 112 (i.e., temporal force / torque data 604) and robot states 610. Many potential model architectures exist that can be used to classify time-series data with multiple inputs. A beneficial model is the Long-Short-Term Memory (LSTM) network, as it is more stable than a typical recurrent neural network (RNN). Other examples of possible networks include echo-state networks, convolutional neural networks (CNNs), convolutional LSTMs, and handcrafted networks using multiplayer perceptrons, decision trees, logistic regression, etc.

[0068] Beneficial, the data stream input to the neural network 600 (which may include force / torque data (604), robot state (602), current control mode (see Figure 7 below), etc.) is preprocessed (interpolated, downsampled, upsampled, etc.) so that each data point has the same period (typically the period of the highest frequency data stream, which is time data from the force / torque sensor 112 (e.g., 1 kHz, or 1 ms period)). In some embodiments, the temporal slide window for the neural network 600 may be set to a length of about 3 seconds to capture typical events such as drilling, reaming, pushing, pulling, screwing, and hammering (e.g., hammering has a typical interval of about 1 second), while being short enough to respond to a given task. Each window shift (advance) is a new sample to be classified, and in the training phase, it has an associated robot state label. Windows with smaller sizes may be discarded.

[0069] In some embodiments, a single input sample may contain 12 features for each of N time steps (e.g., 36K individual features for 3 seconds at 1kHz sampling), i.e., 3 for force, 3 for torque, 3 for XYZ robot linear velocity, and 3 for robot angular velocity. In some embodiments, the output of the LSTM is the probability of each robot state for a given input window sample. In some embodiments, the model has two LSTM hidden layers, followed by a dropout layer (to reduce overfitting), followed by a dense fully connected layer with a common rectified linear unit ("ReLU") activation function, and an output layer with a normalized exponential function ("softmax") activation. LSTM and ReLU are common deep learning model components, as will be understood by those skilled in the art. The loss function is categorical cross-entropy, and the model may be optimized using the Adaptive Learning Rate Optimization Algorithm (Adam) optimizer. The Adam optimizer is a widely used optimizer for deep learning models, as described, for example, in Diederik P. Kingma et al., "Adam: A method for stochastic optimization," 3rd International Conference on Learning Representations, (San Diego, 2015).

[0070] Beneficial use in training care is to balance examples of all expected different robot states, particularly those rarely experienced (e.g., puncture) compared to the most common robot state (e.g., inactive). This reduces bias towards common robot states. Techniques may include undersampling the most common robot states and / or oversampling rare robot states in the training sequence. In addition, cost-based classifiers are used to penalize inaccurate classification of robot states of interest while reducing the cost of accurately classifying common robot states. This is particularly useful when rare events occur within a time window (e.g., three inactive hammer strokes versus three consecutive punctures).

[0071] Figures 7 and 8 show two different examples of control mode switching algorithms for collaborative robots, such as collaborative robot 110.

[0072] Figure 7 shows a first exemplary embodiment of a control flow 700 for automatically switching the control mode of the collaborative robot 110 based on force / torque state detection by the collaborative robot 110. The control flow 700 may be implemented by the system controller 300, more specifically by the processor 310 of the system controller 300. The control flow 700 uses a single model (and corresponding single neural network 600) along with the current mode input 606 to detect the robot state 610 during the current control mode 620. The neural network 600 implicitly takes into account the current control mode context in robot state detection.

[0073] First, in operation 702, the system controller 300 selects a start control mode, either in response to direct input from a user (e.g., surgeon 10) or as a pre-programmed initial control mode for the collaborative robot 110 which may be determined for a specific collaborative procedure.

[0074] In operation 704, the system controller 300 initially sets the current control mode 706 for the collaborative robot 110 as the start control mode. The system controller 300 may provide one or more signals to the robot controller 120 to indicate the current control mode 706 and / or to cause the robot controller 120 to control the collaborative robot 110, specifically the robot arm 111, in accordance with the current control mode 706. By setting the current mode, in the example described above, the system controller 300 may control one or more robot control parameters, including, for example, controlling the amount of a given stiffness of the tool guide 30 against a force applied in one or more of the up to six degrees of freedom (e.g., at least one direction).

[0075] In some embodiments, the system controller 300 may control other robot control parameters in the tool guide 30 besides the given stiffness, such as movement limits (trajectory constraints), dwell time (at a specific location), acceleration and vibration of the robot arm 111, drilling speed (on / off), maximum speed and minimum speed of the tool 20.

[0076] The control flow 700 uses the robot state detection network 750 to confirm or detect the robot state 610 of the collaborative robot 110. The state detection network 750 includes a neural network 600 that, as described above, receives the robot state sequence 602 for the collaborative robot 110, temporal force / torque data 604, and the current control mode 606 as inputs, and in response selects a robot state 610 from among several possible robot states for the collaborative robot 110.

[0077] Operation 712 maps the detected robot state 610, detected by the robot state detection network 750, to a mapped control mode 620 for the collaborative robot 110.

[0078] Operation 714 determines whether the mapped control mode 620 is the same as the current control mode 706 for the collaborative robot 110. If so, the current control mode 706 remains the same. Otherwise, the current control mode 706 should be changed or switched to the mapped control mode 620.

[0079] In some embodiments, during operation 716, the system controller 300 may alert the user that the system controller 300 has a pending control mode switching request. The system controller 300 may require the user (e.g., surgeon 10) to confirm or approve the control mode switching request. In some embodiments, operation 716 may be performed only for certain procedures, but may be skipped for other procedures.

[0080] In some embodiments, the control mode switching request of operation 716 may be presented to the user via a user interface associated with the system controller 300, for example, visually via a display device (e.g., display device 130) or aurally (e.g., verbally) via a speaker or the like.

[0081] In an embodiment or procedure in which the control mode switch request of operation 716 is performed, in operation 718, the system controller determines whether the user (e.g., surgeon 10) acknowledges or approves the control mode switch request. The user or surgeon may acknowledge or approve (or conversely, reject or deny) the control mode switch request in any of the following ways. Examples include: The user or surgeon may respond by clicking a pedal or a button on the user interface to confirm the change in control mode. • Voice recognition may be used to confirm user approval or acceptance of changes in control modes. The user may acknowledge or accept a change in control mode via hand / body gestures, which may be detected using a visual or depth-tracking camera and provided as supplemental or auxiliary data input to the system controller 300.

[0082] In some cases, switching control modes without confirmation may be acceptable, and these cases may be mixed with some cases that require user input. Therefore, operations 716 and 718 may be optional. In these cases, a simple audible effect indicating which mode has been entered to the user or surgeon may suffice. The user or surgeon may then cancel or stop the robot's movement if this is not the desired mode. Beneficially, the current detected robot state and control mode may be clearly communicated to the user or surgeon using audiovisual means such as a digital display, LED lights on the robot, or audio feedback describing the system as it senses and changes.

[0083] If the control mode switching request is not approved, the current control mode 706 is maintained.

[0084] On the other hand, if the control mode switching request is approved or operations 716 and 718 are omitted, operation 704 is repeated to set mode 620, which is mapped as the new current control mode 706 for the collaborative robot 110. The new current control mode 706 is provided to input 606 of the neural network 600 and provided as one or more output signals to the robot controller 120.

[0085] Figure 8 shows a second exemplary embodiment of a control flow 800 for automatically switching the control mode of a collaborative robot based on force / torque state detection by the collaborative robot. The control flow 800 may be implemented by the system controller 300, more specifically by the processor 310 of the system controller 300.

[0086] For the sake of brevity, the description of the operation and flow path in control mode 800, which is the same as that in control mode 700, will not be repeated.

[0087] In contrast to control flow 700, control flow 800 employs multiple robot state detection networks 850A, 850B, 850C, etc., one for each control mode selected in response to a control mode switching event, in order to detect the robot state. Each of the robot state detection networks 850A, 850B, 850C, etc., executes a corresponding model for robot state detection and outputs the corresponding detected robot state. In control flow 800, the detected robot state output from one of the multiple models (and its corresponding neural network 600) is explicitly selected for each control mode in operation 855.

[0088] In some embodiments, a handcrafted state machine layer may be added to prevent false positives and negatives, add time filters, and consider treatment plans or long-term state transitions. For example, in the case of pedicle perforation, the physician is unlikely to hammer the drill bit after the perforation has been performed, and a higher level of state machine may be included in the control flow for the detection of this inconsistency. Errors indicating that the collaborative robot 110 is not being used properly or that the procedure is not being followed may be communicated.

[0089] Figure 9 shows a flowchart of an exemplary embodiment of a method 900 for controlling a collaborative robot (e.g., collaborative robot 110) based on force / torque state detection by the collaborative robot 110 during treatment or intervention.

[0090] In operation 910, the user (e.g., surgeon 10) operates an instrument interface (e.g., tool guide 30) or an instrument or tool (e.g., tool 20) that applies force to another part of the robot arm 111 during a collaborative procedure (e.g., spinal fusion surgical procedure).

[0091] In operation 920, the force / torque sensor 112 senses the force applied to the tool interface or other part of the robot arm 111, for example, in the tool guide 30.

[0092] In operation 930, the processor 310 of the system controller 300 receives temporal force / torque data 604 generated from the force / torque sensor 112.

[0093] In operation 940, the processor 310 analyzes the temporal force / torque data 604 to determine the user's current intention and / or one or more robot states during the collaborative action.

[0094] In operation 950, the system controller 400 determines the control mode of the collaborative robot 110 based on the user's current intention and / or the current robot state and / or past robot states during collaborative action.

[0095] In operation 960, the system controller 300 notifies the user of the determined control mode to be set for the collaborative robot and waits for user confirmation before setting or changing the current control mode to the determined control mode.

[0096] In operation 970, the system controller 300 sets the control mode of the collaborative robot 110.

[0097] In operation 980, the system controller 300 sets one or more robot control parameters based on the current control mode. One or more robot control parameters may, for example, control the amount of stiffness given to the tool guide 30 in one or more of the up to six degrees of freedom. In some embodiments, the system controller 300 may control other operational parameters other than the stiffness given to the tool guide 30, such as movement restrictions (trajectory constraints) and dwell time (at a particular location).

[0098] Many variations of the above-described embodiments are conceivable.

[0099] For example, in the basic case described above, the force / torque sensor 112 is positioned between the robot body 114 and the tool guide 30. However, in some embodiments, the force / torque sensor 112 may be positioned near the tool guide 30 or integrated into the robot body 114. Beneficially, six-degree-of-freedom force / torque sensing technology may be used. Torque measurements on the joints of the robot arm 111 can also provide basic information regarding force / torque decomposed in the tool guide 30. The force may be decomposed in the tool guide 30 or at the estimated or measured position of the tip of the tool 20.

[0100] In some robot / sensor configurations, the system controller 300 may identify the user's applied input force / torque from forces / torques applied to the instrument by the environment (e.g., a force / torque sensor integrated into the instrument tip and another force / torque sensor on the tool guide). For example, environmental forces (e.g., tissue pushing the tool) can be a primary source of feedback information to the data model to determine whether the tool is passing through soft tissue or bone. That is, environmental forces include the results of anatomical structures responding to stimuli provided by the user and the robot via the tool.

[0101] In some embodiments, the system controller 300 may consider different inputs for detecting the robot state during collaborative action or intervention. Examples of such inputs include: • Frequency range of force / torque data • Frequency domain of velocity / acceleration data • Current robot status • Estimated position relative to the target • Types of treatment • Estimated bone type • Estimated tissue type at the instrument tip (from navigation) • Robot rigidity • Robot control mode • Computed tomography data • Magnetic resonance imaging data Includes.

[0102] Each of these data inputs has a different behavior and can be considered depending on the desired focus of the collaborative robot 110.

[0103] In some embodiments, the system controller 300 may receive supplementary or auxiliary data inputs to help identify the context (search space) in order to improve robot state detection. Such data may include one or more of the following: video data, diagnostic data, image data, audio data, surgical planning data, time data, robot vibration data, etc. The system controller 300 may be configured to determine the user's (surgeon's) current intent or the state of the collaborative procedure based on the temporal force / torque data and auxiliary data.

[0104] In some embodiments, the user (e.g., surgeon 10) may also apply force / torque to the collaborative robot 110 in a very specific way to engage a particular control mode. For example, the collaborative robot 110's system controller 300 may be configured to recognize when the user applies a circular force to the tool guide 30 (either through an instrument or tool 209 in the tool guide 20, or by applying force directly to the tool guide 30), and in response, the system controller 300 may place the collaborative robot 110 into a standard force control mode (e.g., an admittance controller that allows the operator to move the robot by applying force in a desired direction). In other words, the system controller 300's processor may be configured to analyze temporal force / torque data 604 to identify commands provided to the system controller 300 by the user to instruct the system controller 300 to switch the control mode for the collaborative robot 110 to a predetermined control mode. Some other examples of specific pressure actions by the user that may be interpreted as control mode commands may include: • User or surgeon presses down and up three times - Translation-only mode. - The user performs a circular motion by pressing upwards twice - Insertion only mode. • The user applies pressure in a specific sequence (e.g., left, right, up, down) - then selects the next planned trajectory.

[0105] Many other examples of specific commands for the corresponding control modes may also be used.

[0106] In some embodiments, the force / torque sensing described above may also be supplemented or replaced by vibration sensing of the robot itself (e.g., via an accelerometer). Events such as hammering and drilling are detected away from the robot tool effector and induce vibrations within the robot structure, which can be used in the same manner as described above.

[0107] Various embodiments may be combinations of the modifications described above.

[0108] While preferred embodiments are disclosed in detail herein, many other modifications are possible that remain within the concept and scope of the present invention. Such modifications will become apparent to those skilled in the art after examining this specification, the drawings, and the claims. Therefore, the present invention should not be limited beyond the scope of the appended claims.

Claims

1. A robotic arm having control of one or more degrees of freedom, wherein the robotic arm includes a device interface, The instrument interface includes at least one force and / or torque sensor configured to sense force and / or torque, A robot controller configured to control the robot arm to move the instrument interface to a determined position and to control at least one robot control parameter, The system receives temporal force and / or torque data, which represents the force and / or torque at the instrument interface over time, as sensed by at least one force and / or torque sensor during a collaborative procedure in which the instrument interface is operated by the user. The temporal pattern of the temporal force and / or torque data is analyzed in order to determine at least one of the user's current intentions and the type of task being performed. The robot controller is instructed to control the robot arm in a predetermined control mode for the user's current intent or the type of task being performed. A system controller configured as follows, A system that has

2. The system according to claim 1, wherein the instrument interface has a tool guide configured to interface with a tool that can be operated by the user during the collaborative procedure, and the force and / or torque comprises at least one of (1) a force and / or torque indirectly applied to the tool guide during user operation of the tool, (2) a force and / or torque directly applied to the tool guide by the user, (3) a force and / or torque from the environment of the robot arm, and (4) a force and / or torque generated by the tool.

3. The system according to claim 2, wherein the system controller is configured to apply the temporal force and / or torque data to a neural network to determine the user's current intention or the type of task being performed.

4. The system according to claim 3, wherein the neural network is configured to determine from the temporal force and / or torque data when the user is drilling with the tool, and is further configured to determine from the temporal force and / or torque data when the user is hammering with the tool.

5. The system according to claim 4, wherein the at least one robot control parameter controls the given stiffness of the tool guide with respect to the force applied in at least one direction.

6. When the neural network determines from the temporal force and / or torque data that the user is hammering with the tool, the neural network further determines whether the tool is hammering through bone or through soft tissue, and when it is determined that the tool is hammering through soft tissue, the control mode is a first stiffness mode in which the robot controller controls the tool guide to have a first stiffness, and when it is determined that the tool is hammering through bone, the control mode is a second stiffness mode in which the robot controller controls the tool guide to have a second stiffness, the second stiffness being less than the first stiffness, the system according to claim 5.

7. The system according to claim 1, wherein the system provides a warning to the user when the system changes the control mode.

8. The system according to claim 1, wherein the system controller is further configured to receive auxiliary data having at least one of video data, image data, audio data, surgical plan data, diagnostic plan data, and robot vibration data, and is further configured to determine the user's current intention or the type of task being performed based on the temporal force and / or torque data and the auxiliary data.

9. In a method for operating a system that operates a robot arm having one or more degrees of freedom, the robot arm includes a device interface, and the method is The system controller of the system receives temporal force and / or torque data representing the force and / or torque in the instrument interface over time, which is sensed by at least one force and / or torque sensor during a collaborative procedure in which the instrument interface is operated by a user. The system controller analyzes the temporal pattern of the temporal force and / or torque data to determine at least one of the user's current intentions and the type of task being performed. The system controller controls the robot arm in a predetermined control mode for the determined current intent of the user or the type of task being performed, wherein the control mode determines at least one robot control parameter. A method having

10. The apparatus according to claim 9, wherein the instrument interface has a tool guide configured to interface with a tool that can be operated by the user during the collaborative procedure, and the force and / or torque sensor measures at least one of: (1) a force and / or torque indirectly applied to the tool guide by the user during user operation of the tool; (2) a force and / or torque directly applied to the tool guide by the user; (3) a force and / or torque from the environment of the robot; and (4) a force and / or torque generated by the tool.

11. The method according to claim 10, wherein analyzing the temporal pattern of the temporal force and / or torque data to determine at least one of the user's current intention and the type of task being performed includes applying the temporal force and / or torque data to a neural network to determine the user's current intention or the type of task being performed.

12. The method according to claim 11, wherein the neural network determines from the temporal force and / or torque data when the user is drilling with the tool, and further determines from the temporal force and / or torque data when the user is hammering with the tool.

13. The method according to claim 12, wherein the at least one robot control parameter controls the given stiffness of the tool guide with respect to a force applied in at least one direction.

14. The method according to claim 13, wherein when the neural network determines from the temporal force and / or torque data that the user is hammering with the tool, the neural network further determines whether the tool is hammering through bone or through soft tissue, and when it is determined that the tool is hammering through soft tissue, the control mode is a first stiffness mode in which the tool guide has a first stiffness, and when it is determined that the tool is hammering through bone, the control mode is a second stiffness mode in which the tool guide has a second stiffness, the second stiffness being less than the first stiffness.

15. The method according to claim 9, further comprising the step of providing a warning to the user when the control mode is changed.

16. The steps include receiving auxiliary data having at least one of video data, image data, audio data, surgical planning data, diagnostic planning data, and robot vibration data, A step of determining the user's current intention or the type of task being performed based on the aforementioned temporal force and / or torque data and the aforementioned auxiliary data, The method according to claim 9, further comprising:

17. A processing system for controlling a robot arm having control of one or more degrees of freedom, wherein the robot arm includes an instrument interface, and the processing system is Processor and The memory that stores the instructions, The instruction, when executed by the processor, causes the processor to The device receives temporal force and / or torque data, which represents the force and / or torque at the device interface over time during a collaborative procedure in which the device interface is operated by the user. The temporal pattern of the temporal force and / or torque data is analyzed to determine at least one of the user's current intentions and the type of task being performed. The robot arm is controlled in a predetermined control mode for the determined current intent of the user or the type of task being performed, the control mode setting at least one robot control parameter. Processing system.

18. The system according to claim 17, wherein the instrument interface has a tool guide configured to interface with a tool that can be operated by the user during the collaborative procedure, and the force and / or torque comprises at least one of (1) a force indirectly applied to the tool guide by the user during user operation of the tool, (2) a force directly applied to the tool guide by the user, (3) a force from the environment of the robot arm, and (4) a force generated by the tool.

19. The system according to claim 18, wherein the instruction further causes the processor to analyze the temporal pattern of the temporal force and / or torque data to identify a command provided to the system by the user in order to instruct the system to switch the control mode to a predetermined mode.

20. The system according to claim 18, wherein the at least one robot control parameter controls a given stiffness of the tool guide with respect to the force applied in at least one direction.

Images