Techniques for estimating deflection of surgical robot arm

Abstract

Surgical systems and methods include a robotic arm including a plurality of links and joints; and a surgical tool supported by the robotic arm and movable by the robotic arm, and configured to interact with an anatomical structure. A controller coupled to the robotic arm is configured to estimate a robotic arm deflection based on the tool interaction and / or characterize the tool-anatomical interaction based on the estimated arm deflection. To estimate the robotic arm deflection, the controller inputs a pose of the surgical tool and a tool interaction force to a machine learning model. To characterize the interaction of the surgical tool with the anatomical structure, the controller inputs operating parameters of the surgical tool and the estimated deflection into a machine learning model.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and all benefits to U.S. Provisional Patent Application No. 63 / 700,968, filed September 30, 2024, and U.S. Provisional Patent Application No. 63 / 749,842, filed January 27, 2025, the entire contents of each of which are incorporated herein by reference. Background Technology

[0003] Robotic surgical systems for performing surgical procedures are well-known and typically consist of a surgical robot and surgical instruments attached to the surgical robot. In orthopedic surgery, the surgical instruments are usually controlled to remove tissue from the surgical site.

[0004] To ensure the accurate operation of surgical robots during procedures, one or more robot calibration procedures are typically performed. Calibration procedures involve identifying specific parameters of the surgical robot that may affect its accuracy.

[0005] Various levels of robot calibration exist to improve the accuracy of surgical robots. For example, Level 1 calibration identifies the difference between actual and expected joint displacements. A higher level of calibration, Level 2, involves kinematic calibration. Level 2 calibration is used to detect "geometric errors," such as those caused by link lengths, non-parallel axes, base misalignment, part tolerances, and manufacturing and assembly errors. Kinematic calibration addresses geometric errors by identifying the geometric parameters of the robot's kinematic structure to determine, for example, angular offsets and joint lengths.

[0006] While Level 1 and Level 2 calibrations are sufficient to address most errors in robotic surgical applications, they cannot address non-geometric errors, which cause the robotic arm to deflect in response to forces acting on the tool and / or the robotic arm. Non-geometric errors can arise from payload, compliance / deflection, thermal deformation, and gear backlash. These errors can affect the accuracy of surgical instruments. For example, in robot-assisted orthopedic surgery, a large cutting force is applied during bone resection. This cutting force is typically dependent on the bone density of the patient's anatomy. In this case, joint / link compliance / deflection (a non-geometric parameter) plays a significant role in cutting accuracy. For instance, a robotic manipulator with 40 micrometers / N compliance can introduce a 2-mm error with a typical 50N cutting force simply because of its compliance. However, Level 1 and Level 2 calibrations are not suitable for addressing this type of error. Furthermore, the robot's compliance depends on dynamic factors such as pose and external forces and is difficult to model mathematically.

[0007] Furthermore, another aspect that can affect the accuracy of tools during surgery is the interaction between the tool and anatomical structures. Typically, the tool can withstand excessive forces while performing surgical tasks such as cutting. Conventional surgical systems usually obtain basic measurements related to the tool, such as force or temperature, and implement simple feedback loops to control the tool and reduce the force exerted by the tool on the anatomical structures. Conventional surgical robotic systems are not suited to a high level of understanding of the tool-anatomical interaction. For example, such systems cannot predict the level of workload or difficulty of the tool, or why the tool experiences such excessive workload or difficulty (e.g., due to dense tissue interactions). This lack of a deeper understanding of the tool-anatomical interaction can lead to inaccuracies and inefficiencies in the procedure. Moreover, when tool-anatomical forces are excessive, the non-geometric errors discussed above can exacerbate the errors.

[0008] Therefore, techniques are needed to estimate the impact of non-geometric errors on the accuracy of robotic guidance tools used in surgical applications. Techniques also need to be developed to classify / characterize the interactions between robotic guidance tools and anatomical structures. Summary of the Invention

[0009] The present invention is presented in a simplified form, with the following detailed description of selected concepts. This summary is not intended to limit the scope of the claimed subject matter, nor is it intended to identify key or essential features of the claimed subject matter.

[0010] According to a first aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect interaction forces applied to the surgical instrument based on the interaction between the surgical instrument and the anatomical structure; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical instrument based on kinematic data from the robotic arm; obtain interaction forces from the sensing system; input the pose of the surgical instrument and the interaction forces into a machine learning model; estimate a deflection of the robotic arm based on the output of the machine learning model; and optionally control the robotic arm and / or the surgical instrument based on the estimated deflection.

[0011] According to a second aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect: interaction forces applied to the surgical instrument based on the interaction between the surgical instrument and the anatomical structure; disturbance forces applied to one or more links in the links; and user-applied forces on the surgical instrument; and one or more controllers coupled to the robotic arm and the sensing system, and configured to: obtain the pose of the surgical instrument based on kinematic data from the robotic arm; obtain the interaction forces, disturbance forces, and user-applied forces from the sensing system; input the pose of the surgical instrument, interaction forces, disturbance forces, and user-applied forces to a machine learning model; and estimate the deflection of the robotic arm based on the output of the machine learning model.

[0012] According to a third aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm; a sensing system configured to detect disturbance forces applied to one or more links; and one or more controllers coupled to the robotic arm and the sensing system, and configured to: obtain the pose of the surgical instrument based on kinematic data from the robotic arm; obtain the disturbance forces from the sensing system; input the pose of the surgical instrument and the disturbance forces into a machine learning model; and estimate the deflection of the robotic arm based on the output of the machine learning model.

[0013] According to a fourth aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm; a sensing system configured to detect forces applied by a user on the surgical instrument; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain the pose of the surgical instrument based on kinematic data from the robotic arm; obtain the forces applied by the user from the sensing system; input the pose of the surgical instrument and the forces applied by the user into a machine learning model; and estimate the deflection of the robotic arm based on the output of the machine learning model.

[0014] According to a fifth aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect interaction forces applied to the surgical instrument based on the interaction between the surgical instrument and the anatomical structure; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain the pose of the surgical instrument based on kinematic data from the robotic arm; obtain the interaction forces from the sensing system; input the pose of the surgical instrument and the interaction forces into a machine learning model; and estimate the deflection of the robotic arm based on the output of the machine learning model.

[0015] According to a sixth aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with anatomical structures; a sensing system configured to detect one or more forces applied to the surgical instrument and / or the robotic arm; and one or more controllers coupled to the robotic arm and the sensing system and configured to: detect the presence of the one or more forces to trigger a deflection monitoring mode; and in the deflection monitoring mode, use a machine learning model to estimate the deflection of the robotic arm.

[0016] According to a seventh aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with an anatomical structure; and one or more controllers coupled to the robotic arm and configured to: acquire operating parameters of the surgical instrument during the interaction between the surgical instrument and the anatomical structure; estimate deflection of the robotic arm caused by the interaction between the surgical instrument and the anatomical structure; input the operating parameters of the surgical instrument and the estimated deflection into a machine learning model; and characterize the interaction between the surgical instrument and the anatomical structure based on the output of the machine learning model.

[0017] According to an eighth aspect, a surgical system is provided, comprising: a robotic arm including a plurality of links and joints; a surgical instrument supported by and movable by the robotic arm and configured to interact with an anatomical structure; and one or more controllers coupled to the robotic arm and configured to: estimate a deflection of the robotic arm caused by the interaction between the surgical instrument and the anatomical structure; input the estimated deflection into a machine learning model; and characterize the interaction between the surgical instrument and the anatomical structure based on the output of the machine learning model.

[0018] Also provided are: a computer-implemented method for operating a surgical system of any of the foregoing aspects; a non-transitory computer-readable medium or computer program product comprising instructions configured, when executed by one or more processors, to perform features of one or more controllers described in any of the foregoing aspects; and a non-transitory computer-readable medium or computer program product comprising instructions configured, when executed by one or more processors, to train or implement any of the machine learning models in the machine learning models of any of the foregoing aspects.

[0019] Any of the above aspects can be combined in whole or in part.

[0020] Any aspect of the foregoing can be combined, in whole or in part, with any implementation of the following embodiments.

[0021] The controller can be configured to control the robotic arm and / or surgical tool based on the estimated deflection by performing the following actions: modifying the tool path; modifying the feed rate; adjusting the command pose of the surgical tool; adjusting the cutting rate of the tool; adjusting the command pose of the surgical tool; aborting the movement of the surgical tool; and / or moving the surgical tool away from the anatomical structure; or any combination thereof. The controller can input the operating parameters of the surgical tool (such as velocity) and the estimated deflection into a second machine learning model and characterize the interaction between the surgical tool and the anatomical structure based on the output of the second machine learning model. The controller can characterize the type of anatomical structure, the workload of the surgical tool, the density of the bone, or a combination thereof. The controller can control the robotic arm and / or surgical tool based on the characterized interaction by performing one or more of the following actions: modifying the tool path of the surgical tool; modifying the feed rate of the surgical tool; modifying the cutting rate / direction of the surgical tool; adjusting the command pose of the surgical tool; aborting the movement of the surgical tool; and / or moving the surgical tool away from the anatomical structure. Monitoring of deflection can occur in response to the detection of specific system conditions (such as the detection of a force applied to the tool or manipulator). The machine learning model can be a deep learning model (such as a deep learning model including at least two hidden layers). The estimated deflection can be a non-geometric deflection. The operating parameters of the surgical tool can be the speed of the surgical tool, the average speed of the surgical tool, the acceleration of the surgical tool, the displacement of the surgical tool, the motor current of the surgical tool, etc. The robotic arm can operate in manual mode, semi-automatic mode, automatic mode, or manually guided mode. Attached Figure Description

[0022] The advantages of this disclosure will be readily apparent, as they will be better understood by referring to the following specific embodiments considered in conjunction with the accompanying drawings.

[0023] Figure 1 This is a perspective view of a robotic surgical system according to one embodiment.

[0024] Figure 2 This is a block diagram of an example control system for controlling a robotic surgical system according to one embodiment.

[0025] Figure 3 It is a functional block diagram of a module implemented by a control system according to one embodiment.

[0026] Figure 4 Example output of the boundary generator is shown.

[0027] Figure 5 Example output of the path generator is shown.

[0028] Figure 6 This is a flowchart illustrating a technique or method, according to one embodiment, for estimating the deflection of a robotic arm caused by tool-anatomical interactions using a machine learning model.

[0029] Figure 7 This is a flowchart illustrating a technique or method, according to one embodiment, for estimating the “total” deflection of a robotic arm caused by one or more force sources by using a machine learning model.

[0030] Figure 8 This is a flowchart illustrating a technique or method for characterizing tool-anatomical interactions using a machine learning model, according to one embodiment.

[0031] Figure 9 This is an example Figure 8 The tool-anatomical structure interaction characterization example output graph, where various interaction predictions are grouped into categories of bone, soft tissue, or free zone interactions.

[0032] Figure 10 This is an example Figure 8 Another example of the tool-anatomical structure interaction characterization output is a graph where various interaction predictions are grouped into tissue density categories. Detailed Implementation

[0033] I. Overview of the Example System

[0034] refer to Figure 1 Examples of surgical systems include robotic surgical system 10. System 10 is used to treat surgical sites or anatomical volumes (A) of a patient 12, such as bone or soft tissue. Figure 1 In the middle, patient 12 is undergoing surgical procedures. Figure 1The anatomical structures included are the femur (F) and tibia (T) of patient 12. The surgical procedure may involve tissue removal or other forms of treatment. Treatment may include tissue cutting, coagulation, lesion ablation, other in situ tissue treatments, etc. In some examples, the surgical procedure involves partial or total knee or hip replacement surgery, shoulder replacement surgery, spinal surgery, craniotomy, or ankle surgery. In some examples, system 10 is designed to remove material to be replaced by a surgical implant, such as hip and knee implants, including single-chamber, double-chamber, multi-chamber, or total knee implants. Some of these types of implants are illustrated in U.S. Patent Application Publication 2012 / 0330429 entitled “Prosthetic Implant and Method of Implantation,” the disclosure of which is hereby incorporated herein by reference. The system 10 and techniques disclosed herein may be used to perform other surgical or non-surgical procedures, or may be used for industrial applications or other applications utilizing robotic systems.

[0035] System 10 includes a robot manipulator 14. The manipulator 14 has a base 16 and a plurality of links 18. A manipulator trolley 17 supports the manipulator 14, such that the manipulator 14 is fixed to the manipulator trolley 17. The links 18 together form the robot arm of the manipulator 14. The manipulator 14 may have a tandem arm configuration (e.g., Figure 1 (as shown), parallel arm configuration, or any other suitable manipulator configuration. In other examples, more than one manipulator 14 may be used in a multi-arm configuration.

[0036] exist Figure 1 In the example shown, the manipulator 14 includes multiple joints J and multiple joint encoders 19 located at the joints J for determining the position data of the joints J. For simplicity, only one joint encoder 19 is shown. Figure 1 As shown, although other joint encoders 19 may be similarly shown. According to one example, the manipulator 14 has six joints J1 to J6, which implement at least six degrees of freedom (DOF) of the manipulator 14. However, the manipulator 14 may have any number of degrees of freedom and may have any suitable number of joints J and may have redundant joints.

[0037] Manipulator 14 does not necessarily require joint encoder 19, but may instead use motor encoders present on the motors at each joint J. Furthermore, manipulator 14 does not necessarily require rotary joints, but may instead use one or more prismatic joints. Any suitable combination of joint types is contemplated.

[0038] The base 16 of the manipulator 14 is part of a fixed reference coordinate system that provides a reference coordinate system for the manipulator 14 or, in general, other components of the system 10. The origin of the manipulator coordinate system MNPL is defined at the fixed reference point of the base 16. The base 16 may be defined relative to any suitable part of the manipulator 14, such as one or more of the links 18. Alternatively or additionally, the base 16 may be defined relative to the manipulator trolley 17 (such as at the location where the manipulator 14 is physically attached to the manipulator trolley 17). In one example, the base 16 is defined at the intersection of the axes of joints J1 and J2. Thus, although joints J1 and J2 are moving parts in reality, the intersection of the axes of joints J1 and J2 remains a virtual fixed reference pose that provides both a fixed position and orientation reference and does not move relative to the manipulator 14 and / or the manipulator trolley 17. In other examples, the manipulator 14 may be a handheld manipulator, wherein the base 16 is the base portion of the tool (e.g., the portion held by the user's hand) and the tool tip is movable relative to the base portion. The base portion has a tracked reference coordinate system, and the tool tip has a tool tip coordinate system calculated relative to the reference coordinate system (e.g., calculated via motor and / or joint encoders and forward kinematics). The movement of the tool tip can be controlled to follow a path because its pose relative to the path can be determined.

[0039] Manipulator 14 and / or manipulator trolley 17 house manipulator controller 26 or other types of control units. Manipulator controller 26 may include one or more computers, or any other suitable form of controller that directs the movement of manipulator 14. Manipulator controller 26 may have a processing unit (CPU or GPU) and / or other processors and nontransitory memory. Manipulator controller 26 is loaded with software as described below. Processor may include one or more processors for controlling the operation of manipulator 14. Processor may be any type of microprocessor, multiprocessor, and / or multicore processing system. Manipulator controller 26 may additionally or alternatively include: one or more microcontrollers, field-programmable gate arrays, system-on-a-chip, discrete circuitry, and / or other suitable hardware, software, or firmware capable of implementing the functions described herein. The term processor is not intended to limit any implementation to a single processor. Manipulator 14 may also include a user interface (UI) with one or more displays and / or input devices (e.g., buttons, keyboard, mouse, microphone (voice-activated), gesture control devices, touchscreen, etc.).

[0040] Tool 20 is coupled to manipulator 14 and is movable relative to base 16 to interact with anatomical structures in certain modes. Tool 20 is a physical and surgical tool and, in some embodiments, is or forms part of an end effector 22 supported by manipulator 14. Tool 20 can be gripped by a user. One arrangement of manipulator 14 and tool 20 may resemble the arrangement described in U.S. Patent No. 9,119,655 entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is hereby incorporated by reference. Manipulator 14 and tool 20 may be configured alternatively. Tool 20 may resemble the tool shown in U.S. Patent Application Publication 2014 / 0276949 entitled “End Effector of a Surgical Robotic Manipulator,” filed March 15, 2014, which is hereby incorporated by reference.

[0041] Tool 20 may include an energy applicator 24 designed to contact and remove tissue from patient 12 at a surgical site. In one example, energy applicator 24 is a file 25. File 25 may be generally spherical and includes a spherical center, radius (r), and diameter. Alternatively, energy applicator 24 may be a drill bit, saw blade, ultrasonic vibrating tip, etc. Tool 20 and / or energy applicator 24 may include any geometric features such as perimeter, circumference, radius, diameter, width, length, volume, area, surface / plane, range of motion envelope (along any one or more axes), etc. Geometric features can be considered to determine how to position tool 20 relative to tissue at the surgical site to perform the desired treatment. In some embodiments described herein, for convenience and ease of illustration, a spherical file having a tool center point (TCP) will be described, but it is not intended to limit tool 20 to any particular form. In other examples, tool 20 does not include energy applicator 24. For example, tool 20 may be a grooving cutting guide for sawing, a guide tube for receiving another tool, etc.

[0042] Tool 20 may include a tool controller to control the operation of tool 20, such as controlling the power of the tool (e.g., controlling the power of a rotary motor of tool 20), controlling the movement of tool 20, controlling flushing / suction and / or similar operations of tool 20. The tool controller may communicate with manipulator controller 26 or other components. Tool 20 may also include a user interface (UI) with one or more displays and / or input devices (e.g., buttons, keyboard, mouse, microphone (voice activated), gesture control, touchscreen, etc.). For example, one of the user input devices on the user interface UI of tool 20 may be a tool input (e.g., a switch or other form of user input device) having a first input state and a second input state (see...). Figure 1 The tool input can be actuated by the user (e.g., pressed and held) to be placed in a first input state and can be released to be placed in a second input state. The tool 20 may have a handle on which the tool input is located. In some forms, the tool input is a presence detector that detects the presence of the user's hand, such as a momentary contact switch that toggles between on / off states, a capacitive sensor, an optical sensor, etc. The tool input is thus configured such that the first input state indicates that the user is actively engaging the tool 20 and the second input state indicates that the user has released the tool 20. The tool input can be a continuously activated device, i.e., an input that must be continuously actuated to allow the tool 20 to move in manual or semi-autonomous mode depending on which user input is actuated. For example, when the user continuously actuates the tool input and activates manual mode, the manipulator 14 will move in response to the input force and torque applied by the user, and the control system 60 will enforce a virtual object VO or virtual boundary 71 to protect the patient's anatomy. When the tool input is released, the input from the force / torque sensor S can be disabled, so that the manipulator 14 no longer responds to the force and torque applied to the tool 20 by the user.

[0043] Manipulator controller 26 controls the state (position and / or orientation) of tool 20 (e.g., TCP) relative to a coordinate system such as the manipulator coordinate system MNPL. Manipulator controller 26 can control the (linear or angular) velocity, acceleration, or other kinematic derivatives of tool 20. In one example, the tool center point (TCP) is a predetermined reference point defined at energy applicator 24. TCP has a known or computable (i.e., not necessarily static) pose relative to other coordinate systems. The geometry of energy applicator 24 is known in or defined relative to the TCP coordinate system. TCP may be located at the spherical center of file 25 of tool 20, such that tracking is done at only one point. TCP can be defined in different ways depending on the configuration of energy applicator 24. Manipulator 14 may employ an articulated / motor encoder or any other non-encoder position sensing method to enable determination of the TCP's pose. Manipulator 14 may use articulation measurements to determine the TCP pose and / or may employ techniques to directly measure the TCP pose. Control of tool 20 is not limited to a center point. For example, any suitable primitive, mesh, etc., can be used to represent tool 20.

[0044] System 10 also includes a navigation system 32. An example of the navigation system 32 is described in U.S. Patent 9,008,757, filed September 24, 2013, entitled "Navigation System Including Optical and Non-Optical Sensors," which is incorporated herein by reference. The navigation system 32 tracks the movement of various objects. Such objects include, for example, manipulators 14, tools 20, and anatomical structures (e.g., femur F and tibia T). The navigation system 32 tracks these objects to acquire state information for each object relative to the (navigation) locator coordinate system LCLZ. Coordinates in the locator coordinate system LCLZ can be converted to the manipulator coordinate system MNPL and / or vice versa using transformations.

[0045] The navigation system 32 includes a cart assembly 34 housing a navigation controller 36, and / or other types of control units. A navigation user interface (UI) operatively communicates with the navigation controller 36. The navigation user interface includes one or more displays 38. The navigation system 32 is capable of displaying a graphical representation of the relative status of the tracked object to the user using the one or more displays 38. The navigation user interface (UI) also includes one or more input devices for inputting information into the navigation controller 36 or otherwise selecting / controlling certain aspects of the navigation controller 36. Such input devices include interactive touchscreen displays. However, input devices may include any one or more of buttons, keyboards, mice, microphones (voice-activated), gesture controls, etc.

[0046] The navigation system 32 also includes a navigation locator 44 coupled to the navigation controller 36. In one example, the locator 44 is an optical locator and includes a camera unit 46. The camera unit 46 has an external housing 48 that houses one or more optical sensors 50. The locator 44 may include its own locator controller 49 and may also include a camera VC.

[0047] Navigation system 32 includes one or more trackers. In one example, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker 54, and a second patient tracker 56. Figure 1 In the example shown, the manipulator tracker is coupled to tool 20 (i.e., tracker 52A), the first patient tracker 54 is coupled to the femur F of patient 12, and the second patient tracker 56 is coupled to the tibia T of patient 12. In this example, patient trackers 54 and 56 are coupled to the skeletal portion. A pointer tracker PT is securely attached to a pointer P, which is used to align anatomical structures with the locator coordinate system LCLZ. Manipulator trackers 52A and 52B may be attached to any suitable component of manipulator 14 other than tool 20, such as base 16 (i.e., tracker 52B), or any one or more links 18 of manipulator 14. Trackers 52A, 52B, 54, 56, and PT may be fixed to their respective components in any suitable manner. For example, the trackers may be rigidly fixed, flexibly connected (fiber optic), or not physically connected at all (ultrasound), as long as there is a suitable (supplementary) way to determine the relationship (measurement) between the respective tracker and the object associated with it.

[0048] Any one or more of the trackers may include an active marker 58. The active marker 58 may include a light-emitting diode (LED). Alternatively, trackers 52A, 52B, 54, 56, and PT may have passive markers, such as reflectors that reflect light emitted from camera unit 46. Other suitable markers not specifically described herein may be used.

[0049] Positioner 44 tracks trackers 52A, 52B, 54, 56, and PT to determine the state of each of the trackers 52A, 52B, 54, 56, and PT, which respectively correspond to the state of the object to which they are attached. Positioner 44 can perform known triangulation techniques to determine the state of trackers 52A, 52B, 54, 56, and PT and the associated object. Positioner 44 provides the state of trackers 52A, 52B, 54, 56, and PT to navigation controller 36. In one example, navigation controller 36 determines the state of trackers 52A, 52B, 54, 56, and PT and transmits it to manipulator controller 26. As used herein, the state of an object includes, but is not limited to, data defining the position and / or orientation of the tracked object, or equivalents / derivatives of position and / or orientation. For example, the state can be the pose of the object and may include linear velocity data, and / or angular velocity data, etc.

[0050] The navigation controller 36 may include one or more computers, or any other suitable form of controller. The navigation controller 36 has a central processing unit (CPU) and / or other processor, non-transitory memory (not shown), and storage devices (not shown). The processor may be any type of processor, microprocessor, or multiprocessor system. The navigation controller 36 is loaded with software. For example, the software converts signals received from the locator 44 into data representing the position and orientation of the tracked object. The navigator controller 36 may additionally or alternatively include: one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuitry, and / or other suitable hardware, software, or firmware capable of implementing the functions described herein. The term processor is not intended to limit any implementation to a single processor.

[0051] Although one example of the navigation system 32 is shown as employing triangulation techniques to determine the state of an object, the navigation system 32 may have any other suitable configuration for tracking the manipulator 14, the tool 20, and / or the patient 12.

[0052] In another example, navigation system 32 and / or locator 44 are ultrasound-based. For example, navigation system 32 may include an ultrasound imaging device coupled to navigation controller 36. The ultrasound imaging device images any of the aforementioned objects (e.g., manipulator 14, tool 20, and / or patient 12) and generates a status signal to navigation controller 36 based on the ultrasound images. The ultrasound images may be 2D, 3D, or a combination of both. Navigation controller 36 can process the images in near real-time to determine the status of the object. The ultrasound imaging device may have any suitable configuration and may differ from... Figure 1 The camera unit 46 shown.

[0053] In another example, navigation system 32 and / or locator 44 are based on radio frequency (RF). For example, navigation system 32 may include an RF transceiver coupled to navigation controller 36. Manipulator 14, tool 20, and / or patient 12 may include an RF transmitter or repeater attached thereto. The RF transmitter or repeater may be passive or actively powered. The RF transceiver transmits RF tracking signals based on RF signals received from the RF transmitter and generates status signals to navigation controller 36. Navigation controller 36 may analyze the received RF signals to correlate with relevant statuses. The RF signals may have any suitable frequency. The RF transceiver may be positioned at any suitable location to effectively track objects using the RF signals. Furthermore, the RF transmitter or repeater may have any suitable structural configuration, which may be largely consistent with... Figure 1 The trackers shown are different: 52A, 52B, 54, 56, and PT.

[0054] In yet another example, the navigation system 32 and / or locator 44 are electromagnetic. For example, the navigation system 32 may include an EM transceiver coupled to the navigation controller 36. The manipulator 14, tool 20, and / or patient 12 may include EM components attached thereto, such as any suitable magnetic tracker, electromagnetic tracker, inductive tracker, etc. The tracker may be passive or actively powered. The EM transceiver generates an EM field and generates a status signal to the navigation controller 36 based on the EM signals received from the tracker. The navigation controller 36 may analyze the received EM signals to correlate with the relevant status. Similarly, this example of a navigation system 32 may have... Figure 1 The navigation system 32 shown has different structural configurations.

[0055] Navigation system 32 may have any other suitable components or structures not specifically listed herein. Furthermore, any of the techniques, methods, and / or components described above with respect to the illustrated navigation system 32 may be implemented or provided for any of other examples of navigation system 32 described herein. For example, navigation system 32 may use only inertial tracking or any combination of tracking techniques, and may additionally or alternatively include fiber-optic tracking, machine vision tracking, etc.

[0056] refer to Figure 2 System 10 includes a control system 60, which, among other components, includes a manipulator controller 26, a navigation controller 36, and a tool controller 21. The control system 60 also includes... Figure 3One or more software programs and software modules are shown. A software module may be part of one or more programs that operate on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof to process data to assist in the control of system 10. The software program and / or module includes computer-readable instructions stored in non-transitory memory 64 on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof, executable by one or more processors 70 of controllers 21, 26, 36. Memory 64 may be any suitable memory configuration, such as RAM, non-volatile memory, etc., and may be implemented locally or from a remote database. Additionally, software modules for prompting and / or communicating with the user may form part of one or more programs and may include instructions stored in memory 64 on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof. The user may interact with any of the input devices of the navigation user interface (UI) or other user interface (UI) to communicate with the software modules. The user interface software may run on a device separate from the manipulator controller 26, navigation controller 36, and / or tool controller 21.

[0057] The control system 60 may include any suitable configuration of input, output, and processing means adapted to implement the functions and methods described herein. The control system 60 may include a manipulator controller 26, a navigation controller 36, or a tool controller 21, or any combination thereof, or may include only one of these controllers. These controllers may be accessible via, for example... Figure 2 The wired bus or communication network shown communicates wirelessly or otherwise. Control system 60 may also be referred to as a controller. Control system 60 may include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuitry, sensors, displays, user interfaces, indicators, and / or other suitable hardware, software, or firmware capable of implementing the functions described herein.

[0058] refer to Figure 3 The software used by the control system 60 includes a boundary generator 66. For example... Figure 4 As shown, boundary generator 66 is a software program or module that generates a virtual boundary 71 for the movement and / or manipulation of constraint tool 20. The virtual boundary 71 can be one-dimensional, two-dimensional, or three-dimensional, and can include points, lines, axes, trajectories, planes, or other shapes, including complex geometries. In some embodiments, the virtual boundary 71 is a surface defined by a triangular mesh. Such a virtual boundary 71 can also be referred to as a virtual object. The virtual boundary 71 can be defined with respect to an anatomical model AM, such as a 3D skeletal model. Figure 4In the example, virtual boundary 71 is a planar boundary that delineates the five planes of the entire knee implant and is associated with a 3D model of the femur F. The anatomical model AM is registered to one or more patient trackers 54, 56, such that virtual boundary 71 becomes associated with the anatomical model AM. Virtual boundary 71 can be implant-specific, such as defined based on the size, shape, volume, etc., of the implant, and / or patient-specific, such as defined based on the patient's anatomy. Virtual boundary 71 can be a boundary created preoperatively, intraoperatively, or a combination thereof. In other words, virtual boundary 71 can be defined before the start of the surgical procedure, during the surgical procedure (including during tissue removal), or a combination thereof. In any case, control system 60 obtains virtual boundary 71 by storing / retrieving virtual boundary 71 from memory, obtaining virtual boundary 71 from memory, creating virtual boundary 71 preoperatively, creating virtual boundary 71 intraoperatively, etc.

[0059] Manipulator controller 26 and / or navigation controller 36 track the state of tool 20 relative to virtual boundary 71. In one example, the state of TCP is measured relative to virtual boundary 71 to determine, via virtual simulation 88, the tactile force to be applied to the virtual rigid body model such that tool 20 maintains a desired positional relationship with virtual boundary 71 (e.g., does not move beyond them). The results of virtual simulation 88 are commanded to manipulator 14. Control system 60 controls / positions manipulator 14 in a manner that simulates the response of a physical handheld device in the presence of physical boundaries / obstacles. Boundary generator 66 may be implemented on manipulator controller 26. Alternatively, boundary generator 66 may be implemented on other components such as navigation controller 36.

[0060] refer to Figure 3 and Figure 5The path generator 68 is another software program or module executed by the control system 60. In one example, the path generator 68 is executed by the manipulator controller 26. The path generator 68 generates a tool path TP that the tool 20 will traverse. The tool path TP may include multiple path segments PS, or may include a single path segment PS. Path segments PS may be straight segments, curved segments, or combinations thereof. The tool path TP may be defined relative to the manipulator 14 coordinate system MNPL, the locator coordinate system LCLZ, the coordinate system of the tool 20, the coordinate system of the anatomical structure, or any combination thereof. The tool path TP may be virtually attached to the coordinate system of the corresponding object, such that if the object moves, the tool path TP will move accordingly. The tool path TP may be implant-specific, for example, defined based on the size, shape, volume, etc. of the implant, and / or patient-specific, for example, defined based on the patient's anatomy. The tool path TP may be associated with a virtual model of the anatomical structure, and the virtual model and the tool path may be registered to the anatomical structure using the navigation system 32. The control system 60 can generate or obtain a tool path TP by storing / retrieving it from memory, creating it before operation, or creating it during operation. The tool path TP can have any 3D shape or combination of shapes, such as circular, spiral / drill-shaped, straight, curved, or combinations thereof.

[0061] In one implementation, the tool path TP is defined as a guide or alignment path. In one example, the tool path TP is used to guide tool 20 to a location where it is positioned to begin a surgical procedure or step. For example, if tool 20 is a saw blade, the tool path TP can be configured to guide the saw blade and align the cutting plane associated with the anatomical structure. If tool 20 is a cutting drill, the tool path TP can be configured to guide the cutting drill to a starting point when ready for automated cutting. The introduction path can virtually connect from the starting point to another cutting path for tissue removal. The tool path TP can also enable tool 20 to move along a predetermined path of motion for the purpose of registering components of manipulator 14 to navigation system 32. Navigation system 32 can be used to register the tool path TP to the anatomical structure, such that the tool path TP is virtually fixed to the anatomical structure. In this way, the position of the tool path TP in space will be automatically updated to account for any movement of the anatomical structure.

[0062] In another embodiment, such as Figure 5As shown, the tool path TP is defined as a tissue removal path. An example of a tissue removal path described herein includes a milling path 72. The term "milling path" refers to a path of tool 20 near a target site for milling an anatomical structure, and is not intended to require tool 20 to operatively mill the anatomical structure for the entire duration of the path. For example, as will be understood in further detail below, a milling path 72 may include portions or sections in which tool 20 transitions from one location to another without milling. Additionally, other forms of tissue removal, such as tissue ablation, may be employed along the milling path 72. The milling path 72 may be a predefined path created preoperatively, intraoperatively, or in combination thereof. In other words, the milling path 72 may be defined before the start of the surgical procedure, during the surgical procedure (including during tissue removal), or in combination thereof.

[0063] An example of a system and method for generating virtual boundary 71 and / or milling path 72 is described in U.S. Patent 9,119,655 entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is incorporated herein by reference. In some examples, virtual boundary 71 and / or tool path TP can be generated offline, rather than on manipulator controller 26 or navigation controller 36. Thereafter, virtual boundary 71 and / or tool path TP can be used by manipulator controller 26 at runtime.

[0064] refer to Figure 3 Two additional software programs or modules run on the manipulator controller 26 and / or the navigation controller 36. One software module performs behavior control 74. Behavior control 74 is the process of calculating data indicating the next command pose and / or the orientation (e.g., pose) of tool 20. In some cases, behavior control 74 outputs only the position of TCP, while in others, it outputs the position and orientation of tool 20. Outputs from boundary generator 66, path generator 68, and force / torque sensor S can be fed as inputs to behavior control 74 to determine the next command pose and / or the orientation of tool 20. Behavior control 74 can process these inputs, as well as one or more virtual constraints described further below, to determine the command pose.

[0065] The second software module executes motion control 76. One aspect of motion control is the control of manipulator 14. Motion control 76 receives data from behavior control 74 defining the next command pose. Based on this data, motion control 76 determines the next position of the joint angle of joint J of manipulator 14 (e.g., via inverse kinematics and a Jacobian calculator) so that manipulator 14 can position tool 20 in, for example, the command pose according to the command of behavior control 74. In other words, motion control 76 processes the command pose, which can be defined in Cartesian space, into joint angles of manipulator 14 so that manipulator controller 26 can accordingly command the joint motors to move joint J of manipulator 14 to the command joint angle corresponding to the command pose of tool 20. In one embodiment, motion control 76 adjusts the joint angle of each joint J and continuously adjusts the torque output of each joint motor to ensure, as closely as possible, that the joint motor drives the associated joint J to the command joint angle.

[0066] Boundary generator 66, path generator 68, behavior control 74, and motion control 76 may be subsets of software program 78. Alternatively, each may be a software program that operates separately and / or independently in any combination thereof. The term "software program" is used herein to describe computer-executable instructions configured to implement various capabilities of the described technical solutions. For simplicity, the term "software program" is intended to include at least one or more of boundary generator 66, path generator 68, behavior control 74, and / or motion control 76. Software program 78 may be implemented on manipulator controller 26, navigation controller 36, or any combination thereof, or may be implemented by control system 60 in any suitable manner.

[0067] A clinical application 80 can be provided to manage user interactions. The clinical application 80 manages many aspects of user interactions and coordinates surgical workflows, including preoperative planning, implant placement, registration, bone preparation visualization, and postoperative assessment of implant fit. The clinical application 80 is configured to output to a display 38. The clinical application 80 can run on its own separate processor or can run in conjunction with a navigation controller 36. In one example, after the user sets the implant placement, the clinical application 80 interacts with a boundary generator 66 and / or a path generator 68, and then sends the virtual boundary 71 and / or tool path TP returned by the boundary generator 66 and / or path generator 68 to a manipulator controller 26 for execution. The manipulator controller 26 executes the tool path TP as described herein. The manipulator controller 26 may additionally create certain segments (e.g., import segments) upon starting or restarting processing to smoothly return to the generated tool path TP. The manipulator controller 26 may also process the virtual boundary 71 to generate corresponding virtual constraints, as further described below.

[0068] System 10 can operate in manual mode, as described in U.S. Patent 9,119,655, which is incorporated herein by reference. Here, the user manually guides tool 20 and its energy applicator 24, and manipulator 14 performs movement of the tool and its energy applicator at the surgical site. The user physically contacts tool 20 to move tool 20 in manual mode. In one embodiment, manipulator 14 monitors the forces and torques applied by the user to tool 20 to position tool 20. For example, manipulator 14 may include a force / torque sensor S that detects the forces and torques applied by the user and generates corresponding inputs (e.g., one or more corresponding input / output signals) for use by control system 60. In some embodiments, it may be necessary for the user to continuously grip a trigger or switch on the end effector to activate the force / torque sensor S that detects the forces and torques applied by the user.

[0069] The force / torque sensor S may include a 6-DOF force / torque transducer. The manipulator controller 26 and / or navigation controller 36 receive inputs (e.g., signals) from the force / torque sensor S. In response to forces and torques applied by the user, the manipulator 14 moves the tool 20 in a manner that simulates the movement that would occur based on the forces and torques applied by the user. This allows the manipulator 14 to operate using an admittance-based system. The movement of the tool 20 in manual mode may also be constrained relative to a virtual boundary 71 generated by the boundary generator 66. In some forms, measurements by the force / torque sensor S are transformed from the force / torque coordinate system FT of the force / torque sensor S to another coordinate system, such as a virtual mass coordinate system VM, in which a virtual simulation 88 is performed on a virtual rigid body model of the tool 20, such that forces and torques can be virtually applied to the virtual rigid body in the virtual simulation 88 to ultimately determine how these forces and torques (and other inputs) will affect the movement of the virtual rigid body, as described below. In another example, the manipulator 14 may be operated using an impedance control system. In such examples, the user manipulates tool 20 toward a target and controls its position. Manipulator controller 26 can then generate a reaction force based on the interaction between tool 20 and the physical or virtual object. The techniques described herein can be used with manipulator 14 of any type of impedance or admittance control.

[0070] System 10 can also operate in semi-autonomous or automatic modes, wherein the manipulator 14 moves the tool 20 along the milling path 72 (e.g., the active joint J of the manipulator 14 operates to move the tool 20 without requiring the user to apply force / torque to the tool 20). Examples of operation in automatic mode are also described in U.S. Patent 9,119,655, which is incorporated herein by reference. In some embodiments, when the manipulator 14 operates in automatic mode, the manipulator 14 is able to move the tool 20 without the user applying force. In other words, the user does not need to physically contact the tool 20 to move it. Instead, the user can use some form of remote control to control the start and stop of the movement. For example, the user can press and hold a button on the remote control to start the movement of the tool 20 and release the button to stop the movement of the tool 20.

[0071] System 10 can also operate in a manual-guided mode, as described in U.S. Patent Application Publication US 2020 / 0281676 A1, entitled "Systems and Methods for Controlling Movement of a Surgical Tool Along a Predefined Path," the contents of which are incorporated herein by reference in their entirety. In manual-guided mode, the user applies force / torque to force / torque sensor S, and the applied force / torque is used to determine how far the tool 20 is advanced along the tool path TP. In manual-guided mode, the tool 20 is constrained to the tool path TP at a 2DOF normal to the tool path, but unconstrained at a 1DOF tangent to the tool path TP. In effect, this allows the tool 20 to move freely along the tool path TP based on manual input, but the constraint guides the user by limiting the manual movement of the tool 20 to along the tool path.

[0072] II. Techniques for estimating the deflection of surgical robotic arms

[0073] refer to Figure 6 and Figure 7 This document describes various techniques for estimating the deflection of manipulator 14. The techniques described herein can be implemented by any of the described control system, controller, and / or processor (including, but not limited to, control system 60, manipulator controller 26, tool controller 21, navigation controller 36), or any combination thereof. For simplicity, one or more devices are referred to herein as a “controller.” In some cases, any data described herein may originate from or be transmitted to a remote server.

[0074] The controller is configured to estimate the robotic arm deflection based on the interaction between tool 20 and the tissue of anatomical structure A. The controller can also characterize the tool-anatomical structure interaction based on the estimated arm deflection. To estimate the robotic arm deflection, the controller inputs the pose of tool 20 and tool interaction forces into a machine learning model. To characterize the interaction between the surgical tool and the anatomical structure, the controller inputs the operating parameters of surgical tool 20 and the estimated deflection into the machine learning model. The controller can optionally control manipulator 14 based on any described arm deflection estimate and / or tool-anatomical structure interaction characterization.

[0075] Furthermore, the techniques described herein advantageously provide calibration and / or compensation for the impact of non-geometric errors on the accuracy of manipulator-guided surgical tools 20 used in surgical applications. Unlike prior art, calibration / compensation can be dynamically calculated and implemented in real time. The accuracy of tool 20 in cutting / removing anatomical structures will be increased, thereby improving clinical outcomes and reducing the surgical operation time required to correct cutting inaccuracies caused by non-geometric errors.

[0076] Furthermore, unlike existing techniques, the proposed technique is limited to calibration. In one case, it can be assumed that manipulator 14 has already been calibrated, and the technique provides an "on-the-fly" control process during robotic surgical cutting, rather than a one-time calibration scheme. The machine learning network described in this paper can be trained against real cutting forces in all XYZ directions across the entire arm workspace. In some cases, the technique uses a nonlinear deep neural network for compliance modeling. The nonlinear deep neural network can model the compliance of the arm itself without making any assumptions about the compliance model. Unlike existing techniques, the properties (inputs) of the machine learning model can be measured forces (similar to forces measured by force / torque sensors) and the Cartesian robot pose at the TCP (instead of simply using joint angle inputs). The machine learning network described in this paper may only need to be trained once on a specific robot of this type, and the results can be applied to all other robots of the same type. These techniques eliminate the need to train each individual robotic manipulator.

[0077] A. Sensing systems and force detection

[0078] Various forces can be utilized in the deflection estimation and tissue interaction characterization techniques described herein. These forces include, but are not limited to, interaction forces, disturbance forces, and / or forces applied by the user. These different forces can be detected by any one or more components of the "sensing system." Components of the sensing system can be located at the manipulator 14, surgical instrument 20, navigation system 32, combinations thereof, or any other suitable location. One component of the sensing system may include a force / torque sensor S as described above, which can detect external forces / torques applied to the surgical instrument 20. In other examples, the sensing system may employ a force observer that can estimate any described force using computational methods, with or without sensors. The sensing system may include other components such as joint encoder 19 or other position sensors for the various joints J of the robot manipulator 14. In some cases, the actual joint position / torque can be compared to the expected joint position / torque. The sensing system may also utilize current sensors to detect the current consumption of the motors / actuators of the various joints J of the manipulator 14. The one or more joints may include force / torque sensors for measuring joint torque / force. In some cases, pressure sensors may be applied to the surface of one or more links of manipulator 14 to detect external forces applied to the robotic arm. Navigation system 32 may be used as part of a sensing system, for example, to determine the pose of robotic arm 14 or surgical tool 20 (from trackers 52A, 52B). The pose determined by navigation system 32 may be compared with a pose determined based on kinematic data from the manipulator to compare or infer the applied forces. The sensing system may utilize tool sensors to detect operating parameters of tool 20, such as motor current, resistance / impedance, cutting rate, velocity, etc. The sensing system may include inertial sensors (such as accelerometers, gyroscopes, magnetometers, etc.) coupled to any component of manipulator 14 and / or tool 20. The sensing system may utilize any one or more of these methods for measuring forces applied to manipulator 14. Equivalent techniques for these sensing technologies are also contemplated.

[0079] One type of force that the sensing system can detect is the interaction force. In one implementation, the interaction force is the force applied to the surgical tool 20 based on the interaction between the surgical tool 20 and the anatomical structure or patient tissue. The type of interaction force will depend on the type of tool 20 and the operation of the tool 20. For example, if the tool 20 is a tissue removal tool (such as a rotary cutter or saw blade), the interaction force can be the tissue removal (or cutting) force generated when the tool 20 removes tissue. If the tool 20 is a probing tool, the interaction force can be the pressure generated when the tool 20 presses into the tissue. If the tool 20 is an impactor tool, the interaction force can be the impact force generated when the tool 20 impacts the anatomical structure, and so on. In some cases, the interaction force can be the result of the tool 20 interacting with an obstacle or unintended object (such as a retractor, incision opening, etc.). The interaction force can be tactile feedback provided to the tool 20 in response to the interaction between the tool 20 and a virtual object or boundary associated with the anatomical structure A. The virtual object can generate a reaction force on the user's movement. The interaction force can be any number of forces and / or torques on any number of degrees of freedom. In one example, the interaction force can be measured relative to the TCP of tool 20. One component that may be well-suited for detecting the interaction force in the sensing system is a force / torque sensor S, which may be located between tool 20 and the distal link of manipulator 14. Alternatively, other sensing methods may be used to detect the interaction force, such as sensing tool operating parameters, strain gauges on tool 20 or the energy applicator, etc.

[0080] Another force that the sensing system can detect is the force applied by the user to the surgical tool 20. Here, the user is a human operator (worker / surgeon) who physically interacts with the manipulator 14 or tool 20. For example, the operator may grasp the tool 20 and apply force to move it. This movement could be to perform cutting / manipulation of an anatomical structure using the tool 20, or it could be movement for any other purpose. The type of force applied by the user can depend on the type of tool 20 and the surgical procedure / operation to be performed. For example, if the tool 20 is a tissue removal tool such as a rotary cutter, the force applied by the user may be due to the user applying force to move the tool 20 along a predetermined tool path or free tool path. If the tool 20 is a drilling tool, the force applied by the user may be due to the user applying force to advance the tool 20 along a trajectory, and so on. The force applied by the user can be any number of forces and / or torques on any number of degrees of freedom. In one example, the force applied by the user can be measured relative to the TCP of the tool 20. The sensing system can utilize a force / torque sensor S to detect the force applied by the user. Alternatively or alternatively, other sensing methods (such as the weighing sensor on tool 20 itself or any other sensing method described) can be used to detect the force applied by the user.

[0081] Disturbance force is another type of force that can be detected by the sensing system. Disturbance force is a force applied to one or more links in the linkage 18 of the manipulator 14. Disturbance force can be a contact force from a known or unknown source that can cause deflection of the robot arm links. Disturbance can be caused by a variety of sources. For example, interference force can be caused by interaction forces, user-applied forces on tool 20, user-applied forces on the robot arm, collisions with the robot arm, unexpected payloads, gravity, reverse drive forces, etc. Disturbance force can be any number of forces and / or torques on any number of degrees of freedom. The sensing system can detect interference force using joint motor current or torque sensors in joint J. Alternatively or concurrently, other sensing methods (such as comparing expected joint torque with actual joint torque, or using pressure sensors on linkage 18, etc.) can be used to detect interference force.

[0082] The techniques described herein may utilize one or more, or all, of the described forces (if available and detected). However, it is not necessary to utilize or even detect all forces. For example, these techniques may utilize interaction forces and user-applied forces, but not interference forces. These forces may be detected simultaneously or at different times. These forces may be related to the same event or may be caused by different events. These forces may be identical, partially related, or completely independent. For example, depending on the conditions, interaction forces and user-applied forces may be related or unrelated. For automatic or semi-automatic operation of manipulator 14, interaction forces may exist due to the movement of tool 20 along the path, but no user-applied force is present because tool 20 is moved automatically by manipulator 14. For manual operation of manipulator 14, there may be interaction forces substantially equal to the user-applied force generated by the user manually pushing tool 20 into the tissue.

[0083] The controller is configured to map any of the described forces into the manipulator coordinate system MNPL. For example, forces can be mapped to the base 16 of manipulator 14 to provide a common reference. Mapping to base 16 makes it possible to compare the kinematic data of manipulator 14 and its TCP pose relative to base 16 during the movement of tool 20 tracked by navigation system 32. Having described various detectable forces, the following sections will describe deflection estimation and tissue interaction characterization techniques that utilize some or all of these forces to predict or control manipulator 14.

[0084] B. Arm deflection estimation based on tool-anatomical structure interaction

[0085] According to one implementation method, such as Figure 6 As shown, a technique / method 200 implemented by a controller is provided for estimating the deflection of the manipulator 14 in response to the interaction of the tool 20 with the anatomical structure A. The estimated deflection is a non-geometric error. Figure 6 All the steps shown are required, and some steps are optional.

[0086] At point 202, the controller obtains the pose of the surgical tool 20 based on kinematic data from the manipulator 14. The kinematic data can be derived from the kinematic model of the links 18 and joints J of the manipulator 14, measured and defined by the joint J encoder. Alternatively, the pose of the surgical tool 20 can be obtained by the navigation system 32 using, for example, trackers 52a, 52b, to track the manipulator 14 and / or the surgical tool 20. The kinematic data and tracking data can be fused or compared as needed to obtain the pose of the tool 20. Specifically, the pose of the surgical tool 20 can be the TCP pose of the tool 20. The TCP pose can be mapped to the base 16 of the manipulator 14. The pose of the tool 20 can be defined by its position and / or orientation defined by components in six degrees of freedom, or the pose of the tool 20 can be uniquely defined by its distance to the robot base coordinate system and its orientation. The pose of the tool 20 can be obtained at time step T. The pose of the tool 20 can be obtained in discrete time or continuously over a period of time.

[0087] At 204, the controller receives the interaction force applied to the surgical tool 20 in response to the interaction between the surgical tool 20 and the anatomical structure or patient tissue. The interaction force is derived based on measurements from a sensing system. As described above, the sensing system can obtain such measurements from various sources. In one embodiment, the measurement is provided by a force / torque sensor S coupled between the manipulator 14 and the tool 20, and the interaction force is more specifically a cutting force in response to the tool 20 cutting bone. The interaction force can be defined relative to the TCP of the tool 20 and is defined by force / torque components in three axial forces or six degrees of freedom. The interaction force can be received at a time step T (e.g., simultaneously with obtaining the pose of the tool 20 at step 202). The interaction force can be received in discrete time or monitored over a period of time. This period of time can be triggered in response to specific conditions, such as the initial detection of the interaction force.

[0088] At position 206, the controller inputs the pose and interaction forces of the surgical instrument 20 into the machine learning model ML. The machine learning model ML can be used for "instantaneous" non-geometric error compensation during surgery.

[0089] In one example, the machine learning model (ML) is a deep learning model that includes at least two hidden layers. A machine learning model can be any one or more types of models, such as: neural networks, convolutional neural networks, deterministic policy gradient (DPG) or deep DPG, deep Q-learning, recurrent neural networks, generative adversarial networks, supervised machine learning models, regression algorithms, classification algorithms, Naive Bayes classifiers, random forest algorithms, unsupervised machine learning models, K-means clustering, hierarchical clustering, probabilistic clustering, reinforcement learning, NLP models, transformer networks, autoencoders, actor-critic networks, etc.

[0090] The machine learning model ML receives the pose and interaction force values ​​from the surgical tool 20 at the input layer, which has an appropriate number of input nodes / neurons for that number of input values. The hidden layers learn how much the robotic arm deflects for different interaction forces under different arm configurations during the interaction between the tool and the anatomical structure A. The hidden layers become more accurate iteratively as they make predictions based on the input data and possible output data (if supervised).

[0091] The machine learning model (ML) can be trained or pre-trained in various ways. Hidden layers can be trained online or offline based on predetermined values ​​of tool pose and interaction forces. For example, the ML model can be trained on a manipulator 14 that has already been calibrated using level 1 and level 2 kinematic (geometric) calibrations to isolate and learn the nonlinear compliance of the robotic arm for various arm configurations. In some cases, varying payloads can be attached to the manipulator 14 during training. A laser tracker can be used to train the network. A laser tracker is a precise measuring device with an accuracy up to 16 micrometers that can determine the position of a spherical mounted mirror (SMR) that remains against the object being measured. Special end effectors with adjustable weights (up to 10 kg) can be used with multiple SMRs to collect XYZ position data for different arm poses covering the workspace of the manipulator 14. This helps to better train the joints and collect more points for different arm poses under varying loads of cutting forces during simulated surgery. Alternatively, hidden layers can be trained online based on values ​​of tool pose and interaction forces acquired in real time during surgery.

[0092] The output of the machine learning model ML can be an output layer comprising any suitable number of nodes. For example, the output layer could include three nodes, each predicting the estimated deflection of the manipulator 14 for one axial degree of freedom (x, y, z). This output can be characterized as the x, y, z error displacements of the TCP position relative to its command / control position at a time step T, where T is the time for acquiring the interaction forces and tool pose. The output can also predict the orientational deflection (pitch, roll, yaw) of the tool 20 due to non-geometric errors.

[0093] At 208, the controller uses the output of the machine learning model ML to estimate the deflection of the manipulator 14 and / or surgical instrument 20 caused by the interaction of the tool 20 with the anatomical structure A. In some cases, a deflection threshold can be used to filter out negligible deflection estimates and confirm deflection estimates above the threshold. Alternatively, a time threshold can be used to filter out short-duration deflection estimates and confirm deflection estimates for durations above the time threshold.

[0094] The controller can utilize a machine learning model (ML) to output estimated deflections for any number of time steps during the operation of manipulator 14. For example, estimated deflection values ​​can be generated for each time step. In some cases, to reduce computational resources, the controller can trigger a deflection monitoring mode during which the machine learning model (ML) is used to make deflection predictions. The deflection monitoring mode can be triggered based on specific conditions related to tool 20, anatomical structure A, or surgical procedures. For example, the deflection monitoring mode can be triggered in response to the detection of any interaction force. In other examples, the deflection monitoring mode can be triggered based on tracking data from navigation system 32 (e.g., indicating that tool 20 is within a threshold distance from or in contact with anatomical structure A). In another example, the deflection monitoring mode is triggered in response to a user pressing a trigger to operate tool 20, or in response to a reading generated by force / torque sensor S.

[0095] Using the estimated deflection (optionally filtered by a threshold), the controller can perform a variety of tasks. Any of the following tasks can be executed within a subsequent time step T+1 and can continue thereafter. Any of the tasks described herein can be combined partially or entirely, and can be executed simultaneously or at different times.

[0096] In one example, at 210, the controller can generate a warning / notification related to the deflection. For example, the warning / notification can be immediately displayed on the display 38 of the navigation system 32 to convey a message to the operator such as "Warning—Loss of robot accuracy." Alternatively, a warning can be delivered via the manipulator 14 to indicate that excessive force has been applied. For example, the controller can provide vibration feedback to the user via the surgical tool 20, or an indicator on the manipulator 14 can be illuminated with a color such as red. Other types of audio / video notifications can also be delivered to the operator.

[0097] Alternatively or additionally, at 212, the controller may control the manipulator 14 and / or tool 20 based on the estimated deflection. Various examples exist of how the controller may control the manipulator 14 and / or tool 20 based on the estimated deflection. Controls may be performed to improve the accuracy of tool 20. Alternatively or additionally, controls may be performed to counteract or reduce the estimated deflection.

[0098] For example, at 214, the controller can adjust the toolpath TP of tool 20. One or more segments of the toolpath TP can be dynamically modified to reduce the force exerted on tool 20 by the anatomical structure A. For example, the toolpath TP can be partially offset to reduce the overlap between the radius of the ball file and the anatomical structure A. In other examples, the number of passes (reciprocating path oscillations) of the toolpath TP can be dynamically adjusted to be more frequent. The depth of the toolpath TP can also be modified. For example, arm deflection often leads to undercutting. Therefore, in the case of arm deflection, the depth of the toolpath can be increased to compensate for undercutting. Although this may result in more force being applied to tool 20, the result will be improved cutting accuracy. The degree of modification to the toolpath TP can correspond to the estimated amount of deflection. If the estimated deflection no longer persists, the toolpath TP can return to the default or last setting.

[0099] At 216, the controller can modify the feed rate of tool 20. The feed rate is the velocity or speed at which tool 20 travels along the tool path TP. Based on the estimated deflection, the feed rate of tool 20 can be slowed down to reduce the interaction force exerted on tool 20 by the anatomical structure A. The slowing down can be gradual or immediate. The degree of reduction in the feed rate can correspond to the estimated magnitude of the deflection. If the estimated deflection no longer persists, the feed rate can be restored to the default setting or the last setting.

[0100] At 218, the controller can modify the pose of tool 20. This pose may include the TCP and axis / body of tool 20. The pose of tool 20 can be modified to improve tool accuracy and / or offset or reduce the estimated deflection. For example, the position of the TCP can be adjusted to account for the estimated deflection value (x, y, z) output by the machine learning model ML. In other examples, the orientation of tool 20 can be adjusted to reduce the interaction between tool 20 and anatomical structure A. In some cases, the tool axis orientation can be adjusted if the tool axis collides with anatomical structure A. In some cases, avoiding arm deflection may be a suitable action. The pose change can be gradual or immediate. The degree of pose change can correspond to the magnitude of the estimated deflection. If the estimated deflection no longer persists, the pose can return to the standard command pose / control pose of tool 20.

[0101] At 220, the controller can actively move tool 20 away from the anatomical structure. This technique can be used as a precaution to provide the operator with a pause to reassess the operating conditions of tool 20 or mitigate deflection. Tool 20 can move a short distance away from anatomical structure A, allowing it to easily resume its movement upon reactivation. In other examples, tool 20 can be moved significantly to, for example, a resting position separated from anatomical structure A. This pull-away action can be gradual or immediate, and the timing or rate of execution can correspond to the estimated magnitude of deflection. If the estimated deflection is no longer sustained, tool 20 can return to its last known pose or position on the tool path TP. The return can be performed automatically or manually.

[0102] At 222, the controller can modify the cutting rate and / or direction of tool 20. The cutting rate is the rotational rate (e.g., RPM) of an energy applicator such as a cutting drill. The cutting direction is the rotational direction of the energy applicator (e.g., clockwise or counterclockwise). Based on the estimated deflection, the cutting rate and / or direction can be modified to reduce the interaction forces exerted on tool 20 by the anatomy A. For example, the cutting rate can be gradually or immediately reduced to decrease the contact forces. For certain sections of the tool path TP, the cutting direction can be changed to utilize climb milling instead of conventional milling. The timing and extent of the modification to the cutting rate and / or direction can correspond to the estimated magnitude of the deflection. If the estimated deflection no longer persists, the cutting rate and / or direction can be reverted to the default or last setting.

[0103] At 224, the controller can actively stop tool 20. Stopping can include ceasing all energy applied to tool 20, including zeroing the feed rate and cutting rate. Similar to pulling the tool back, this technique can be used as a precaution to provide the operator with a pause to reassess the operating conditions of tool 20 or mitigate the estimated deflection. Tool 20 can be stopped for any appropriate amount of time. The stop can be gradual or immediate, and the timing or rate of execution of the stop can correspond to the estimated magnitude of the deflection. If the estimated deflection is no longer sustained, the power supplied to tool 20 can be automatically or manually returned in response to user confirmation.

[0104] At 226, the controller can generate or modify the virtual haptic settings of the manipulator 14 and / or tool 20 based on the estimated deflection. For example, the controller can dynamically generate a virtual object VO or boundary 71 based on the estimated deflection. The virtual object or boundary can be any of those described above. In one example, the virtual boundary or object can be generated based on the location and / or magnitude of the estimated deflection. The virtual object or boundary can (e.g., during a specific time period or in a specific region susceptible to high deflection) restrict the movement of the tool 20 and / or robotic arm to prevent deflection. In one example, the virtual object can be a volume in which the tool 20 is constrained to remain or excluded. In another example, the virtual object can be a mesh of variable constraint (stiffness / damping) parameters determined based on the estimated deflection on the skeletal surface. Such implementations can be similar to those described in U.S. Patent No. 11,986,260 entitled “Robotic surgical system and methodsutilizing virtual boundaries with variable constraint parameters,” the entire contents of which are incorporated herein by reference. In yet another example, virtual boundaries can be layered based on their criticality or proximity to the target surface, as described in U.S. Patent No. 10,098,704 entitled "System and method for manipulating an anatomy," the entire contents of which are incorporated herein by reference. Virtual objects or boundaries can be enforced for any suitable amount of time. If the estimated deflection no longer persists, the virtual object or boundary can be automatically or manually disabled in response to user confirmation. Any of the example steps 214 through 226 can be performed concurrently or in combination.

[0105] C. Total arm deflection estimation

[0106] According to another embodiment, such as Figure 7As shown, a technique / method 300 implemented by a controller is provided for estimating the total deflection of the manipulator 14 in response to various forces (such as interaction forces, user-applied forces, and disturbance forces) described herein. Again, this estimated total deflection is a non-geometric error. Some of the steps described below can be performed in the same manner as those described with respect to technique / method 200, and therefore, for readability purposes, the description of such steps has been reduced. Furthermore, not... Figure 7 All steps shown are required, and under given conditions, some steps may be optional or not applicable.

[0107] At 302, the controller obtains the pose of the surgical tool 20 based on kinematic data from the manipulator 14. At 304, if applicable, the controller obtains the interaction forces applied to the surgical tool 20 in response to the interaction between the surgical tool 20 and anatomical structures or patient tissues, in a manner described above.

[0108] At 306, if applicable, in response to a user applying an external force to move tool 20, the controller acquires the force applied by the user on surgical tool 20. The user-applied force is derived based on measurements from a sensing system. As described above, the sensing system can acquire such measurements from various sources. In one embodiment, the measurement is provided by a force / torque sensor S. The user-applied force can be defined relative to the TCP of tool 20 and is defined by force / torque components in three axial forces or six degrees of freedom. The user-applied force can be mapped to robot base 16. The user-applied force can be acquired at a time step T (e.g., simultaneously with acquiring the pose of tool 20 at step 302 and / or acquiring the interaction forces at 304). The user-applied force can be acquired in discrete time or monitored over a period of time. This period of time can be triggered in response to specific conditions, such as the initial detection of the user-applied force.

[0109] At 306, if applicable, the controller obtains a disturbance force on one or more links 18 of the manipulator 14 in response to some external force applied to the robot arm. The disturbance force is derived based on measurements from a sensing system. In one embodiment, the measurement is provided by a current sensor from the joint J actuator / motor. However, other sensing techniques are contemplated. The disturbance force can be defined relative to the TCP of the tool 20 and is defined by force / torque components in three axial forces or six degrees of freedom. The disturbance force can be mapped to the robot base 16. The disturbance force can be obtained at a time step T (e.g., simultaneously obtaining the pose of the tool 20 at step 302, the interaction force at step 304, and / or the force applied by the user at step 306). Similarly, the disturbance force can be obtained in discrete time or monitored over a period of time. This period of time can be triggered in response to specific conditions, such as the initial detection of the disturbance force.

[0110] At 310, the controller inputs the pose of the surgical tool 20, along with (where applicable) interaction forces, user-applied forces, and / or disturbance forces, into the machine learning model ML'. The machine learning model ML' can then utilize these diverse inputs for "instantaneous" non-geometric error compensation during surgery. The machine learning model can be similar to any of the examples described above, such as, but not limited to, a deep learning model comprising at least two hidden layers. The machine learning model ML receives the pose of the surgical tool 20 and values ​​of various forces (however applicable) at an input layer with an appropriate number of input nodes / neurons for that number of input values. The hidden layers understand how much total robotic arm deflection has occurred in the presence of the various forces. The machine learning model ML' can be trained or pre-trained in the manner described above. The output of the machine learning model ML' can be an output layer comprising any suitable number of nodes, such as three nodes, each predicting the estimated total deflection of the manipulator 14 for one axial degree of freedom (x, y, z). This output can be characterized as the x, y, z error displacement of the TCP position relative to its command / control position at time step T, where T is the time for acquiring the applicable force and tool pose. This output can also predict the directional deflection (pitch, roll, yaw) of the tool 20 due to non-geometric errors.

[0111] At 312, the controller uses the output of the machine learning model ML' to estimate the total deflection of manipulator 14 and / or surgical instrument 20. Deflection and / or duration thresholds can be used to filter out negligible deflection estimates. The controller can use the machine learning model ML' to output an estimated total deflection for any number of time steps during the operation of manipulator 14. The controller can trigger a deflection monitoring mode to estimate the total deflection, and can do so based on any of the triggering conditions described above.

[0112] Using the estimated total deflection, the controller can perform the various tasks described above, which can be executed within a subsequent time step T+1 and can continue thereafter. Any of the tasks described in this paper can be combined partially or entirely, and can be executed simultaneously or at different times.

[0113] In one example, at 314, the controller can generate a warning / notification related to the estimated total deflection. Alternatively, at 316, the controller can control the manipulator 14 and / or tool 20 based on the estimated total deflection. Examples of how the controller can control the manipulator 14 based on the estimated deflection can be similar to those described above. For example, at 318, the controller can adjust the tool path TP of tool 20. At 320, the controller can modify the feed rate of tool 20. At 322, the controller can modify the pose of tool 20. At 324, the controller can actively move tool 20 away from the anatomical structure. At 326, the controller can modify the cutting rate and / or orientation of tool 20. At 328, the controller can stop all power supply to tool 20 or disable all tool operations. At 330, the controller can generate or modify virtual haptic settings for manipulator 14 and / or tool 20 based on the estimated deflection, for example, to guide / constrain the tool to improve accuracy. Any of the example steps 318 to 330 can be performed simultaneously or in combination.

[0114] III. Techniques for characterizing tool-anatomical interactions based on estimated arm deflection

[0115] refer to Figures 8 to 10 This paper describes various techniques for characterizing tool-anatomical interactions, for example, using the estimated deflection output described above as input. Furthermore, by using the estimated deflection, the techniques described herein advantageously provide higher-order predictions of tool-anatomical interactions beyond conventional feedback mechanisms responding to tool sensors. For example, with these techniques, a controller can determine the nature of the interacting tissue, the workload of tool 20, and / or the difficulty of the task performed by tool 20. The controller can then react dynamically to this determination in real time. Consequently, the accuracy of tool 20 in cutting / resecting anatomical structures will increase, thereby improving clinical outcomes and reducing surgical operation time required to correct for cutting inaccuracies caused by suboptimal tool-anatomical interactions. This technique enables the characterization of tool-anatomical interactions based on estimated robotic arm deflection. Furthermore, the techniques described herein can characterize interactions on the robotic arm side, thus avoiding complete reliance on measurements at the site of interaction. For example, if tool 20 jams, these techniques can still estimate robotic arm deflection, while other techniques relying on tool manipulation cannot.

[0116] Similarly, the techniques described herein can be implemented by any of the described control system, controller, and / or processor (including, but not limited to, control system 60, manipulator controller 26, tool controller 21, navigation controller 36) or any combination thereof. For simplicity, one or more devices are referred to herein as a “controller.” In some cases, any data described herein may originate from or be transmitted to a remote server.

[0117] According to one implementation, such as Figure 8 As shown, a technique / method 400, implemented by a controller, is provided for characterizing tool-anatomical interactions. Some steps described below can be performed in the same manner as those previously described with respect to technique / method 200 or 300, and therefore, for readability purposes, the description of such steps has been reduced. Furthermore, not... Figure 8 All steps shown are required, and under given conditions, some steps may be optional or not applicable.

[0118] At 402, the controller acquires one or more operating parameters of the surgical tool 20 during its interaction with the anatomical structure A. Examples of operating parameters of the surgical tool 20 may include, but are not limited to, any one or more of the following tool parameters: displacement, velocity, acceleration, motor current, torque / force, pressure, feed rate, cutting rate, cutting direction, temperature, etc. Operating parameters can be acquired through sensing system components. Additionally, the navigation system 32 may provide operating parameters based on tracking data. The controller can calculate the average of any of the parameters (such as average velocity or average motor current). Any type of averaging filter, such as a moving average filter, can be used. Operating parameters can be associated with any component of the tool 20 (such as energy applicator, TCP, tool axis, handpiece, motor / actuator, controller / PCB, vision indicator, trigger / button, etc.). Operating parameters can be acquired at any time during the procedure. Any of the operating parameters can be acquired discretely or monitored over a period of time. In one example, operating parameters of the tool 20 can be acquired at time step T. In some cases, the controller may request operating parameters in response to the detection of specific conditions (such as detecting that tool 20 interacts with or is near anatomical structure A).

[0119] As described above, the controller also acquires an estimate of the robot arm's deflection. For example, at 404, the controller can acquire the estimated deflection of the robot arm due to the interaction between the surgical instrument and anatomical structure A. It can also be based on... Figure 6The output of step 208 generates the estimated deflection. Alternatively, at 406, the controller may optionally obtain the total estimated deflection of the robot arm, which is based on... Figure 7 The output of step 312 is generated. As described, this total estimated deflection can be generated in part based on the estimated deflection based on the tool-anatomical structure interaction. In any case, the various implementations for estimating robot arm deflection described above are fully incorporated herein by reference and will not be repeated for simplicity.

[0120] At point 408, the controller inputs the obtained operating parameters of tool 20 and the estimated (tool-anatomy and / or total) deflection of the robotic arm into the machine learning model ML". The machine learning model ML can utilize these different inputs to provide an "on-the-fly" representation of the tool-anatomy interaction during surgery. The machine learning model ML can be similar to any of the examples described above, such as, but not limited to, a deep learning model including at least two hidden layers. In some cases, the controller utilizes at least two machine learning models, for example, one model ML / ML′ for estimating arm deflection and a second model ML" for representing the tool-anatomy interaction. In other cases, any of the described machine learning models ML, ML′, and ML" can be combined into a single trained machine learning model, for example, a single trained machine learning model that can estimate arm deflection and represent the tool-anatomy interaction.

[0121] To characterize tool-anatomical interactions, a machine learning model ML'' can be trained or pre-trained in various ways. Hidden layers can be trained online or offline based on predetermined values ​​of tool operating parameters and estimated deflections. For example, the machine learning model ML'' can be trained on a manipulator 14 that has been fully calibrated using level 1 and level 2 kinematic (geometric) calibrations and optionally further combined with the estimated deflection compensation techniques described above. In some cases, during training, the manipulator 14 can be moved to enable the tool 20 to interact with physical anatomy structures with various applied forces and poses, thereby simulating tool interactions during surgery. Alternatively, hidden layers can be trained online based on tool operating parameters acquired in real-time during surgery and estimated deflection values. Training can include tissue parameter / type labeling for a specific patient or for a statistical population of patients. Offline autonomous training can be performed using multiple tissue samples (real samples and / or phantoms) with different elastic moduli, while measuring tool (arm) parameters and arm deflections. Training can also be performed by analyzing historical data from multiple cases with different bone densities, extracting average measured forces during bone resection, and extracting data that can be fed into the machine learning model ML (e.g., Figure 6The arm pose for a specific bone resection is used to train a machine learning model for bone mineral density (BMD) to estimate arm deflection for a given case. Then, since the BMD for that specific case is known, the estimated arm deflection and arm parameters can be used to train the machine learning model (ML) (e.g., ...). Figure 8 (Supervised learning).

[0122] The machine learning model "ML" receives operational parameters from tool 20 and estimated (tool-anatomy and / or total) deflection values ​​of the robotic arm at an input layer with an appropriate number of input nodes / neurons for that number of input values. Hidden layers learn how to represent the tool-anatomy interaction in the presence of various estimated robotic arm deflections. Depending on the nature / type of the representation, the output of the machine learning model "ML" can be an output layer comprising any suitable number of nodes. This output can include text, measurements, images, or any suitable output format appropriate for representing the tool-anatomy interaction.

[0123] At point 410, based on the output of the machine learning model ML", the controller obtains a representation and / or classification of the tool-anatomical interaction. The tool-anatomical interaction can be represented in various ways. In one example, the machine learning model ML" represents the tool-anatomical interaction by representing the type of anatomical structure A. For example, the type of anatomical structure A can be represented by its tissue type (such as bone, soft tissue, ligament, tendon, cartilage, osteophyte, cancellous bone, cortical tissue, etc.). Alternatively, the type of anatomical structure A can be represented by the label / name of the body part (such as femur, tibia, acetabulum, scapula, glenoid cavity, etc.). For reconstructive surgery, in the presence of an existing primary implant in the bone, the machine learning model ML" can be able to identify other materials or objects (such as the presence of the implant) or characterize the features or type of the implant (e.g., metal implant, joint implant, screw, etc.).

[0124] To further illustrate, Figure 9A diagram illustrating example classifications output by the machine learning model ML” is provided. Here, the operating parameters of tool 20 include the average velocity of TCP, and the estimated deflection is based on tool-anatomical structure interactions. During training, the machine learning model ML” learns that the tissue interaction arm deflection values ​​exhibit a pattern, where these values ​​are grouped into skeletal regions, soft tissue regions, or free zone regions. Free zone regions indicate that tool 20 is moving in space but not directly interacting with anatomical structure A. For any given time step T, the machine learning model ML” can predict whether tool 20 is interacting with bone, soft tissue, or air. Similar machine learning recognition can be achieved by using other values ​​to determine the various other classifications described above.

[0125] Continuing to 410, the tool-anatomical interaction can also be characterized by classifying the parameters / features of anatomical structure A. By estimating arm deflection and tool manipulation parameters, the machine learning model ML can make predictions about anatomical structure density, stiffness, flexibility, etc. To further illustrate, Figure 10 Another figure is provided illustrating an example classification output by the machine learning model ML”. Here, the operating parameters of tool 20 again include the average velocity of TCP, and the estimated deflection is based on tool-anatomical structure interaction. During training, the machine learning model ML” learns that the tissue interaction arm deflection values ​​exhibit a pattern where these values ​​are grouped into six regions of anatomical structure (tissue) density, ranging from very soft to very hard. The labels for these groupings are provided only as examples. These groupings can have any labels, or no labels at all. Alternatively, the classification groupings can be labeled by density / hardness values ​​or ranges of values. Very soft regions can indicate that tool 20 is interacting with soft tissue, and very hard regions can indicate that tool 20 is interacting with particularly hard materials, such as dense femoral cortical bone. For any given time step T, the machine learning model ML” can predict the parameters of the anatomical structure. Similar machine learning recognition can be achieved by using other values ​​to determine the various other classifications described above, or combinations thereof. In addition to the average TCP velocity, any of the various operating parameters described herein can be used to estimate the parameters of the anatomical structure.

[0126] The technique described in this paper is highly advantageous for characterizing bone mineral density (BMD), which is correlated with arm deflection. Many traditional methods in clinical applications rely on tissue deformation and the forces applied to the tissue to characterize it. When it comes to bone tissue, there is minimal tissue deformation (if any). Conversely, robotic arm deflection typically occurs because external loads cannot deform bone tissue. Therefore, the technique proposed in this paper will estimate BMD more efficiently compared to existing methods.

[0127] Continuing with reference to 410, tool-anatomical structure interactions can also be characterized by workload or task difficulty. By estimating arm deflection and tool operation parameters, a machine learning model ML” can make predictions about how difficult the tool 20 is working during operation and / or how difficult the task the tool 20 is performing. For example, the machine learning model ML” can be trained to combine classifying task difficulty with classifying anatomical structure type / parameters. In other examples, the machine learning model ML” can estimate workload or task difficulty by predicting energy transfer or work as a function of force and displacement. For example, it can be similar to Figure 10 The groupings shown are used to classify groups where the machine learning model "ML" can predict that tool 20 is interacting with soft tissue and therefore will not exhibit much difficulty in performing its task (e.g., cutting / probing / etc.). The machine learning model "ML" can predict that tool 20 is interacting with particularly hard materials (such as dense femoral cortical bone) and therefore will exhibit great work in performing its task. For any given time step T, the machine learning model "ML" can predict the workload or task difficulty of tool 20. The workload or task difficulty can be estimated using any of the various operating parameters described herein.

[0128] After appropriate tool-anatomical interaction characterization / classification, the controller can then perform various tasks. Any of these tasks can be performed within a subsequent time step T+1 and can continue thereafter. Any of the tasks described in this paper can be combined partially or entirely, and can be performed simultaneously or at different times.

[0129] In one example, at 412, the controller can generate warnings / notifications related to the representation of the tool-anatomical interaction. For example, the warning / notification could be immediately presented on the display 38 of the navigation system 32 to convey a message to the operator such as “excessive tool workload.” In another example, after characterizing the tissue and workload, the controller can display a message to the user and / or issue an alarm to prevent tissue damage. Alternatively or additionally, any appropriate warnings can be delivered via the manipulator 14 (e.g., via vibration feedback to the user through the surgical tool 20 or by illuminating an indicator on the manipulator 14 with a color such as red). Other types of audio / video notifications can also be delivered to the operator.

[0130] Alternatively or additionally, at 414, the controller may control the manipulator 14 and / or tool 20 based on the tool-anatomical interaction characterization. For example, the controller may control the manipulator 14 and / or tool 20 to dynamically adapt to changes in tissue / material type or reduce task difficulty / workload. Examples of how the controller controls the manipulator 14 based on tool-anatomical interaction characteristics can be similar to those described above. For example, at 416, the controller may adjust the tool path TP of tool 20. At 418, the controller may modify the feed rate of tool 20. At 420, the controller may modify the pose of tool 20. At 422, the controller may actively move tool 20 away from the anatomical structure. At 424, the controller may modify the cutting rate and / or orientation of tool 20. At 426, the controller may stop all power supply to tool 20 or disable all tool operations. At 428, the controller may generate or modify virtual haptic settings for manipulator 14 and / or tool 20 based on estimated deflection, for example, to guide / constrain the tool to improve accuracy. Any of the example steps 416 to 428 can be performed simultaneously or in combination.

[0131] In one example, the machine learning model ML predicts the task difficulty level at time step T, and the controller then calculates a reduced feed rate for tool 20 to avoid congestion during cutting. The reduced feed rate is transmitted to tool controller 21 to dynamically slow down tool 20 at T+1.

[0132] In another example, at 430, the total estimated arm deflection (312, 406) is utilized (not input to the machine learning model ML”). Here, the controller can control the manipulator 14 and / or tool 20 based on the tool-anatomical interaction representation and the total estimated arm deflection. For example, the tool path generator 68 can utilize task difficulty classification (or reduced feed rate) and the total estimated deflection to adjust the tool path TP in real time or perform any of the tasks 416 to 428 described above.

[0133] In other examples, manipulator 14 can be coupled to a supplementary system that can be controlled based on the tool-anatomical interaction output of 410. For example, a flushing or aspiration system coupled to manipulator 14 and / or tool 20 can be used to flush and / or aspirate surgical sites, which can affect the tool-anatomical interaction. Based on the tool-anatomical interaction characterized at 410, the controller can control the flushing or aspiration system to modify the amount / frequency of flushing / aspiration. For example, if the machine learning model ML predicts that tool 20 is experiencing a heavy workload, flushing can be increased to reduce the tool workload. In other examples, navigation system 34 can be controlled based on the output of 410. For example, the tracking modality / performance can be modified according to the type / parameter of tissue detection. If the tool is a camera or equipped with illumination, the camera or illumination can be controlled based on the output of 410. Other examples are envisioned.

[0134] Several embodiments have been described in the foregoing description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive in nature. In view of the above teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.

Claims

1. A surgical system, comprising: a robotic arm including a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect an interaction force imparted to the surgical tool based on the interaction of the surgical tool with the anatomical structure; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the interaction force from the sensing system; input the pose of the surgical tool and the interaction force to a machine learning model; estimate a deflection of the robotic arm based on an output of the machine learning model; and control the robotic arm and / or the surgical tool based on the estimated deflection.

2. The surgical system of claim 1, wherein: the sensing system is further configured to detect a user-imparted force on the surgical tool; and the one or more controllers are further configured to: obtain the user-imparted force from the sensing system; input the user-imparted force to the machine learning model; estimate the deflection of the robotic arm based on the output of the machine learning model; and control the robotic arm and / or the surgical tool based on the estimated deflection.

3. The surgical system of claim 1, wherein: the sensing system is further configured to detect an interference force imparted to one or more of the links of the robotic arm; and the one or more controllers are further configured to: obtain the interference force from the sensing system; input the interference force to the machine learning model; estimate the deflection of the robotic arm based on the output of the machine learning model; and control the robotic arm and / or the surgical tool based on the estimated deflection.

4. The surgical system of claim 1, wherein: the sensing system is further configured to detect: an interference force imparted to one or more of the links of the robotic arm; and a user-imparted force on the surgical tool; and the one or more controllers are further configured to: obtain the interference force from the sensing system; obtain the user-imparted force from the sensing system; input the interference force and the user-imparted force to the machine learning model; estimate the deflection of the robotic arm based on the output of the machine learning model; and control the robotic arm and / or the surgical tool based on the estimated deflection.

5. The surgical system of claim 1, wherein the one or more controllers are configured to: control the robotic arm to move the surgical tool along a tool path to interact with the anatomical structure; and ​ ​ ​ ​ ​ by being configured to modify the tool path, control the robotic arm and / or the surgical tool based on the estimated deflection.

6. The surgical system of claim 1, wherein the one or more controllers are configured to: control the robotic arm to move the surgical tool at a feed rate to interact with the anatomical structure; and by being configured to modify the feed rate, control the robotic arm and / or the surgical tool based on the estimated deflection.

7. The surgical system of claim 1, wherein the one or more controllers are configured to: control the robotic arm to move the surgical tool to a commanded pose; and by being configured to adjust the commanded pose, control the robotic arm and / or the surgical tool based on the estimated deflection.

8. The surgical system of claim 1, wherein the one or more controllers are configured to: obtain an operational parameter of the surgical tool; input the operational parameter of the surgical tool and the estimated deflection into a second machine learning model; and characterize the interaction of the surgical tool with the anatomical structure based on an output of the second machine learning model.

9. The surgical system of claim 8, wherein the one or more controllers characterize the interaction by characterizing a type of the anatomical structure.

10. The surgical system of claim 8, wherein the one or more controllers characterize the interaction by characterizing a workload of the surgical tool.

11. The surgical system of claim 8, wherein the one or more controllers control the robotic arm and / or the surgical tool based on the characterized interaction by being configured to perform one or more of: modify a tool path of the surgical tool; modify a feed rate of the surgical tool; modify a cutting rate of the surgical tool; adjust a commanded pose of the surgical tool; suspend movement of the surgical tool; move the surgical tool away from the anatomical structure; and / or generate or modify a virtual haptic setting of the surgical tool.

12. The surgical system of claim 1, wherein: the surgical tool is configured to cut tissue of the anatomical structure; and the interaction force is based on the interaction of the surgical tool with the tissue during cutting of the tissue.

13. The surgical system of claim 1, wherein the sensing system comprises at least a force / torque sensor coupled to a distal link of the robotic arm.

14. The surgical system of claim 1, wherein the machine learning model is a deep learning model comprising at least two hidden layers.

15. The surgical system of claim 1, wherein the estimated deflection is a non-geometric deflection.

16. The surgical system of claim 1, wherein the one or more controllers are configured to trigger a deflection monitoring mode in response to detecting a particular condition, and wherein in the deflection monitoring mode the one or more controllers are configured to: input the pose of the surgical tool and the interaction force to the machine learning model; and estimate the deflection of the robotic arm based on the output of the machine learning model.

17. A method of operating a surgical system, the surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect an interaction force imparted to the surgical tool based on the interaction of the surgical tool with the anatomical structure; and one or more controllers coupled to the robotic arm and the sensing system, the method comprising the one or more controllers performing the following operations: obtaining a pose of the surgical tool based on kinematic data from the robotic arm; obtaining the interaction force from the sensing system; inputting the pose of the surgical tool and the interaction force to a machine learning model; estimating a deflection of the robotic arm based on an output of the machine learning model; and controlling the robotic arm based on the estimated deflection.

18. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; a sensing system configured to detect: an interaction force imparted to the surgical tool based on the interaction of the surgical tool with the anatomical structure; an interference force imparted to one or more of the links of the robotic arm; and a user-applied force on the surgical tool; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the interaction force, the interference force, and the user-applied force from the sensing system; input the pose of the surgical tool, the interaction force, the interference force, and the user-applied force to a machine learning model; and estimate a deflection of the robotic arm based on an output of the machine learning model.

19. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm; a sensing system configured to detect an interference force imparted to one or more of the links of the robotic arm; and a user-applied force on the surgical tool. ​ ​ ​ one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the disturbance force from the sensing system; input the pose of the surgical tool and the disturbance force to the machine learning model; and estimate a deflection of the robotic arm based on an output of the machine learning model.

20. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm; a sensing system configured to detect a user-applied force on the surgical tool; one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the user-applied force from the sensing system; input the pose of the surgical tool and the user-applied force to a machine learning model; and estimate a deflection of the robotic arm based on an output of the machine learning model.

21. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm, and configured to interact with an anatomical structure; a sensing system configured to detect an interaction force applied to the surgical tool based on an interaction of the surgical tool with the anatomical structure; and one or more controllers coupled to the robotic arm and the sensing system and configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the interaction force from the sensing system; input the pose of the surgical tool and the interaction force to a machine learning model; and estimate a deflection of the robotic arm based on an output of the machine learning model.

22. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm, and configured to interact with an anatomical structure; a sensing system configured to detect one or more forces applied to the surgical tool and / or the robotic arm; and one or more controllers coupled to the robotic arm and the sensing system and configured to: detect a presence of the one or more forces to trigger a deflection monitoring mode; and in the deflection monitoring mode, estimate a deflection of the robotic arm using a machine learning model.

23. A surgical system comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; one or more controllers coupled to the robotic arm and configured to: obtain an operational parameter of the surgical tool during the interaction of the surgical tool with the anatomical structure; estimate a deflection of the robotic arm caused by the interaction of the surgical tool with the anatomical structure; input the operational parameter of the surgical tool and the estimated deflection into a machine learning model; and characterize the interaction of the surgical tool with the anatomical structure based on an output of the machine learning model.

24. The surgical system of claim 23, wherein the one or more controllers characterize the interaction by characterizing a type of the anatomical structure.

25. The surgical system of claim 24, wherein the one or more controllers characterize the type of the anatomical structure as bone or soft tissue.

26. The surgical system of claim 23, wherein the one or more controllers characterize the interaction by characterizing a workload of the surgical tool.

27. The surgical system of claim 26, wherein the one or more controllers are configured to modify a feed rate of the surgical tool in response to characterizing the workload of the surgical tool.

28. The surgical system of claim 23, wherein the one or more controllers are configured to control the robotic arm and / or the surgical tool based on the characterized interaction.

29. The surgical system of claim 28, wherein the one or more controllers control the robotic arm and / or the surgical tool based on the characterized interaction by being configured to perform one or more of: modify a tool path of the surgical tool; modify a feed rate of the surgical tool; modify a cutting rate of the surgical tool; adjust a commanded pose of the surgical tool; suspend movement of the surgical tool; move the surgical tool away from the anatomical structure; and / or generate or modify a virtual haptic setting of the surgical tool.

30. The surgical system of claim 23, wherein the one or more controllers are configured to generate a notification or warning based on the characterized interaction.

31. The surgical system of claim 23, comprising: a sensing system configured to detect an interaction force imparted to the surgical tool based on the interaction of the surgical tool with the anatomical structure; and the one or more controllers are configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the interaction force from the sensing system; inputting the pose of the surgical tool and the interaction force to a second machine learning model; and estimating, based on an output of the second machine learning model, the deflection of the robotic arm caused by the interaction of the surgical tool with the anatomical structure.

32. The surgical system of claim 23, comprising: a sensing system configured to detect: an interaction force applied to the surgical tool based on an interaction of the surgical tool with the anatomical structure; an interference force applied to one or more of the links of the robotic arm; and a user-applied force on the surgical tool; and the one or more controllers are configured to: obtain a pose of the surgical tool based on kinematic data from the robotic arm; obtain the interaction force, the interference force, and the user-applied force from the sensing system; input the pose of the surgical tool, the interaction force, the interference force, and the user-applied force to a second machine learning model; and estimate a total deflection of the robotic arm based on an output of the second machine learning model.

33. The surgical system of claim 32, wherein the one or more controllers control the robotic arm and / or the surgical tool based on the characterized interaction and based on the total deflection of the robotic arm by being configured to perform one or more of: modifying a tool path of the surgical tool; modifying a feed rate of the surgical tool; modifying a cut rate of the surgical tool; adjusting a commanded pose of the surgical tool; suspending movement of the surgical tool; moving the surgical tool away from the anatomical structure; and / or generating or modifying a virtual haptic setting of the surgical tool.

34. The surgical system of claim 23, wherein: the surgical tool comprises a tool center point (TCP); and the one or more controllers obtain the operational parameters of the surgical tool during the interaction of the surgical tool with the anatomical structure by being configured to obtain the operational parameters of the TCP during the interaction.

35. The surgical system of claim 23, wherein the operational parameters of the surgical tool comprise one or more of: a velocity of the surgical tool; an average velocity of the surgical tool; an acceleration of the surgical tool; a displacement of the surgical tool; and / or a motor current of the surgical tool.

36. The surgical system of claim 23, wherein the robotic arm is configured to operate in a manual mode in which the robotic arm is configured to move the surgical tool in response to a force applied to the surgical tool by a user.

37. The surgical system of claim 23, wherein the robotic arm is configured to operate in an autonomous mode in which the robotic arm is configured to autonomously move the surgical tool along a tool path.

38. A surgical system, comprising: a robotic arm comprising a plurality of links and joints; a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; one or more controllers coupled to the robotic arm and configured to: estimate a deflection of the robotic arm caused by the interaction of the surgical tool with the anatomical structure; input the estimated deflection into a machine learning model; and characterize the interaction of the surgical tool with the anatomical structure based on an output of the machine learning model. a robotic arm comprising a plurality of links and joints; 39. A method of operating a surgical system, the surgical system comprising: a surgical tool supported by the robotic arm and movable by the robotic arm and configured to interact with an anatomical structure; and one or more controllers coupled to the robotic arm, the method comprising the one or more controllers performing the following operations: obtaining an operational parameter of the surgical tool during the interaction of the surgical tool with the anatomical structure; estimating a deflection of the robotic arm caused by the interaction of the surgical tool with the anatomical structure; inputting the operational parameter of the surgical tool and the estimated deflection into a machine learning model; and characterizing the interaction of the surgical tool with the anatomical structure based on an output of the machine learning model. ​ ​

Images