Robotic surgical systems and methods employing machine learning models to characterize tool interactions

Abstract

Surgical systems and methods involving machine learning models to characterize tool interactions. A tool may be coupled to the robotic manipulator and configured to manipulate tissue. The sensing system measures a displacement of the tool, a speed of the tool, and an interaction force applied to the tool. A control system controls the manipulator to move the tool to interact with the tissue. In response to interaction of the tool with tissue over time, the control system obtains measurements of displacement, speed, and interaction force from the sensing system. The control system implements a machine-learned model for receiving and processing the measurements to estimate stiffness / damping parameters of the tissue, and evaluating variations in the estimated stiffness / damping parameters to generate a characterization regarding the tissue or tool interaction with the tissue. The control system controls the manipulator and / or the tool based on the characterization.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and all benefits to U.S. Provisional Patent Application No. 63 / 700,968, filed September 30, 2024, and U.S. Provisional Patent Application No. 63 / 749,842, filed January 27, 2025, the entire contents of each of which are hereby incorporated by reference. Background Technology

[0003] Surgical robotic systems typically include robotic arms that support and move surgical instruments to assist in performing surgical procedures at the surgical site. These surgical instruments are often used to manipulate tissue. For example, in robotic joint replacement surgery, the robotic arm controls surgical instruments to sculpt or remove bone in preparation for implantation. The surgical site is a confined workspace and is limited by the size of the incision, making robotic navigation at the site difficult. The target bone may be surrounded by cartilage, soft tissues (e.g., ligaments, tendons, adipose tissue, blood vessels, etc.), tissue retractors, etc.

[0004] Despite the complexity of the objects / tissues at the surgical site, conventional surgical robots lack the capacity to understand the true nature of the objects / tissues with which the tool interacts. A major reason for this limitation is that conventional surgical robots fail to identify, let alone interpret, the unknown contact dynamics resulting from tool-tissue interactions. These unknown contact dynamics can be caused by complex factors such as tissue properties, friction, contact forces, or geometric variations. For example, existing surgical robots fail to characterize the nature of the tissue with which the tool interacts or to differentiate between different tissue types. Existing surgical robot control systems are typically non-dynamic and non-adaptive, relying instead on predefined control models to control the tool relative to the tissue in a "one-size-fits-all" manner. Such conventional surgical robot control systems generally include simple feedback control in response to tool position or tool load relative to the tissue. This approach is based solely on tool measurements and fails to make any intelligent predictions about the contact dynamics occurring between the tool and the tissue.

[0005] The accuracy of surgical instruments, particularly relative to the surgical site, is crucial. However, existing surgical robots are susceptible to tool inaccuracies due to the lack of characterization of unknown tool-tissue contact dynamics. Inaccuracies of just a few millimeters can lead to suboptimal control or performance of the surgical robot and / or surgical instruments. Therefore, such inaccuracies can interfere with the surgeon, or worse, result in intraoperative complications or suboptimal surgical outcomes for the patient. Furthermore, the lack of resolution of unknown tool-tissue contact dynamics means that existing surgical robots may cause surgical instruments to erroneously contact prohibited tissue areas or types. This limitation further restricts the ability of existing surgical robots to perform higher-order tasks of greater complexity. Summary of the Invention

[0006] The present invention is presented in a simplified form, with the following detailed description of selected concepts. The present invention is not intended to limit the scope of the claimed subject matter, nor to define the key or essential features of the claimed subject matter.

[0007] According to a first aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize features or parameters of the tissue.

[0008] According to a second aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize the type of tissue.

[0009] According to a third aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to distinguish between different tissue types.

[0010] According to a fourth aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize parameters of the tissue, said parameters including one or more of the following: geometry, size, shape, depth, and thickness.

[0011] According to a fifth aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize parameters of the tissue, said parameters including one or more of the following: density, stiffness, hardness, and softness.

[0012] According to a sixth aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize one or more of the following: the tissue's environment, the tissue's hierarchical structure, and an identification that the tissue is an embedded object.

[0013] According to a seventh aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to characterize the interaction type of the surgical instrument, wherein the interaction type includes soft tissue interaction, skeletal tissue interaction, or non-contact interaction.

[0014] According to an eighth aspect, a surgical system is provided, comprising: a robotic manipulator including a plurality of links and joints; a surgical instrument coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: displacement of the surgical instrument, velocity of the surgical instrument, and interaction force applied to the surgical instrument; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical instrument to interact with the tissue; obtain measurements of displacement, velocity, and interaction force from the sensing system in response to the interaction between the surgical instrument and the tissue; and input the measurements into a machine learning model configured to estimate stiffness and / or damping parameters of the tissue.

[0015] According to a ninth aspect, a surgical system is provided, comprising: a robotic manipulator including a plurality of links and joints; a surgical instrument coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: displacement of the surgical instrument, velocity of the surgical instrument, and interaction force applied to the surgical instrument; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical instrument to interact with tissue; obtain measurements of displacement, velocity, and interaction force from the sensing system in response to the interaction between the surgical instrument and tissue; input the measurements into a machine learning model configured to predict interaction representations; and control the robotic manipulator and / or the surgical instrument based on the interaction representations.

[0016] According to a tenth aspect, a surgical system is provided, comprising: a robotic manipulator including a plurality of links and joints; a surgical instrument coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: displacement of the surgical instrument, velocity of the surgical instrument, and interaction force applied to the surgical instrument; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical instrument to interact with the tissue; obtain measurements of displacement, velocity, and interaction force from the sensing system in response to the interaction of the surgical instrument with the tissue over time; and implement a machine learning model configured to: receive and process the measurements to estimate stiffness and damping parameters of the tissue; and monitor changes in the estimated stiffness and damping parameters to predict tissue characterization; and wherein the control system is configured to control the robotic manipulator and / or the surgical instrument based on tissue characterization.

[0017] According to an eleventh aspect, a surgical manipulator is provided for controlling the interaction between a surgical instrument and tissue. The control system acquires sensed measurements during the interaction between the surgical instrument and tissue and applies the sensed measurements to a machine learning model configured to identify foreign objects located on or embedded in the tissue. The foreign object may include one or more of the following: an existing implant, a metal block, a surgical fastener, and another surgical instrument.

[0018] According to a twelfth aspect, a surgical system is provided, comprising: a robotic manipulator including a plurality of links and joints; a surgical instrument coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: displacement of the surgical instrument, velocity of the surgical instrument, and interaction force applied to the surgical instrument; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical instrument to interact with tissue; obtain measurements of displacement, velocity, and interaction force from the sensing system in response to the interaction between the surgical instrument and tissue; input the measurements into a machine learning model configured to predict a representation of a foreign object located on or embedded in the tissue; and control the robotic manipulator and / or the surgical instrument based on the representation of the foreign object. The foreign object may include one or more of the following: an existing implant, a metal block, a surgical fastener, and another surgical instrument.

[0019] Also provided are: a computer-implemented method for performing any step of the steps implemented by a surgical system or control system of any one or more of the foregoing aspects; a non-transitory computer-readable medium or computer program product comprising instructions configured, when executed by one or more processors, to implement a surgical system or control system of any one or more of the foregoing aspects; a control system of any one or more of the foregoing aspects; and a surgical system of any one or more of the foregoing aspects further comprising a navigation system for tracking tissue.

[0020] Any aspect described above can be combined in whole or in part. Any aspect above can be used in part or in part with any of the following implementations:

[0021] The machine learning model can be configured to estimate the stiffness parameters of an organization (undamped) or vice versa. The machine learning model may include a first neural network. The first neural network may include an input layer, hidden layers, and an output layer. The input layer of the first neural network receives measurements. The output layer of the first neural network outputs the estimated stiffness and damping parameters. Optionally, the machine learning model may include or be connected to a first Long Short-Term Memory (LSTM) or Recurrent Neural Network (RNN). The first LSTM / RNN may be connected to the input layer of the first neural network. The first LSTM / RNN receives measurements over time. The first LSTM / RNN can modify weights and biases to learn long-term dependencies between measurements and selectively filter measurements. The first LSTM / RNN can provide the filtered measurements to the input layer of the first neural network. The estimated stiffness and damping parameters may be embeddings of lower dimensionality than the measurements.

[0022] The machine learning model may include a second neural network. The second neural network may include an input layer, hidden layers, and an output layer. The input layer of the second neural network may receive the estimated stiffness and damping parameters. The output layer of the second neural network may output a tissue representation. The machine learning model may optionally include or be connected to a second LSTM / RNN. The second LSTM / RNN may be connected to the input layer of the second neural network. The second LSTM / RNN may receive the estimated stiffness and damping parameters from the output layer of the first neural network over time. The second LSTM / RNN may modify the weights and biases to learn the long-term dependencies between the estimated stiffness and damping parameters and selectively filter the estimated stiffness and damping parameters. The second LSTM / RNN may provide the filtered estimated stiffness and damping parameters to the input layer of the second neural network. The second LSTM / RNN may further receive measurements of the velocity of the surgical instrument over time. The tissue representation may be represented by vectors and may have a higher dimension than the estimated stiffness and damping parameters.

[0023] Organizational characterization can include parameters of the organization. Parameters can include geometric features, material properties, material type, etc. Organizational parameters can include one or more of the following: geometry, size, shape, depth, and thickness. Organizational parameters can include one or more of the following: density, stiffness, hardness, softness, smoothness, and roughness. Organizational characterization can include one or more of the following: the environment of the organization, the layering of the organization, and the identification of the organization as an embedded object.

[0024] Tissue characterization may include tissue type. Tissue type may include one or more of the following: bone, osteophytes, cartilage, soft tissue, muscle, ligament tissue, tendon tissue, blood vessels, healthy tissue, and malignant tissue. A machine learning model can monitor changes in estimated stiffness and damping parameters to predict a first tissue type and then predict a second tissue type. The first tissue type can be any tissue type described, and the second tissue type can be any other tissue type described. For example, the first tissue type can be bone, and the second tissue type can be soft tissue. The first tissue type can be bone, and the second tissue type can be cartilage. The first tissue type can be healthy tissue, and the second tissue type can be malignant tissue.

[0025] The control system can record the position of the surgical instrument in response to its interaction with the tissue over time. The control system can then use the recorded position to register tissue characterization to the location on the tissue.

[0026] The control system can record the position of a surgical instrument in response to its interaction with a first tissue type and a second tissue type over time. The control system can use the recorded position to register the first and second tissue types to positions on the tissue. The control system can generate a virtual boundary for delineating a first region of the tissue including the first tissue type and a second region of the tissue including the second tissue type. The control system can register the virtual boundary to the tissue. The control system can use the virtual boundary to constrain the movement or manipulation of a robotic manipulator and / or surgical instrument relative to the virtual boundary.

[0027] A machine learning model monitors changes in estimated stiffness and damping parameters to predict whether the tissue type is bone. A control system records the position of the surgical instrument in response to its interaction with the bone over time. The control system uses the recorded position to generate a 3D surface model of the bone. The control system then registers the 3D surface model to the bone.

[0028] The control system can control the robotic manipulator and / or surgical tool based on tissue representation by being configured to perform one or more of the following: modifying the tool path of the surgical tool; modifying the feed rate of the surgical tool; modifying the cutting speed of the surgical tool; adjusting the command pose of the surgical tool; pausing the movement of the surgical tool; moving the surgical tool toward or away from the tissue; and / or generating or modifying the virtual haptic settings of the surgical tool.

[0029] The measured displacement of the surgical instrument can indicate the displacement of the surgical instrument performed by the surgical instrument. The measured velocity of the surgical instrument can indicate the velocity of the instrument during the displacement performed by the surgical instrument. The control system can calculate the deformation of the tissue based on the measured displacement of the surgical instrument and the measured interaction force of the surgical instrument. A machine learning model can receive and process the deformation value to estimate the stiffness and damping parameters of the tissue and can use the deformation value instead of the displacement value. The measured interaction force can be measured directly or indirectly at the surgical instrument. The sensing system may include at least one force / torque sensor to measure the interaction force applied to the surgical instrument. At least one force / torque sensor may be coupled to one or both of the following: a distal link of a robotic manipulator and the surgical instrument.

[0030] Manipulators may include robotic arms with motorized joints. Manipulators may also include robotic arms with manually adjustable passive joints. Surgical tools are configured to manipulate tissue. Surgical tools may be saws, drills, grinders, planers, burrs, impactors, pin drivers, screwdrivers, tool guides, tool holders, scalpels, catheters, etc. Interactive forces are applied to the surgical tool based on the interaction between the surgical tool and the tissue during tissue cutting.

[0031] Any of the implementations described can be combined in part or in whole. Attached Figure Description

[0032] The advantages of this disclosure will be readily apparent, as they will be better understood by referring to the following detailed description taken in conjunction with the accompanying drawings.

[0033] Figure 1 This is a perspective view of a robotic surgical system based on one implementation method.

[0034] Figure 2 This is a block diagram of an example control system used to control a robotic surgical system.

[0035] Figure 3 It is a functional block diagram of a module implemented by a control system according to an implementation method.

[0036] Figure 4 Example output of the boundary generator is shown.

[0037] Figure 5 Example output of the path generator is shown.

[0038] Figure 6 This is a block diagram of an example control scheme based on one implementation of a manipulator.

[0039] Figure 7It is a block diagram of a control system for a machine learning model used to characterize tool interactions, based on an example of an implementation method.

[0040] Figure 8A It is an example organization model with embedded objects, and a spring damper model of the organization and embedded objects generated based on the applied interaction force.

[0041] Figure 8B It is a set of equations based on an example that can be used by a machine learning model to characterize tool interactions.

[0042] Figure 9 It is a block diagram of a machine learning model, including example representations, for interaction with representation tools based on an implementation method.

[0043] Figure 10 This is a block diagram of the interaction evaluator portion of an example machine learning model, where the interaction evaluator is implemented using a neural network.

[0044] Figure 11 It is a graph that illustrates the representation of the type of an example object, which can be predicted by a machine learning model, based on an example.

[0045] Figure 12 This is a simplified diagram illustrating how a robotic manipulator interacts with multi-layered tissue, including embedded objects, using surgical tools, based on an example.

[0046] Figures 13A to 13F This is an example of a sensing system based on and Figure 12 The interactive examples of multi-layered organizations provide charts of various measurement results that can be obtained. Detailed Implementation

[0047] I. Overview of the Example System

[0048] refer to Figure 1 An example is illustrated by a surgical system 10. The surgical system 10 can be used to treat surgical sites or anatomical volumes (A) of a patient 12, such as treating bone or soft tissue. Figure 1 In the middle, patient 12 is undergoing surgical procedures. Figure 1The anatomical structures included are the femur (F) and tibia (T) of patient 12. Surgical procedures may involve tissue removal or other forms of treatment. Treatment may include cutting, coagulation, damaging tissue, other in-situ tissue treatments, etc. In some examples, surgical procedures involve partial or total knee or hip replacement surgery, shoulder replacement surgery, spinal surgery, or ankle surgery. In some examples, system 10 is designed to remove material to be replaced by a surgical implant, such as hip and knee implants, including unicompartmental, bicompartmental, multicompartmental, or total knee implants. Some of these types of implants are illustrated in U.S. Patent Application Publication 2012 / 0330429 entitled “Prosthetic Implant and Method of Implantation,” the disclosure of which is hereby incorporated by reference. The surgical system 10 and techniques disclosed herein may be used to perform other surgical or non-surgical procedures, or may be used for industrial applications or other applications utilizing robotic systems.

[0049] The surgical system 10 includes a manipulator 14 or a robotic manipulator. The manipulator 14 has a base 16 and a plurality of links 18. A manipulator trolley 17 supports the manipulator 14, such that the manipulator 14 is secured to the manipulator trolley 17. The links 18 together form one or more arms of the manipulator 14. The manipulator 14 may have a tandem arm configuration (e.g., Figure 1 (As shown), parallel arm configuration, or any other suitable manipulator configuration. In other examples, more than one manipulator 14 may be used in a multi-arm configuration. In other configurations, manipulator 14 may include part or all of an adjustable arm (AA) with manually adjustable passive joints. The user may optionally lock the joints of the adjustable arm (AA) in place. In other cases, joint motors may be used to actively drive some or all of the joints of the adjustable arm (AA).

[0050] Manipulator 14 may include passive (manually articulated) joints, such as planar-extending joints. For example, a passive joint may be the most distal joint attached to a robotic arm that includes multiple active joints. A planar-extending joint may support a tool, such as a saw blade, to allow a user to manually move the blade along the plane. In this case, the tool is mechanically constrained to the plane by the mechanical constraints of the passive joint, while the pose of the plane is actively constrained by the active joints of the robotic arm. In other cases, one or more joints of manipulator 14 may be controlled to constrain a surgical tool within a specific degree of freedom while allowing the surgical tool to move freely passively in other degrees of freedom or in other ways, such as rotation about a point or selected linear motion.

[0051] Manipulator 14 can exhibit kinematic redundancy, where the tool position is limited by up to 6 DOF, but the robotic arm can operate over a range exceeding 6 DOF. This redundancy allows manipulator 14 to utilize zero-space control, whereby the joints of manipulator 14 can change position while maintaining the pose of tool 20. Zero-space control can also be used to avoid singularities.

[0052] exist Figure 1 In the example shown, the manipulator 14 includes multiple joints J and multiple joint encoders 19 located at the joints J for determining the position data of the joints J. For simplicity, Figure 1 The illustration shows only one joint encoder 19, but other joint encoders 19 can be illustrated similarly. According to one example, the manipulator 14 has six joints J1-J6 that implement at least six degrees of freedom (DOF) of the manipulator 14. However, the manipulator 14 may have any number of degrees of freedom and any suitable number of joints J, and may have redundant joints.

[0053] Manipulator 14 does not necessarily require joint encoder 19, but may instead utilize motor encoders present on the motors at each joint J. Furthermore, manipulator 14 does not necessarily require rotary joints, but may instead utilize one or more prismatic joints. Any suitable combination of joint types is contemplated.

[0054] The base 16 of the manipulator 14 is part of a fixed reference coordinate system that provides a reference coordinate system for the manipulator 14 or, more generally, other components of the surgical system 10. The origin of the manipulator coordinate system MNPL is defined at the fixed reference point of the base 16. The base 16 may be defined relative to any suitable part of the manipulator 14, such as one or more links in the linkage 18. Alternatively or additionally, the base 16 may be defined relative to the manipulator trolley 17, such as at the location where the manipulator 14 is physically attached to the manipulator trolley 17. In one example, the base 16 is defined at the intersection of the axes of joints J1 and J2. Thus, although joints J1 and J2 are moving parts in reality, the intersection of the axes of joints J1 and J2 remains a virtual fixed reference pose that provides both fixed position and orientation references and does not move relative to the manipulator 14 and / or the manipulator trolley 17.

[0055] In other examples, the manipulator 14 may be a handheld manipulator, wherein the base 16 is the base portion of the tool (e.g., a portion held by the user's hand) and the tool tip is movable relative to the base portion. The base portion has a tracked reference coordinate system, and the tool tip has a tool tip coordinate system calculated relative to the reference coordinate system (e.g., calculated via motor and / or joint encoders and forward kinematics). Since the pose of the tool tip relative to the path can be determined, movement of the tool tip can be controlled to follow the path. The handheld manipulator 14 may be as described and shown in US20230255701 entitled "Systems and Methods for Guiding Movement of a Handheld Medical Robotic Instrument," the entire disclosure of which is hereby incorporated by reference.

[0056] Manipulator 14 and / or manipulator trolley 17 house manipulator controller 26 or other types of control units. Manipulator controller 26 may include one or more computers, or any other suitable form of controller that directs the movement of manipulator 14. Manipulator controller 26 may have a central processing unit (CPU), graphics processing unit (GPU) and / or other processors, memory, and storage devices. Manipulator controller 26 is loaded with software as described below. Processor may include one or more processors for controlling the operation of manipulator 14. Processor may be any type of microprocessor, multiprocessor, and / or multicore processing system. Manipulator controller 26 may additionally or alternatively include one or more microcontrollers, field-programmable gate arrays, system-on-a-chip, discrete circuits, and / or other suitable hardware, software, or firmware capable of implementing the functions described herein. The term processor is not intended to limit any implementation to a single processor. Manipulator 14 may also include a user interface UI with one or more displays and / or input devices (e.g., buttons, keyboard, mouse, microphone (voice-activated), gesture control devices, touchscreen, etc.).

[0057] Tool 20 is coupled to manipulator 14 and is movable relative to base 16 to interact with anatomical structures in certain modes. Tool 20 is a physical and surgical tool and, in some implementations, is or forms part of an end effector 22 supported by manipulator 14. Tool 20 can be gripped by a user. One possible arrangement of manipulator 14 and tool 20 is described in U.S. Patent 9,119,655 entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is incorporated herein by reference. Manipulator 14 and tool 20 may be arranged in alternative configurations. Tool 20 may be similar to the tool shown in U.S. Patent Application Publication 2014 / O276949 entitled “End Effector of a Surgical Robotic Manipulator,” filed March 15, 2014, which is incorporated herein by reference.

[0058] Tool 20 may include an energy applicator 24 designed to contact and remove tissue from patient 12 at a surgical site. In one example, energy applicator 24 is a drill 25. Drill 25 may be substantially spherical and include a spherical center, radius (r), and diameter. Alternatively, energy applicator 24 may be a drill bit, saw blade, ultrasonic vibrating tip, etc. Tool 20 and / or energy applicator 24 may include any geometric features such as perimeter, circumference, radius, diameter, width, length, volume, area, surface / plane, range of motion envelope (along any one or more axes), etc. Geometric features may be considered to determine how tool 20 is positioned relative to tissue at the surgical site to perform the desired treatment. In some embodiments described herein, for convenience and ease of illustration, a spherical drill with a tool center point (TCP) will be described, but it is not intended to limit tool 20 to any particular form. In other examples, tool 20 does not include energy applicator 24. For example, tool 20 may be a grooving cutting guide for sawing, a guide tube for receiving another tool, etc.

[0059] Tool 20 may include a tool controller to control the operation of tool 20, such as controlling the power supplied to the tool (e.g., power supplied to a rotary motor of tool 20), controlling the movement of tool 20, controlling flushing / suction and / or similar operations of tool 20. The tool controller may communicate with manipulator controller 26 or other components. Tool 20 may also include a user interface (UI) with one or more displays and / or input devices (e.g., buttons, keyboard, mouse, microphone (voice activated), gesture control, touchscreen, etc.). For example, one of the user input devices on the user interface UI of tool 20 may be a tool input device (e.g., a switch or other form of user input device) having a first input state and a second input state (see...). Figure 1 The tool input can be actuated by the user (e.g., pressed and held) to be placed in a first input state and can be released to be placed in a second input state. The tool 20 may have a handle on which the tool input is located. In some forms, the tool input is a presence detector that detects the presence of the user's hand, such as an instantaneous contact switch that toggles between on / off states, a capacitive sensor, an optical sensor, etc. The tool input is thus configured such that the first input state indicates that the user is actively engaging the tool 20 and the second input state indicates that the user has released the tool 20. The tool input can be a continuously activated device, i.e., an input that must be continuously actuated to allow the tool 20 to move in manual or semi-autonomous mode depending on which user input is actuated. For example, when the user continuously actuates the tool input and manual mode is enabled, the manipulator 14 will move in response to the input force and torque applied by the user, and the control system 60 will impose a virtual boundary VB to protect the patient's anatomy. When the tool input is released, the input from the force / torque sensor S can be disabled, so that the manipulator 14 no longer responds to the force and torque applied to the tool 20 by the user.

[0060] Manipulator controller 26 controls the state (position and / or orientation) of tool 20 (e.g., TCP) relative to a coordinate system such as the manipulator coordinate system MNPL. Manipulator controller 26 can control the (linear or angular) velocity, acceleration, or other kinematic derivatives of tool 20. In one example, the tool center point (TCP) is a predetermined reference point defined at energy applicator 24. TCP has a known or computable (i.e., not necessarily static) pose relative to other coordinate systems. The geometry of energy applicator 24 is known in or defined relative to the TCP coordinate system. TCP may be located at the spherical center of drill 25 of tool 20, such that tracking is done at only one point. TCP may be defined in several ways depending on the configuration of energy applicator 24. Manipulator 14 may employ a joint / motor encoder or any other non-encoder position sensing method to enable determination of the TCP pose. Manipulator 14 may use joint measurements to determine the TCP pose and / or employ techniques to directly measure the TCP pose. Control of tool 20 is not limited to the center point. For example, any suitable primitive, grid, etc. can be used to represent tool 20.

[0061] The surgical system 10 also includes a navigation system 32. An example of the navigation system 32 is described in U.S. Patent No. 9,008,757, filed September 24, 2013, entitled “Navigation System Including Optical and Non-Optical Sensors,” which is incorporated herein by reference. The navigation system 32 tracks the movement of various objects. Such objects include, for example, manipulators 14, tools 20, and anatomical structures (e.g., femur F and tibia T). The navigation system 32 tracks these objects to acquire state information for each object relative to the (navigation) locator coordinate system LCLZ. Transformations can be used to transform the coordinates in the locator coordinate system LCLZ to the manipulator coordinate system MNPL and / or vice versa.

[0062] The navigation system 32 includes a cart assembly 34 housing a navigation controller 36, and / or other types of control units. The navigation user interface (UI) operatively communicates with the navigation controller 36. The navigation user interface includes one or more displays 38. The navigation system 32 can use the one or more displays 38 to display a graphical representation of the relative status of the tracked object to the user. The navigation user interface (UI) also includes one or more input devices to input information into the navigation controller 36 or otherwise select / control certain aspects of the navigation controller 36. Such input devices include interactive touchscreen displays. However, input devices may include any one or more of buttons, keyboards, mice, microphones (voice-activated), gesture controls, etc.

[0063] The navigation system 32 also includes a locator 44 coupled to the navigation controller 36. In one example, the locator 44 is an optical locator and includes a camera unit 46. The camera unit 46 has an external housing 48 that houses one or more optical sensors 50. The locator 44 may include its own locator controller 49 and may also include a video camera VC.

[0064] Navigation system 32 includes one or more trackers. In one example, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker 54, and a second patient tracker 56. Figure 1 In the illustrated example, the manipulator tracker is rigidly attached to tool 20 (i.e., tracker 52A), the first patient tracker 54 is securely attached to the femur F of patient 12, and the second patient tracker 56 is securely attached to the tibia T of patient 12. In this example, patient trackers 54 and 56 are securely attached to segments of bone. A pointer tracker PT is securely attached to a pointer or probe P used to calibrate anatomical structures to the locator coordinate system LCLZ. Manipulator trackers 52A and 52B may be attached to any suitable component of manipulator 14 other than tool 20, such as base 16 (i.e., tracker 52B), or any one or more links 18 of manipulator 14. Trackers 52A, 52B, 54, 56, and PT may be secured to their respective components in any suitable manner. For example, the tracker can be rigidly fixed, flexibly connected (fiber optic), or not physically connected at all (ultrasound), as long as there is a suitable (supplementary) way to determine the relationship (measurement result) between the respective tracker and the object associated with it.

[0065] Any one or more trackers in the tracker may include an active marker 58. The active marker 58 may include a light-emitting diode (LED). Alternatively, trackers 52A, 52B, 54, 56, and PT may have passive markers, such as reflectors that reflect light emitted from camera unit 46. Other suitable markers not specifically described herein may be utilized.

[0066] Positioner 44 tracks trackers 52A, 52B, 54, 56, and PT to determine the state of each of the trackers 52A, 52B, 54, 56, and PT, which respectively correspond to the state of the object to which they are attached. Positioner 44 can perform known triangulation techniques to determine the state of trackers 52, 54, 56, and PT and the associated object. Positioner 44 provides the state of trackers 52A, 52B, 54, 56, and PT to navigation controller 36. In one example, navigation controller 36 determines the state of trackers 52A, 52B, 54, 56, and PT and transmits it to manipulator controller 26. As used herein, the state of an object includes, but is not limited to, data defining the position and / or orientation of the tracked object, or equivalents / derivatives of position and / or orientation. For example, the state can be the pose of the object and may include linear velocity data and / or angular velocity data, etc.

[0067] The navigation controller 36 may include one or more computers, or any other suitable form of controller. The navigation controller 36 may have a central processing unit (CPU), graphics processing unit (GPU) and / or other processors, non-transitory memory, and storage devices. The processor may be any type of processor, microprocessor, or multiprocessor system. The navigation controller 36 is loaded with software. For example, the software converts signals received from the locator 44 into data representing the position and orientation of the tracked object. The navigation controller 36 may additionally or alternatively include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuitry, and / or other suitable hardware, software, or firmware capable of performing the functions described herein. The term processor is not intended to limit any implementation to a single processor.

[0068] Although one example of the navigation system 32 is shown as employing triangulation techniques to determine the state of an object, the navigation system 32 may have any other suitable configuration for tracking the manipulator 14, the tool 20, and / or the patient 12.

[0069] In another example, navigation system 32 and / or locator 44 are ultrasound-based. For example, navigation system 32 may include an ultrasound imaging device coupled to navigation controller 36. The ultrasound imaging device images any of the aforementioned objects (e.g., manipulator 14, tool 20, and / or patient 12) and generates status signals to navigation controller 36 based on the ultrasound images. The ultrasound images may be 2D, 3D, or a combination of both. Navigation controller 36 can process the images in near real-time to determine the state of the object. The ultrasound imaging device may have any suitable configuration and may differ from, for example, […]. Figure 1 The camera unit 46 shown.

[0070] In another example, navigation system 32 and / or locator 44 are radio frequency (RF) based. For example, navigation system 32 may include an RF transceiver coupled to navigation controller 36. Manipulator 14, tool 20, and / or patient 12 may include an RF transmitter or repeater attached thereto. The RF transmitter or repeater may be passive or actively powered. The RF transceiver transmits RF tracking signals based on RF signals received from the RF transmitter and generates status signals to navigation controller 36. Navigation controller 36 may analyze the received RF signals to correlate with relevant statuses. The RF signals may have any suitable frequency. The RF transceiver may be positioned at any suitable location to effectively track objects using the RF signals. Furthermore, the RF transmitter or repeater may have any suitable structural configuration, which may be largely consistent with... Figure 1 The trackers shown are different: 52A, 52B, 54, 56, and PT.

[0071] In yet another example, navigation system 32 and / or locator 44 are electromagnetic. For example, navigation system 32 may include an EM transceiver coupled to navigation controller 36. Manipulator 14, tool 20, and / or patient 12 may include EM components attached thereto, such as any suitable magnetic tracker, electromagnetic tracker, inductive tracker, etc. The tracker may be passive or actively powered. The EM transceiver generates an EM field and generates a status signal to navigation controller 36 based on the EM signals received from the tracker. Navigation controller 36 may analyze the received EM signals to correlate with the relevant status. Similarly, such navigation system examples may have... Figure 1 The navigation system 32 shown has different structural configurations.

[0072] Navigation system 32 may have any other suitable components or structures not specifically listed herein. Furthermore, any of the techniques, methods, and / or components described above with respect to the illustrated navigation system 32 may be implemented or provided for any of the other examples of navigation system 32 described herein. For example, navigation system 32 may utilize only inertial tracking or any combination of tracking techniques, and may additionally or alternatively include fiber-optic tracking, machine vision tracking, etc.

[0073] The pointer P can be handheld or optionally attached to an adjustable arm (AA), such as... Figure 1 As shown. The pointer P may include a force / torque sensor S′ configured to measure the force / torque applied to the pointer tip. The pointer P may also include, for example, a controller housed within the pointer body to process the measurement results from the force / torque sensor S′. The measurement results from the pointer P can be transmitted to the control system 60 using a wired or wireless connection.

[0074] refer to Figure 2The surgical system 10 includes a control system 60, which, among other components, includes a manipulator controller 26, a navigation controller 36, and a tool controller 21. The control system 60 also includes... Figure 3 The software program and software module are shown. The software module may be part of one or more programs that operate on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof to process data to assist in the control of the surgical system 10. The software program and / or module includes computer-readable instructions stored in non-transitory memory 64 on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof, executable by one or more processors 70 of the controllers 21, 26, 36. Memory 64 may be any suitable memory configuration, such as RAM, non-volatile memory, etc., and may be implemented locally or from a remote database. Additionally, software modules for prompting and / or communicating with the user may form part of one or more programs and may include instructions stored in memory 64 on the manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof. The user may interact with any of the input devices of the navigation user interface UI or other user interface UIs to communicate with the software module. The user interface software may run on a device separate from the manipulator controller 26, navigation controller 36, and / or tool controller 21.

[0075] The control system 60 may include any suitable configuration of input devices, output devices, and processing devices adapted to perform the functions and methods described herein. The control system 60 may include a manipulator controller 26, a navigation controller 36, or a tool controller 21, or any combination thereof, or may include only one of these controllers. These controllers may be accessible via, for example... Figure 2 The wired bus or communication network shown communicates wirelessly or otherwise. Control system 60 may also be referred to as a controller. Control system 60 may include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuits, sensors, displays, user interfaces, indicators, and / or other suitable hardware, software, or firmware capable of performing the functions described herein.

[0076] refer to Figure 3 The software used by the control system 60 includes a boundary generator 66. For example... Figure 4As shown, boundary generator 66 is a software program or module that generates virtual boundaries VB for the movement and / or manipulation of constraint tool 20. The virtual boundary VB can be one-dimensional, two-dimensional, or three-dimensional, and can include points, lines, axes, trajectories, planes, or other shapes, including complex geometries. In some embodiments, the virtual boundary VB is a surface defined by a triangular mesh. Such a virtual boundary VB can also be referred to as a virtual object. The virtual boundary VB can be defined relative to an anatomical model AM, such as a 3-D skeletal model. Figure 4 In the example, the virtual boundary VB is a planar boundary used to define the five planes of the entire knee implant and is associated with a 3-D model of the femur F. The anatomical model AM is registered to one or more patient trackers 54, 56, such that the virtual boundary VB becomes associated with the anatomical model AM. The virtual boundary VB can be implant-specific, such as defined based on the size, shape, volume, etc., of the implant, and / or patient-specific, such as defined based on the patient's anatomy. The virtual boundary VB can be a boundary created preoperatively, intraoperatively, or a combination thereof. In other words, the virtual boundary VB can be defined before the start of the surgical procedure, during the surgical procedure (including during tissue removal), or a combination thereof. In any case, the control system 60 obtains the virtual boundary VB by storing / retrieving the virtual boundary VB from memory, obtaining the virtual boundary VB from memory, creating the virtual boundary VB preoperatively, creating the virtual boundary VB intraoperatively, etc.

[0077] Manipulator controller 26 and / or navigation controller 36 track the state of tool 20 relative to a virtual boundary VB. In one example, the state of TCP is measured relative to the virtual boundary VB to determine, via virtual simulation 88, the tactile force to be applied to the virtual rigid body model such that tool 20 maintains a desired positional relationship with the virtual boundary VB (e.g., does not move beyond the virtual boundary). The results of virtual simulation 88 are commanded to manipulator 14. Control system 60 controls / positions manipulator 14 in a manner that simulates the response of a physical handheld device in the presence of physical boundaries / obstacles. Boundary generator 66 may be implemented on manipulator controller 26. Alternatively, boundary generator 66 may be implemented on other components such as navigation controller 36.

[0078] refer to Figure 3 and Figure 5Path generator 68 is another software program or module executed by control system 60. In one example, path generator 68 is executed by manipulator controller 26. Path generator 68 generates a tool path TP for tool 20 to traverse. Tool path TP may include multiple path segments PS, or may include a single path segment PS. Path segments PS may be straight segments, curved segments, or combinations thereof. Tool path TP may be defined relative to manipulator 14 coordinate system MNPL, locator coordinate system LCLZ, tool 20 coordinate system, anatomical coordinate system, or any combination thereof. Tool path TP may be virtually attached to the coordinate system of the corresponding object, such that if the object is moved, tool path TP will move accordingly. Tool path TP may be implant-specific, for example defined based on the size, shape, volume, etc. of the implant, and / or patient-specific, for example defined based on the patient's anatomy. Tool path TP may be associated with a virtual model of the anatomical structure, and the virtual model and tool path may be registered to the anatomical structure using navigation system 32. The control system 60 can generate or obtain the tool path TP by storing / retrieving the tool path TP in memory, creating the tool path TP preoperatively, creating the tool path TP intraoperatively, etc. The tool path TP can have any 3D shape or combination of shapes, such as circular, spiral / drill-shaped, straight, curved, and combinations thereof.

[0079] In one implementation, the tool path TP is defined as a guide or alignment path. In one example, the tool path TP is used to guide tool 20 to a location where it is positioned to begin a surgical procedure or step. For example, if tool 20 is a saw blade, the tool path TP may be configured to guide the saw blade to align with a cutting plane associated with an anatomical structure. If tool 20 is a cutting drill, the tool path TP may be configured to guide the cutting drill to a starting point in preparation for automated cutting. The introduction path may virtually connect from the starting point to another cutting path for tissue removal. The tool path TP also enables tool 20 to move along a predefined motion path to align components of manipulator 14 with navigation system 32. Navigation system 32 can be used to align the tool path TP to an anatomical structure, such that the tool path TP is virtually fixed to the anatomical structure. In this way, the position of the tool path TP in space is automatically updated to account for any movement of the anatomical structure.

[0080] In another implementation, such as Figure 5As shown, the tool path TP is defined as a tissue removal path. An example of a tissue removal path described herein includes a milling path 72. The term "milling path" generally refers to a path of tool 20 near a target site for milling an anatomical structure, and is not intended to require tool 20 to operatively mill the anatomical structure for the entire duration of the path. For example, as will be understood in further detail below, a milling path 72 may include segments or sections in which tool 20 transitions from one location to another without milling. Additionally, other forms of tissue removal, such as tissue ablation, may be employed along the milling path 72. The milling path 72 may be a predefined path created preoperatively, intraoperatively, or a combination thereof. In other words, the milling path 72 may be defined before the start of the surgical procedure, during the surgical procedure (including during tissue removal), or a combination thereof.

[0081] An example of a system and method for generating the virtual boundary VB and / or milling path 72 is described in U.S. Patent No. 9,119,655, entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is incorporated herein by reference. In some examples, the virtual boundary VB and / or tool path TP may be generated offline, rather than on the manipulator controller 26 or navigation controller 36. The virtual boundary VB and / or tool path TP may then be utilized by the manipulator controller 26 at runtime.

[0082] refer to Figure 3 Two additional software programs or modules run on the manipulator controller 26 and / or navigation controller 36. One software module is the behavior controller 74. The behavior controller 74 can calculate data indicating the next command pose and / or orientation (e.g., pose) of the tool 20. In some cases, the behavior controller 74 only outputs the TCP position, while in other cases, it outputs the position and orientation of the tool 20. The outputs from the boundary generator 66, the path generator 68, and the force / torque sensor S can be fed as inputs into the behavior controller 74 to determine the next command pose and / or orientation of the tool 20. The behavior controller 74 can process these inputs, along with one or more virtual constraints described further below, to determine the command pose. The behavior controller 74 can be implemented in an admittance control mode, in which the robotic surgical system actively generates the command position based on the output of a virtual rigid body simulation.

[0083] The second software module may include a motion controller 76. One aspect of the motion controller is the control of the manipulator 14. The motion controller 76 receives data from the behavior controller 74 defining the next command pose. Based on this data, the motion controller 76 can determine the next position of the joint angle of the joint J of the manipulator 14 (e.g., via inverse kinematics and a Jacobian calculator) so that the manipulator 14 can position the tool 20 in, for example, the command pose as commanded by the behavior controller 74. In other words, the motion controller 76 processes the command pose, which can be defined in Cartesian space, into the joint angle of the manipulator 14 so that the manipulator controller 26 can accordingly command the joint motors to move the joint J of the manipulator 14 to the command joint angle corresponding to the command pose of the tool 20. In one embodiment, the motion controller 76 adjusts the joint angle of each joint J and continuously adjusts the torque output of each joint motor to ensure, as closely as possible, that the joint motor drives the associated joint J to the command joint angle.

[0084] Boundary generator 66, path generator 68, behavior controller 74, and motion controller 76 may be subsets of software program 78. Alternatively, each may be a software program that operates individually and / or independently in any combination thereof. The term "software program" is used herein to describe computer-executable instructions configured to perform the various capabilities described in the technical solution. For simplicity, the term "software program" is intended to include at least one or more of boundary generator 66, path generator 68, behavior controller 74, and / or motion controller 76. Software program 78 may be implemented on manipulator controller 26, navigation controller 36, or any combination thereof, or may be implemented by control system 60 in any suitable manner.

[0085] A clinical application 80 can be provided to manage user interactions. The clinical application 80 handles many aspects of user interactions and coordinates surgical workflows, including preoperative planning, implant placement, registration, visualization of bone preparation, and postoperative assessment of implant fit. The clinical application 80 is configured to output to a display 38. The clinical application 80 can run on its own separate processor or in conjunction with a navigation controller 36. In one example, after the user sets the implant placement, the clinical application 80 interfaces with a boundary generator 66 and / or a path generator 68, and then sends the virtual boundary VB and / or tool path TP returned by the boundary generator 66 and / or path generator 68 to a manipulator controller 26 for execution. The manipulator controller 26 executes the tool path TP as described herein. The manipulator controller 26 may additionally create certain segments (e.g., introduce segments) at the start or resumption of processing to smoothly return to the generated tool path TP. The manipulator controller 26 may also process the virtual boundary VB to generate corresponding virtual constraints, as further described below.

[0086] The surgical system 10 can be operated in a manual mode, as described in U.S. Patent No. 9,119,655, which is incorporated herein by reference. Here, the user manually guides the movement of the tool 20 and its energy applicator 24 at the surgical site, and the manipulator 14 performs the movement of the tool and its energy applicator at the surgical site. The user physically contacts the tool 20 to move it in manual mode. In one embodiment, the manipulator 14 monitors the forces and torques applied by the user to the tool 20 to position it. For example, the manipulator 14 may include a force / torque sensor S that detects the forces and torques applied by the user and generates corresponding inputs (e.g., one or more corresponding input / output signals) utilized by the control system 60. In some implementations, it may be necessary for the user to continuously grip a trigger or switch on the end effector 22 to activate the force / torque sensor S that detects the forces and torques applied by the user.

[0087] The force / torque sensor S may include a 6-DOF force / torque transducer. The manipulator controller 26 and / or navigation controller 36 receive inputs (e.g., signals) from the force / torque sensor S. In response to forces and torques applied by the user, the manipulator 14 moves the tool 20 in a manner that simulates the movement that should occur based on the forces and torques applied by the user. The movement of the tool 20 in manual mode may also be constrained relative to a virtual boundary VB generated by the boundary generator 66. In some forms, measurements obtained by the force / torque sensor S are transformed from the force / torque coordinate system FT of the force / torque sensor S to another coordinate system, such as a virtual mass coordinate system VM, in which a virtual simulation 88 is performed on a virtual rigid body model of the tool 20, such that forces and torques can be virtually applied to the virtual rigid body in the virtual simulation 88 to ultimately determine how these forces and torques (and other inputs) will affect the movement of the virtual rigid body, as described below.

[0088] The surgical system 10 can also operate in a semi-autonomous or automated mode, wherein the manipulator 14 moves the tool 20 along the milling path 72 (e.g., the movable joint J of the manipulator 14 is operated to move the tool 20 without requiring force / torque from the user on the tool 20). Examples of operation in automated mode are also described in U.S. Patent No. 9,119,655, which is incorporated herein by reference. In some embodiments, when the manipulator 14 operates in automated mode, the manipulator 14 is able to move the tool 20 without force applied by the user. In other words, the user does not need to physically contact the tool 20 to move it. Instead, the user can use some form of remote control to control the start and stop of the movement. For example, the user can press and hold a button on the remote control to start the movement of the tool 20 and release the button to stop the movement of the tool 20.

[0089] The surgical system 10 can also be operated in a guided manual mode, as described in U.S. Patent Application Publication No. US2020 / 0281676A1, entitled "Systems and Methods for Controlling Movement of a Surgical Tool Along a Predefined Path," the contents of which are incorporated herein by reference in their entirety. In guided manual mode, the user applies force / torque to force / torque sensor S, and the applied force / torque is used to determine the distance the tool 20 is advanced along the tool path TP. In guided manual mode, the tool 20 is constrained to the tool path TP at a 2DOF normal to the tool path, but unconstrained at a 1DOF tangent to the tool path TP. In effect, this allows the tool 20 to move freely along the tool path TP based on manual input, but the constraint guides the user by limiting the manual movement of the tool 20 to along the tool path.

[0090] The techniques described herein may utilize constraint equations and data, forward dynamics algorithms, rigid body calculations, constraint force calculations, and virtual simulations, such as those described in U.S. Patent Application Publication No. US 2020 / 0281676 A1, entitled “Systems and Methods for Controlling Movement of a Surgical Tool Along a Predefined Path,” the contents of which are incorporated herein by reference in their entirety.

[0091] II. Example Manipulator Control Scheme

[0092] refer to Figure 6This section describes an example control scheme implemented by control system 60 for controlling manipulator 14 using tactile force and joint torque. In one implementation, control system 60 is configured to impose constraints on manipulator 14 or surgical tool 20 based on predefined virtual fixation devices or tactile objects (Hobj). Although robot control problems vary depending on the surgical procedure or tool, these constraints can be divided into two subspaces: active constraints and boundary constraints. Active and boundary constraints impose “task space” constraints on the movement of surgical tool 20. The task space is a Cartesian space defined by the task being performed by manipulator 14. The task space can be defined by a set of all possible poses of surgical tool 20 for a given task. The dimensions of the task space will depend on the surgical procedure, the steps of the procedure, the type, or the surgical tool 20, etc. For example, a task in robot-assisted joint replacement surgery requires manipulator control with up to six DOFs, depending on the type of resection and the surgical tool 20 used for said resection. As an example, total knee arthroplasty (TKA) saw cutting might be a fully constrained task requiring complete control of the saw blade across all six degrees of freedom (DOF). Screw / pole placement tasks in shoulder and spinal surgery might require a drill attachment with robotic control across five DOFs (without requiring rolling motion control during drilling). The task space for robotic surgery can be defined by a coordinate system of the implant, patient, or robot, or a combination thereof.

[0093] Active constraints attempt to actively and virtually constrain a specified DOF (Domain of Frame) of the surgical tool 20. Active constraints provide resistance or guidance to the surgeon by actively limiting the surgeon's movement of the surgical tool in a specified manner. For example, an active constraint may constrain certain DOFs of the surgical tool such that if the surgeon attempts to move the surgical tool on one of the constrained DOFs, the tool will feel difficult to move. Such active constraints can be applied in any of the described operating modes of the manipulator 14 (i.e., manual mode, automated mode, guided manual mode, etc.). The surgeon does not need to interact with the tool 20 to apply the active constraints.

[0094] Boundary constraints are used to provide boundaries for the movement of the surgical instrument 20, for example, keeping the surgical instrument within or outside a certain area. The boundary constraint is the aforementioned virtual boundary VB generated by the boundary generator 66. For the boundary constraint, a reaction force is generated in response to contact or potential contact between the tool 20 and the virtual boundary VB. Such boundary constraints can be applied in any of the described operating modes of the manipulator 14 (i.e., manual mode, automated mode, guided manual mode, etc.).

[0095] like Figure 6As shown, the control system 60 can be configured to implement a tactile model (HMb) for calculating the tactile force for achieving boundary constraints and a tactile model (HMa) for calculating the tactile force for achieving active constraints. These tactile models (HMa, HMb) can be individual or combined. As shown, when individual, one or more tactile objects (Hobj) are input into each tactile model (HMa, HMb). For each model, the tactile objects (Hobj) can be the same or different. Any of these tactile models can be implemented using any suitable control scheme, including but not limited to: spring-damper models; proportional derivative (PD) models; or impulse models. The active constraint tactile model (HMa) generates a tactile force (F) intended to actively constrain a surgical instrument at a specified DOF. active (That is, based on the definition of the tactile object). The boundary-constrained tactile model (HMb) generates a reactive tactile force (F) configured to reduce the interaction between the surgical tool 20 and the virtual boundary VB or the geometry of the tactile object (if such interaction exists or is about to occur). reactive Pulse modeling can be used to calculate reaction tactile forces without requiring boundary penetration.

[0096] Numerous examples of active and boundary constraints are envisioned. For boundary constraints, the haptic object (Hobj) can be implemented in a variety of shapes, including some predefined geometric constraints such as planar, line, or volumetric (e.g., cylinder, cone, or box) haptics. The shape of the haptic object can also be defined more generally using polygonal mesh constraints, which consist of a set of triangles connected by their common edges and vertices. For example, a boundary-constrained haptic object (Hobj) can be implemented as a line haptic defined by a mesh volume.

[0097] For active constraints, the tactile object (Hobj) can be implemented by limiting the DOF of the surgical tool 20, which mimics the intended tactile geometry. In one example, the actively constrained tactile object (HobJ) is a tactile plane, and the tactile control model (HMa) is configured to calculate the tactile force (F) of the surgical tool 20 to be constrained relative to at least three specified DOFs of the tactile plane. active In this case, at least three specified DOFs include two rotational DOFs (Rx, Ry) and at least one translational DOF (Tz). Thus, tool 20 is allowed to translate in and out of the plane (Tx), translate left or right in the plane (Ty), and rotate in the plane (Rz), while adhering to 3 DOFs (Tz, Rx, Ry) of planar constraints. For example, such planar constraints can be suitable for constraining a saw blade during cutting. It is worth noting that planar constraints can have up to 6 DOFs. When the robotic system employs passive joints or planar-extended joints, fewer DOFs may be required for planar constraints.

[0098] The actively constrained tactile object (HobJ) can also be a tactile line. The tactile control model (HMa) is configured to calculate the tactile force (F) of the intentionally constrained surgical tool 20 relative to at least four specified degrees of freedom (DOF) of the tactile line. active The at least four specified DOFs may include at least two rotational DOFs (Ry, Rz) and two translational DOFs (Ty, Tz). Thus, the tool 20 is allowed to translate up / down (Tx) along the tactile line and rotate correspondingly (about the tool axis, Rx) while adhering to the 4 DOFs (Ty, Tz, Ry, Rz) of the line constraint. Similarly, the line constraint can be more restrictive, for example, up to 6 DOFs, if desired. In the case where the active constraint is implemented by the tactile volume, the tactile control model (HMa) is configured to calculate the tactile force (F) of the surgical tool 20 intended to constrain relative to the tactile volume for up to three specified DOFs. active For example, up to three rotational DOFs can be constrained. Notably, planar constraints can reach up to 6 DOFs. For example, such line constraints are suitable for constraining cutting drills, planers, drilling machines, screwdrivers, or any other tool with a straight axis.

[0099] Tactile forces can be combined to generate a total tactile force. If the described force / torque sensor (S) is used (e.g., Figure 6 As shown), the force / torque value obtained from the sensor (S) can be combined with the total tactile force to generate a total force. Based on the total force, the control system 60 (or the manipulator controller 26 and / or the motion controller 76) is configured to generate a command joint torque (τ) to control the corresponding joint (J) of the manipulator 14. If the total force includes the active tactile force (F... active If the command joint torque includes a component for moving to the restrained pose to attempt active restraint of the surgical tool at a specified DOF, then the total force will include the reaction tactile force (F). reactive If the total force includes the force from the force / torque sensor S, then the command joint torque will include a component for mitigating tool boundary interactions. If the total force includes the force from the force / torque sensor S, then the command joint torque will include a component for moving the surgical tool 20 in a manner that mimics the interaction between a surgeon and the surgical tool 20, while adhering to active and boundary constraints. This type of calculation can be part of an admittance-based system, which optionally utilizes a behavior controller 74 to perform a virtual rigid body simulation of the total force to determine the command position or joint torque.

[0100] The calculated joint torque (τ) generated based on the haptic model can optionally be combined with an additional feedforward torque (τ). ffThe feedforward torque can be a combination of gravitational torque and the force required for the joint to maintain its position under a specified gravity. The gravitational torque and force can be calculated based on the actual measured joint position (qm). The feedforward torque may also include joint damping torque to smooth movement and prevent joint oscillation. The damping torque can be calculated based on the joint velocity (qm). The feedforward torque (τ) ff The command joint torque (τ) from tactile constraints and the command joint torque (τ) from tactile constraints can be combined to form the total command torque (τ). t The control system 60 will command the joint via the total command torque. Forward kinematics calculations can be used to determine the measured pose (X) of the surgical instrument 20 or TCP. m The measured pose (X) of the surgical instrument 20 m This can be fed back into the control loop to determine the measured pose (X) of the surgical instrument relative to the tactile object. m ) and desired pose (X) d The difference between ).

[0101] The difference (AX) from this comparison is used in the next time step to recalculate the necessary boundaries and active constraints, and this process can be repeated up to multiple time steps. The process described herein can be iteratively executed and repeated up to any number of time steps during the operation of manipulator 14, and can do so depending on the presence or absence of conditions such as detected tool-tissue interactions, surgical procedures, certain operating modes (e.g., manual, automated, guided manual), etc. In this way, the machine learning model (MLM) can continue to adaptively learn and optimize the robot's behavior.

[0102] III. Technologies for using machine learning models to characterize organizational and tool interactions

[0103] refer to Figures 7 to 13. This document describes systems, methods, and non-transitory computer-readable media (computer program products) for employing machine learning models to characterize the interactions between tool 20 and other objects. As described below, such objects can include a wide variety of objects that may be located at a surgical site, such as soft tissue, bone, foreign objects, or other surgical tools. The techniques also enable the characterization of aspects of the objects with which tool 20 interacts. In short, the techniques relate to a control system 60 acquiring sensed measurements during the interaction of surgical tool 20 with a surgical site (e.g., tissue / bone), and applying the sensed measurements to one or more machine learning models (MLMs) configured to perform characterizations related to the tissue or tool interaction. Such characterizations may include, but are not limited to: object type / subtype classification; material property characterization; geometric feature characterization; embedded object characterization; and interaction type characterization. Based on these characterizations, manipulator 14 can be controlled in numerous ways to perform tasks such as anatomical registration, modification of tactile or tactile controls, implementation of protective measures, implementation of adaptive control, etc.

[0104] The techniques described herein offer numerous technical solutions and advantages. For example, the described solutions address the complexity of objects / tissues at surgical sites by providing control schemes capable of understanding the true nature of the objects with which the tool 20 interacts. The control schemes described herein can fully identify and interpret unknown contact dynamics arising from tool interaction. These unknown contact dynamics may be caused by complex factors such as object properties, friction, contact forces, or geometric variations. In turn, the described solutions can characterize the properties of the objects with which the tool interacts or distinguish different object types. The machine learning techniques described herein are dynamic and adaptive, thus providing customized control schemes based on real-time environmental interaction and by making intelligent predictions about the contact dynamics occurring between the tool and the object. Therefore, the accuracy of the tool is improved, particularly relative to the surgical site. This improved tool accuracy will enhance the performance of the manipulator 14, exhibiting more predictive tool behavior and providing better surgical outcomes for the patient. Furthermore, the described machine learning techniques unlock the ability of the manipulator 14 to perform higher-order tasks with greater complexity, as described below.

[0105] A. Introduction to Machine Learning Models and Inputs

[0106] refer to Figure 7Example block diagrams illustrating control processes for tool-based interactive execution of representations, which can be performed through the described techniques, are provided. Throughout this description, the steps or processes of the control scheme will be described as being performed by control system 60. As described, control system 60 may include any one or more of the controllers or components described in the surgical system. Figure 7 The block diagram is expanded by including a machine learning model (MLM) configured for the described solution. Figure 6 The block diagram is shown. Machine learning models (MLMs) can be added to or incorporated into control system 60 to improve various aspects of control system 60. In other implementations, machine learning models (MLMs) can be used as replacements for components / software of control system 60. For example, a machine learning model (MLM) can replace behavior controller 74 and / or motion controller 76.

[0107] As shown in the figure and as will be described below, the machine learning model (MLM) is configured to receive and process various input measurements, including interaction forces, measured tool displacement (or deformation, if applicable), and measured tool velocity (or deformation rate, if applicable). The control system 60 can acquire these measurements in response to the interaction of the surgical instrument 20 with the surgical site over time.

[0108] Surgical tools 20 typically perform some type of action when interacting with an object at a surgical site. The action of tool 20 can be sliding across the object, pressing into the object, and / or palpating the object. The action of tool 20 can also be surgical actions, such as cutting, drilling, sharpening, sawing, planing, reaming, impacting, milling, carving, or any other form of manipulation.

[0109] Interactive force is, for example, the actual or predicted force applied to the surgical tool 20 during interaction with a corresponding object at the surgical site. Various methods can be used to obtain interactive force. For example, such as... Figure 7 As shown, the interaction force can be obtained via a force / torque sensor (S) coupled to the surgical tool 20 or the manipulator 14. If an adjustable arm (AA) is used, a force sensor (S′) of the adjustable arm (AA) or a pointer tool (P) attached to the adjustable arm (AA) can be used. Other forms of force sensing can be utilized, such as implementing additional force sensors, load cells, strain gauges, or pressure sensors that may be located at or near the TCP of the surgical tool 20. In other cases, the control system 60 may implement a force observer to indirectly infer the force acting on the tool 20 using indirect parameters such as velocity and / or acceleration.

[0110] The measured displacement is the actual or predicted displacement generated by the surgical instrument 20 based on its interaction with a corresponding object at the surgical site. In one example, the control system 60 may use robot data to calculate the measured displacement. Figure 7 As shown, forward kinematics calculations can be applied to the measured joint position (qm) from the manipulator 14 to obtain the measured tool pose (Xm). The measured tool pose (Xm) can be compared or evaluated over time to determine the displacement of the tool 20. Other methods can be used to measure displacement, such as tracking data obtained from the navigation system 32 (e.g., tracking surgical sites and tools), inertial sensors provided on the surgical tool 20, etc.

[0111] Force sensor (S) measurements and the measured tool pose (Xm) (and / or displacement) can be used as variables to determine deformation, which indicates the degree to which the object deforms in response to the interaction forces applied by tool 20. Deformation can be actual or predicted. Calculation of deformation may be appropriate when the object is flexible / deformable (such as soft tissue). However, it is not necessary to calculate deformation, especially when the tool interacts with a rigid object such as bone or metal, in which case displacement of the tool may be sufficient rather than deformation. In one example, the measured deformation can be calculated by measuring the displacement of surgical tool 20 relative to the object between the initial and final points of contact. Force sensors (S), navigation system 32, or combinations thereof can be used to identify these initial and final points of contact. Other methods can be used to measure deformation, such as tracking data obtained from navigation system 32 (e.g., tracking the surgical site and tool), inertial sensors provided on surgical tool 20 and tissue, etc.

[0112] The measured velocity is the actual or predicted velocity of the surgical instrument 20 during interaction with a corresponding object at the surgical site. In some cases, the velocity can be calculated using the derivative of displacement. When used in conjunction with displacement / deformation, the measured velocity can indicate the measured rate of deformation (or rate of change of deformation) of the tissue in response to the force applied to the tissue by the surgical instrument. In one example, such as Figure 7 As shown, the joint velocity can be measured based on the manipulator 14. To determine the measured velocity, the Jacobian transform can be used to convert the measured joint velocity. The measured velocity of tool 20 is mathematically related. Other methods can be used to measure tool velocity, such as tracking data obtained from navigation system 32, inertial sensors provided on surgical tool 20, etc. In other examples, control system 60 may obtain the measured velocity by calculating the derivative of the measured deformation, as described above.

[0113] To further illustrate the relationship between the measured interaction forces, displacements, and velocities, Figure 8A An example spring-damper model is illustrated, incorporating tissue embedded with a tumor. Assuming the tissue exhibits a homogeneous structure, it can be linearly modeled as a spring and damper with constant stiffness (Ke) and damping (Be). The interaction force (Fe) applied to the tissue by tool 20 produces a measured deformation (Xd). The rate of deformation can be calculated using the derivative of the measured deformation (Xd). (For example, speed). Using this linear organization model, as... Figure 8B Equation (1) is shown, along with the applied interaction force (Fe), the measured deformation (Xd), and the velocity. There is a relationship between them. Equation (1) illustrates how organizational properties can be measured given the forces, displacements, and velocities. Under the linear assumption of the organizational model, stiffness (Ke) and damping (Be) can be measured when there are two independent measurements (from two different regions or from the same region at two different times).

[0114] It is worth noting that the surgical system 10 may employ a "sensing system" to obtain any of the measurements described. The sensing system may include any suitable components, sensors, and controllers described above. For example, the sensing system may include, but is not limited to: any controller in the control system 60 and the described controllers (26, 74, 76), force sensors (S, S′), tracking data (position, orientation, velocity) from the navigation system 32, robot data from the manipulator (e.g., joint position, velocity, acceleration), joint torque values, current drawn from the joint motors, joint position data from the adjustable arm (AA), or any other supplementary sensors such as inertial sensors (on the robot, tool, or tissue), pressure or force sensors (near the TCP of the tool), tissue sensors (e.g., strain gauges, optical sensors, capacitive sensors, etc.).

[0115] Any of the techniques described herein for characterizing or classifying objects or interactions can be used in any of the control modes described for the manipulator 14, including manual mode, semi-autonomous or automated mode, guided manual mode, etc. The techniques are also applicable to situations where any of the described active or boundary constraints are activated or deactivated.

[0116] B. Machine learning for estimating tissue stiffness and / or damping

[0117] refer to Figure 9An example architecture of a machine learning model (MLM) is shown. The machine learning model (MLM) is configured to receive and process measurements of interactive forces, displacements / deformations, and velocities to estimate the stiffness and / or damping parameters of an object based on the interactions performed by the surgical tool 20.

[0118] Stiffness parameters indicate the resistance of an object to deformation under applied loads of interaction forces (e.g., the degree to which the object pushes back against a tool). Damping parameters indicate the ability of an object to absorb or dissipate energy in response to applied loads of interaction forces (e.g., the rate at which the object loses energy under deformation). Stiffness parameters can be measured using any suitable value (such as Young's modulus), where greater stiffness indicates a stiffer structure, and vice versa. Damping parameters can be measured using any suitable damping coefficient, where higher damping indicates a faster rate at which the object loses energy in response to applied loads.

[0119] Figure 8A The linear organization model described includes constant stiffness (Ke) and damping (Be) and is used to illustrate the relationship between interaction forces, displacement / deformation, and velocity. However, due to the generally nonlinear behavior of organizations and the realities of dynamic stiffness / damping, the stiffness and damping parameters of organizations are much more complex and difficult to derive. Therefore, machine learning models (MLMs) are configured to intelligently predict the nonlinear and dynamic stiffness / damping parameters of organizations. Notably, although stiffness and damping provide synergistic analysis when combined, machine learning models (MLMs) can be configured to estimate the stiffness parameters of an organization (undamped) or to estimate the damping parameters (unstiffened).

[0120] To achieve this capability, a first neural network (NN1) can be used to configure a machine learning model (MLM). The first neural network (NN1) may include an input layer, hidden layers, and an output layer. The input layer of the first neural network (NN1) receives measurements of interaction forces, displacement / deformation, and velocity. The output layer of the first neural network (NN1) outputs the estimated stiffness and damping parameters.

[0121] Machine learning models (MLMs) can be configured to output stiffness / damping parameters in various ways. For example... Figure 9As shown, for example, the output layer of the first neural network (NN1) may include two nodes, one for outputting the estimated stiffness parameter and the other for outputting the estimated damping parameter. However, it is envisioned that the output layer may include a node that outputs both the (combined) stiffness and damping parameters, only the stiffness parameter, or only the damping parameter. Any node may output different parameter values ​​over time. For example, the stiffness / damping value may change over time. Furthermore, it is possible that one node may output a parameter while another node does not. The output layer may include any suitable number of nodes, depending on how the stiffness / damping parameters may need to be estimated. For example, one node may output the Young's modulus value of the stiffness parameter, while another node may output a value relating the stiffness parameter to the damping ratio.

[0122] The first neural network (NN1) can be any suitable type of neural network, such as, but not limited to: artificial neural networks, deep learning networks, transfer learning networks, convolutional neural networks (CNN), recurrent neural networks (RNN), multilayer perceptrons (MLP), generative adversarial networks (GAN), feedforward neural networks, encoder networks, or transformers.

[0123] The machine learning model (MLM) may optionally include or be connected to a first Long Short-Term Memory (LSTM) or Recurrent Neural Network (RNN). The first LSTM / RNN may be connected to the input layer of a first neural network (NN1). The first LSTM / RNN receives measurements over time. The first LSTM / RNN can modify weights and biases to learn long-term dependencies between measurements and selectively filter measurements. The first LSTM / RNN can then feed the filtered measurements to the input layer of the first neural network (NN1). The LSTM processes measurements within a configurable time window. In other cases, such as when no velocity measurement is available or when performing a more straightforward classification, the first LSTM / RNN may be omitted or bypassed, and the input data may flow directly into the first neural network (NN1).

[0124] Numerous samples of measurements can be collected over time. For example, measurements can be collected from N different regions (distributed tactile force) or from the same region at N different sampling times (single-point force) to estimate tissue characteristics. Therefore, reference Figure 8B The system of equations in (2), f e1 , ..., f eN There are N different interaction force measurement results, (x d1 , ..., x dN ) represents N different deformation measurement results, and There are N different velocity measurement results. The control system 60 can use an optimization algorithm to solve this set of equations (2) to estimate the stiffness and damping parameters of the object.

[0125] While simple optimization algorithms can lead to measurements of Ke and Be for certain interactive forces and displacement / deformation, the accuracy of the results depends on whether a linear model can be applied to the interactive objects. However, as described, the models for most objects are typically not linear. Furthermore, the objects may be non-uniform, and there may be some embedded objects. To illustrate this in the context of the organization, Figure 8A The example illustrates a case where tissue contains a tumor. The tumor is harder than the surrounding healthy tissue, and the stiffness and damping measured during tissue palpation can vary significantly depending on the depth of the tumor relative to the healthy tissue. Figure 8A The diagram further illustrates a spring-damped model of a tumor and surrounding healthy tissue with different stiffness and damping constants. Now considering both the nonlinearity and inhomogeneity of the tissue, it can be assumed that there is a general nonlinear relationship between the measured forces, displacements, velocities, and the stiffness and damping of the tissue, as represented by equation (3). This equation replaces equation (1), which assumes a linear relationship between the measurements and the tissue properties. For the set of equations in (2), equation (4) can be considered, where a machine learning model (MLM) is used to solve for the stiffness Ke and damping Be.

[0126] The LSTM component is responsible for memorizing N previous samples of force, displacement / deformation, and velocity measurements, which can be used to form the inputs defined in equation (4). The first neural network (NN1) implements the nonlinear function (g) in equation (4). The machine learning model (MLM) can be trained to determine the weights and biases of the first neural network (NN1) based on sample objects with known properties, which can later be used for any given measured force, displacement / deformation, and velocity to estimate the stiffness and damping of a given object. The first neural network (NN1) can be trained using deep learning, supervised learning, or semi-supervised learning. In other cases, unsupervised learning can be used to implement the first neural network (NN1). The training of the first neural network (NN1) can be based on tool-specific data, procedure-specific data, surgical procedure-specific data, control mode-specific data, etc. Training can also be based on any tissue / object with various geometric features, material properties, and types described herein.

[0127] It is worth noting that the machine learning model (MLM) generates the estimated stiffness and damping parameters as lower-dimensional embeddings compared to the measurements. The numerical or vector representation of the stiffness and damping data can be made less dimensional than the measurements, allowing the MLM to more easily learn patterns and relationships, improve computational efficiency, and achieve better predictions for unpredictable data. In one example, the measurements have more than two dimensions, and the estimated stiffness and damping parameters have up to two dimensions. Other ranges of dimensions are also possible and envisioned.

[0128] C. Machine Learning Interactive Evaluator

[0129] Continue to refer to Figure 9 The control system 60 and / or machine learning model (MLM) may employ an interactive evaluator (IE). The interactive evaluator (IE) is configured to receive estimated stiffness and / or damping parameters from the output of a first neural network (NN1) to predict characterizations of the interaction with tool 20. The interactive evaluator (IE) may continuously receive the estimated stiffness and / or damping parameters during a predefined time window, at discrete times, or in discrete batches or groups. The interactive evaluator (IE) may monitor changes in the estimated stiffness and / or damping parameters over time. The interactive evaluator (IE) is configured to process the estimated stiffness and / or damping parameters to predict one or more characterizations based on the interaction.

[0130] By monitoring stiffness and damping parameters, a machine learning model (MLM) can reveal any object or interaction representation described in the object or interaction representation. That is, the stiffness and damping parameters are based on the measured interaction forces, the measured displacement / deformation, and the measured tool velocity. When evaluated over time, these measurements can exhibit specific (force / displacement / velocity) waveforms or signals generated according to the interaction. Since the actions (force, displacement, velocity) of tool 20 are also embedded within the stiffness and damping parameters, predictions can be made using the actions of tool 20. For example, the signal may differ when the tool slides across a fine texture compared to when it slides across a coarse texture, and so on. The techniques described herein can derive a representation in response to any action of tool 20. In some cases, the actions of tool 20 can be random, variable, user-defined, or unknown. In other cases, the actions of tool 20 can be specified, for example, due to surgical procedures, clinical requirements, and / or control system requirements.

[0131] In one implementation, such as Figure 10As shown, the Interactive Evaluator (IE) is part of a Machine Learning Model (MLM) and is implemented using a second neural network (NN2). The second neural network (NN2) may include an input layer, hidden layers, and an output layer. The input layer of the second neural network (NN2) receives the measured and estimated stiffness and / or damping parameters. The output layer of the second neural network (NN2) outputs the corresponding representation. The second neural network (NN2) may implement a nonlinear function to correlate the estimated stiffness / damping parameters with the appropriate representation. The Machine Learning Model (MLM) can be trained to determine the weights and biases of the second neural network (NN2) based on sample stiffness and damping values ​​with known object characteristics, which can later be used for representation prediction. The training of the second neural network (NN2) can be based on tool-specific data, procedure-specific data, surgical procedure-specific data, control mode-specific data, etc. Training can also be based on any tissue / object with various geometric features, material properties, and types described herein. The second neural network (NN2) can be trained using deep learning, supervised learning, or semi-supervised learning. In other cases, unsupervised learning can be used to implement the second neural network (NN2).

[0132] Machine learning models (MLMs) can be configured to output one or more distinct representations. For example... Figure 10 As shown, for example, the output layer of a second neural network (NN2) may include several nodes, each configured to output a representation of a specific type. For example, up to five or more nodes may exist. These nodes may be configured accordingly to output object type / subtype classifications, material property representations, geometric feature representations, embedded object representations, and interaction type representations. Other representations are envisioned besides those shown. Two or more nodes may output values ​​applicable to more than one representation, which a machine learning model (MLM) may combine to make more complex, higher-order predictions about interactions. Over time, any node may output representation values ​​that change over time or in terms of the type of representation. Some nodes may output representations, while others may not. The output layer may include any suitable number of nodes, depending on how many different representations can be expected to be predicted. Furthermore, an output node may be configured to generate numerous types of representations. Therefore, a number of output nodes is not necessarily required, such as... Figure 10 As shown.

[0133] The second neural network (NN2) can be any suitable type of neural network, such as, but not limited to: artificial neural networks, deep learning networks, transfer learning networks, convolutional neural networks (CNN), recurrent neural networks (RNN), multilayer perceptrons (MLP), generative adversarial networks (GAN), feedforward neural networks, decoder networks, or transformers.

[0134] The machine learning model (MLM) may include or be connected to a second Long Short-Term Memory (LSTM2) or Recurrent Neural Network (RNN2). The second LSTM2 / RNN2 component is responsible for memorizing N previous samples of stiffness and / or damping, which can be used to determine the corresponding representations. The second LSTM2 / RNN2 may be connected to the input layer of a second neural network (NN2). The second LSTM2 / RNN2 may receive estimated stiffness and damping parameters from the output layer of a first neural network (NN1) over time. The second LSTM2 / RNN2 may modify the weights and biases to learn the long-term dependencies between the estimated stiffness and damping parameters and selectively filter the estimated stiffness and damping parameters. The second LSTM2 / RNN2 may feed the filtered measurements to the input layer of the second neural network (NN2). The second LSTM2 / RNN2 may process the estimated stiffness and damping parameters within a configurable time window. In some cases, the LSTM2 / RNN2 may be configured to memorize only the most recent stiffness and damping parameters predicted by the first neural network (NN1).

[0135] Optionally, such as Figure 10 As shown, the second LSTM2 / RNN2 can further receive speed measurements over time. These measured speed values ​​may be the same as or different from the values ​​input to the first LSTM1 and the first neural network NN1. For example, the measured speed values ​​may be input to both networks simultaneously, or time-delayed or filtered when input to the second LSTM2 / RNN2. Inputting the measured speeds into the second LSTM2 / RNN2 will allow the machine learning model (MLM) to perform more complex representations, such as, but not limited to, hierarchical representations and representations of embedded object size and depth.

[0136] Machine learning models (MLMs) can optionally generate representations as vectors with a higher dimension than the estimated stiffness and damping parameters. By doing so, MLMs can capture more complex relationships that would otherwise be impossible to understand using lower-dimensional stiffness / damping data. In one example, the representation has more than two dimensions, and the estimated stiffness and damping parameters have up to two dimensions. Other ranges of dimensions are also possible and envisioned.

[0137] In some cases, a machine learning model (MLM) may proactively filter data or reduce its computation in response to the general context of the interaction provided by the control system 60. For example, if preoperative imaging data reveals that the bone does not contain a foreign object, the machine learning model (MLM) may temporarily filter its second neural network (NN2) or utilize a customized second neural network that omits any representations involving foreign objects.

[0138] An interactive evaluator (1E) is envisioned that employs a lookup table (LUT), which the control system 60 utilizes to correlate the estimated stiffness and / or damping parameters with values ​​related to organization or interactive characterization. The lookup table can be used as a supplement to or alternative to a second neural network (NN2) and LSTM2 / RNN2.

[0139] D. Characterization

[0140] As described, the machine learning model (MLM) is configured to predict one or more representations related to the object with which the tool 20 interacts, or a representation of the interaction between the tool 20 and the object. By evaluating stiffness and damping parameters over time, the machine learning model (MLM) can learn properties (e.g., type, geometric features, etc.) of the object with which the tool 20 interacts. Such representations include, but are not limited to: object type / subtype classification, material property representation, geometric feature representation, embedded object representation, and interaction type representation. Figure 9 The corresponding boxes provide a non-exhaustive list of examples of each of these representations, which will be described below. It is worth noting that a machine learning model (MLM) can be configured to combine numerous representations. Therefore, any representation described below can be combined with any other representation. Furthermore, although the term "representation" is chosen in this document, it should be understood that the term "classification" may be used instead in certain situations, such as when classifying object types. Moreover, as the machine learning model (MLM) gains knowledge over time by processing interactions, any representation described will predict with greater accuracy.

[0141] 1. Object type / subtype representation

[0142] Machine learning models (MLMs) characterize the type or subtype of an object that can be identified or predicted. In one example, the object can be tissue, and the characterization identifies the presence or type of tissue. For example, tissue type characterization typically identifies whether the tissue is soft tissue or bone. Additionally, tissue characterization can identify specific subtypes of soft tissue or bone. For example, tissue characterization can identify soft tissue subtypes such as cartilage, muscle, ligament tissue, tendon tissue, blood vessels, healthy tissue, and malignant tissue (or tumor tissue). Tissue characterization can identify bone subtypes such as cortical bone (including endocortical or extrinsic cortical bone), cancellous bone, osteophytes, etc. Therefore, tissue type characterization can advantageously help distinguish materials that humans would not normally be able to easily differentiate, such as osteophytes versus cortical bone, or cartilage versus cortical bone.

[0143] In other cases, the object may be a foreign body. Such foreign bodies can include any object that may be located within, on, or around the surgical site. Foreign bodies can include, but are not limited to, surgical implants (main components, trailing components, revision components, fasteners, screws, pins, plates), surgical instruments (e.g., soft tissue retractors, needles, irrigation tools, aspiration tools, surgical tracker supports, surgical cameras / observators, joint tensioners / retractors, etc.), surgical sponges, etc. In other cases, a foreign body can be a metal fragment, a piece of metal, a fluid (e.g., irrigation / aspiration fluid), or any other known or unknown foreign body. In any case, a machine learning model (MLM) can identify the foreign body and distinguish it from bone or soft tissue. Foreign bodies can also be finely identified based on their material properties. For example, the stiffness characteristics of a metal tool differ from those of a surgical sponge, allowing a particular foreign body to be identified as either a surgical instrument or a sponge.

[0144] Figure 11 This is a plot illustrating sample stiffness and damping parameter values ​​generated over time by a machine learning model (MLM) to characterize various objects at surgical sites. The damping and stiffness parameters are illustrated on their respective axes as ranges from 0 to predefined maximum values. In this specific example, the machine learning model (MLM) learns that cartilage is characterized by stiffness in the range of approximately 0.1 to 0.25 of the maximum stiffness value and damping in the range of approximately 0.1 to 0.25 of the maximum damping value. Bone is characterized by stiffness in the range of approximately 0.25 to 0.75 of the maximum stiffness value and damping in the range of approximately 0.25 to 0.75 of the maximum damping value. Implant material is characterized by stiffness in the range of approximately 0.75 to the maximum stiffness value and damping in the range of approximately 0.75 to the maximum damping value. A specific type (i.e., type A) of soft tissue is characterized by stiffness in the range of approximately 0.1 to 0.3 of the maximum stiffness value and damping in the range of approximately 0.7 to 0.9 of the maximum damping value. Therefore, the machine learning model (MLM) has learned how to distinguish whether the tool interacts with bone, soft tissue, cartilage, or implant material. Foreign objects can be classified in a comparable way. Although not drawn, the machine learning model (MLM) can learn how to classify the type or subtype of any object or material described in this paper.

[0145] 2. Material property characterization

[0146] In another example, a machine learning model (MLM) can characterize the material properties of an object (e.g., any object described, such as soft tissue, bone, or a foreign body object). Material properties may include, but are not limited to: density, bone mineral density (BMD), stiffness, hardness, softness, smoothness / fineness, roughness / coarseness, plasticity, elasticity, ductility, compressive strength, etc. More than one material property can be characterized at the same time or at separate times.

[0147] When monitored over time, stiffness and damping parameters reveal the material properties of an object. That is, stiffness and damping parameters are based on measured interaction forces, measured displacement / deformation, and measured velocity. When evaluated over time, these measurements can exhibit specific (force / displacement / velocity) waveforms or signals based on the object's material properties. Because the motion of tool 20 (force, displacement, velocity) is also embedded in the stiffness and damping parameters, predictions can be made using the motion of tool 20. For example, when the tool slides across the surface of an object, the signal may exhibit a more linear waveform for a smooth object compared to a rough object.

[0148] Machine learning models (MLMs) can characterize the material properties of an object in any suitable way. For example, MLMs can characterize any material property at any scale. Material properties can have predefined “levels.” For example, MLMs can identify density levels I, II, and so on. Density level I can indicate a T-score of 0 or less, density level II can indicate a T-score between 0 and 1, and density level III can indicate a T-score greater than +1. In some examples, the properties of an object can be characterized by a quantity of stiffness / damping. Any other suitable way of characterizing material properties can be utilized.

[0149] 3. Geometric Feature Representation

[0150] Another type of representation that can be performed by a machine learning model (MLM) is a geometric feature representation, which defines the geometric features of any object among the objects undergoing interaction. Examples of geometric features include, but are not limited to: the object's thickness, the object's depth (relative to a reference point), the object's size (e.g., area, surface area, volume, etc.), the object's shape, the object's outline, and other features of interest such as corners, edges, protrusions, grooves / recesses, etc.

[0151] When monitored over time, stiffness and damping parameters can reveal the geometric characteristics of an object. For example, when a tool presses in and slides across a circular object, the signal may exhibit a specific shape for a circular object compared to a flat object.

[0152] 4. Interaction type representation

[0153] In another example, the interaction type of tool 20 can be characterized. For example, a machine learning model (MLM) can characterize tool interactions as skeletal interactions, soft tissue interactions, or even non-contact interactions (interactions with air). The interaction type can be based in part on one of the above representations, such as an object type representation. The interaction type representation can provide insights different from the represented object type because it is designed to help control system 60 understand the current interaction of tool 20, rather than the characteristics of the object itself. For example, an object might be described as tissue (at a certain point in time), but tool 20 might soon be interacting with bones. In this example, the tissue representation does not change (because the tissue itself has not changed), but the interaction representation changes because tool 20 is now interacting with bones. Knowing the interaction type can provide additional insights into the other representations described above. For example, by knowing that the interaction type is a soft tissue interaction, the machine learning model (MLM) can characterize only the soft tissue subtype.

[0154] In some cases, the control system 60 may obtain the interaction type from sources other than the machine learning model (MLM). For example, robot data and patient tracking data from the navigation system can be used to determine the distance between the tool 20 and the surgical site. If the tool 20 is spaced apart from the surgical site, the interaction is non-contact, and this information can be provided to the machine learning model (MLM) to facilitate further representation. Similarly, the control system 60 may obtain this information from other sources, such as control data / patterns, surgical planning information, tool path information, skeletal model data, etc.

[0155] 5. Embedded object representation

[0156] In some cases, an object may be embedded within another object. For example, a tumor may be embedded in surrounding soft tissue, a sublayer of tissue may exist beneath the surface of the tissue, and a portion of an implant may be embedded in bone, etc. Machine learning models (MLMs) can characterize features or parameters associated with this object embedding. Examples of embedded object representations include, but are not limited to, determining: tissue layers; transitions between tissue layers; the geometry of the embedded object; the type of the embedded object (e.g., blood vessels, ligaments, tumors, foreign bodies, etc.), and so on. Such representations have many benefits, such as allowing measurement of cartilage thickness, artery size, the presence of existing implants, the depth of tumors in tissue, and so on.

[0157] To perform embedded object representation, the machine learning model (MLM) can input the measured velocity into a second LSTM2 / RNN2, as described above and as... Figure 10 As shown. In other words, the measured velocity (in addition to stiffness / damping parameters) enables the machine learning model (MLM) to characterize the relationship between the object and its surrounding environment.

[0158] Now refer to Figure 12 Figure 13 illustrates an example of object representation. This example is provided only as one of many possible implementations. For simplicity and to avoid redundancy, it should be understood that the theories, calculations, measurements, graphs, and representations described in this example can be used for other representations described above.

[0159] Figure 12 This is a simplified diagram (scales exaggerated for illustrative purposes) of the task involving manipulator 14 and the patient's anatomical structure (A). The anatomical structure (A) in this example comprises a top layer of tissue with relatively low stiffness K1 and a second layer of tissue with stiffness K2 (e.g., assumed to be five times stiffer than the top layer). Additionally, there is an embedded object (e.g., a tumor or foreign body object) with stiffness K3 (e.g., assumed to be five times stiffer than the second layer of tissue). The aim is to explore how these three types of objects can be classified using the proposed method, and to estimate the thickness of the first layer and the depth of the embedded object within the tissue.

[0160] As previously explained, the machine learning model (MLM) includes a first neural network (NN1) and an interaction evaluator (IE), which may include a second neural network (NN2). The first neural network (NN1) can take into input deformation / displacement, velocity data, and measurements of interaction forces during tissue manipulation. The first neural network (NN1) characterizes the stiffness and damping of the tissue during tissue manipulation.

[0161] The example assumes that the organization's manipulation tasks transition from the top level to the embedded objects, and that these corresponding samples are captured in the order of this transition. Figure 13A A snapshot of deformed data of N samples stored in the first LSTM / RNN component of the first neural network (NN1) is shown. Figure 13A Each color-coded area in the chart represents a layer of the tissue, showing where the layer begins to deform during tissue manipulation. Figure 13B and Figure 13C Snapshots of the measured velocity data and measured interaction forces of N samples in the first LSTM / RNN during tissue manipulation are presented respectively.

[0162] Figure 13D A graph showing the measured interaction force versus deformation is presented, illustrating the linear relationship between these two variables for each layer. Here, stiffness can be defined as the slope of the force-deformation graph, with three different values ​​used for the three layers of the tissue in this example. The first neural network (NN1) can be trained to output stiffness values ​​K1, K2, and K3 when force and deformation data are input into the first neural network (NN1). A similar approach can be applied to output a damping graph, where force-velocity data is used to measure the damping constant of the tissue. This can also be extrapolated to having all three inputs and simultaneously estimating both stiffness and damping.

[0163] Furthermore, tissue classification and geometric feature estimation can be achieved through an interactive evaluator (IE) or a second neural network (NN2), as described above. That is, the measured stiffness and damping from the first neural network (NN1), along with the measured velocity, are input into the second neural network (NN2). The second neural network (NN2) classifies different types of tissue based on the instantaneously measured stiffness and damping, and a second LSTM2 / RNN2 component in the network is responsible for memorizing changes in stiffness and damping, thereby estimating some characteristics of the tissue, such as the thickness of the first layer and the depth of objects embedded within the tissue.

[0164] Figure 13E The stiffness output of the first neural network (NN1) for a given example is shown. Figure 13E As seen, the first neural network (NN1) can be used to determine the instantaneously measured stiffness and damping during tissue manipulation. To extract geometric feature representations, the second LSTM2 / RNN2 component of the second neural network (NN2) can be used to memorize the measured stiffness and damping over a predefined window (N samples). Analysis of these samples can be used to identify the locations where transitions occur from one layer to another (t1→t2 and t2→t3). If the velocity is constant, the thickness of the corresponding layer can be measured as: thickness = Δt x V, where Δt = (t2 - t1). However, if the velocity is not constant, the velocity map ( Figure 13B The region below, between t1 and t2, represents the thickness of the top layer, i.e., in Figure 12 and Figure 13FIn this context, A1 is used as the denoting factor. To enable the network to compute this, the velocity map can be discretized, and the layer thickness can be calculated as: thickness = dt x V(tN) + dt x V(t-(N-1)) + ... + dt x V(t-(N-N1)), where dt is the sampling time of the control system providing the measurement data. From this, we can see that the thickness of the top layer (A1) can be expressed as varying according to the previous velocity between t1 and t2. The depth of the embedded object can be measured as the region below the velocity map where the transition from the first to the third layer occurs, i.e., A1 + A2. Here, A2 represents the thickness of the tissue region from the bottom of the upper layer to the top of the embedded object. The identification of thickness A2 begins when the machine learning model (MLM) first observes stiffness K2 during the penetration task. As tool 20 continues to penetrate the tissue region, tool 20 will pass through the second layer (K2 stiffness) and eventually be pressed into the embedded object (stiffness K3). Thickness A3 is the penetration depth of tool 20 relative to the embedded object, defined from the top of the embedded object to the distal tip of tool 20. The labeling of thickness A3 begins when the machine learning model (MLM) first observes stiffness K3 during the penetration task. Thickness A1+A2+A3 represents the total penetration depth of the tool relative to the entire tissue region (i.e., from the top surface of the upper tissue layer to the distal tip of the tool 20). Using discretized velocities, the depth can be measured as: Depth = dt x V(tN) + ... + dt x V(t-(N-N1)) + ... + dt x V(t-(N-N2)). Feature estimation varies based on N velocity inputs stored in the LSTM2 / RNN2 component of the second neural network (NN2) (some velocity inputs have a weight of 1 and the rest have a weight of 0, depending on where the transition occurs), as follows: Feature = f(V(tN), ..., V(t)). The second neural network (NN2) represents the function (f), which can be trained to correctly estimate the features.

[0165] E. Use Cases and Examples

[0166] Based on any one or more of the above characteristics, the manipulator 14 can be controlled in numerous ways and the manipulator can be controlled to perform tasks (such as...). Figure 7 (As shown), examples include anatomical registration, modification of tactile sensation or control, implementation of protective measures, and implementation of adaptive control. These examples will be described below. However, these examples do not exhaustively list all possible ways in which the described techniques can be utilized. Other solutions are envisioned.

[0167] 1. Anatomical registration

[0168] The predicted representation can be used to register anatomical regions (such as bones). This registration can be performed using image-free techniques that do not rely on preoperative imaging or a pre-defined bone model. To perform registration, the tool tip contacts / tracks individual surface points of the anatomical structure. The traces / points can be used to form a point cloud or virtual (3D) surface model of the anatomical structure. Registration can be performed regardless of whether the navigation system 32 uses a tracker to track the anatomical structure. The tool tip can be the TCP of the tool 20 supported by the manipulator 14. In another example, as described above, the tool tip can be a pointer P, which is coupled to an adjustable arm (AA) and equipped with a force sensor (S′), and optionally, a pointer tracker (PT), such as... Figure 1 As shown. Whether using the robot-supported tool 20 or the arm-supported pointer P, the position of the tool tip can be recorded over time using the navigation system 32. The tool tip position can be additionally or alternatively captured by encoder / joint data from the manipulator 14 or the adjustable arm (AA).

[0169] The movement of the tool tip into contact with the bone surface can be performed automatically (automatic mode) or manually in response to a force applied by the user (e.g., adjusting manipulator 14 or adjustable arm AA). The position of the tool tip as it contacts the bone surface is recorded by robot data and / or navigation data. The bone can optionally be tracked during this process.

[0170] According to the described technique, interaction forces, displacement / deformation, and velocity are measured during bone-on-bone contact interaction. Based on these measurements, a machine learning model (MLM) estimates stiffness and damping parameters and characterizes the tissue at each collected point as bone. The control system 60 then correlates the recorded position of the tool tip with the bone tissue characterization at each point to generate a virtual surface model of the bone. The virtual surface model of the bone is then registered to the bone.

[0171] The described registration technique is particularly advantageous for complex environments on or around bone. For example, a machine learning model (MLM) can characterize some points as soft tissue, others as cartilage, and most points as bone. The recorded location of the tool tip is then correlated with the various characterizations of the individual points to generate a virtual surface model of the bone, along with identified soft tissue regions and cartilage maps. Soft tissue and cartilage (based on their characterization) can then be ignored during registration to ensure that the registration accurately reflects the bone surface. Alternatively, the characterized cartilage and soft tissue regions can be saved and used for other purposes, such as for clinical applications. In another example, osteophytes can be characterized by a machine learning model (MLM) and located during the registration process. The characterized osteophyte locations can be used to form a virtual surface model of the osteophytes, or they can be ignored to form a virtual surface model without osteophytes.

[0172] 2. Haptic Modification / Control

[0173] In another example, the techniques described above can be used to modify the tactile feedback of manipulator 14 or to modify the tactile control of manipulator 14. Such modifications may include adjusting the tactile object (Hobj), adjusting the parameters of the tactile model (HMa, HMb) for boundary constraints or active constraints, changing the reaction force calculation, etc.

[0174] Modifications to the haptic feedback can be responsive to one or more of the described representations. For example, a machine learning model (MLM) might represent a first portion of an anatomical structure as including a first density / stiffness level, and a second portion of the anatomical structure as including a second density / stiffness level that is denser / stiffer than the first level. As described above, these regions can be localized using robot and / or navigation data. The control system 60 can then define a custom haptic object (Hobj) for each region and / or can tune boundary constraints differently for each corresponding region in the boundary constraint model (HMb). In some cases, the haptic object (Hobj) may be temporarily extended based on specific interaction representations.

[0175] For spinal procedures, a machine learning model (MLM) can characterize the outer cortical bone layer and cancellous bone region within the pedicle region. These regions can be characterized by their respective densities and geometries. Based on these characterizations, a tunable active constraint model (HMa) can be used to provide stricter constraints on specified degrees of freedom of movement of the surgical instrument 20 within the cancellous bone region compared to the outer cortical region.

[0176] 3. Protective Measures

[0177] Any of the representations described can be used to prevent interaction with sensitive, critical, and / or adverse areas. Such areas may be part of or separate from anatomical structures. This protection can be achieved using the tactile control techniques described above or other methods.

[0178] For example, a machine learning model (MLM) can characterize a first tissue type and a second tissue type. Robotic data and / or navigation data can be used to locate the first and second tissue type regions. In response to this characterization, the control system 60 can generate one or more virtual boundaries VB to demarcate the first and second tissue type regions. The virtual boundaries VB are then registered to the anatomical structure. The virtual boundaries VB are used to constrain, limit, or prohibit the movement or manipulation of the manipulator 14 and / or surgical instrument 20 relative to the virtual boundaries VB. For example, the first tissue type region may be bone, and the second tissue type region may be soft tissue. The virtual boundaries VB enable manipulation of the skeleton while limiting interaction with the soft tissue.

[0179] Similarly, in the examples above, the first tissue type region can be healthy tissue, and the second tissue type region can be malignant tissue. For example, using embedded objects and geometric features, a machine learning model (MLM) can characterize the size of a tumor and the depth of a tumor embedded within soft tissue. In response, the control system 60 can define a virtual boundary VB around the malignant tissue via predefined edges to facilitate the removal of the malignant tissue while minimizing the risk of recurrence. Similar techniques can be used to define the virtual boundary VB, which demarcates the anatomical region of interest from any objects (e.g., retractors) that may interfere with the region of interest.

[0180] 4. Adaptive Control

[0181] In another example, the control system 60 may adaptively control the manipulator 14 and / or the surgical tool 20 based on any of the described representations. Such adaptive control may include any one or more of the following: generating or modifying the tool path TP of the surgical tool 20; modifying the feed rate (path speed) of the surgical tool 20; modifying the cutting speed of the surgical tool 20 (e.g., sagittal or rotational); defining the cutting depth of the surgical tool 20; defining the oscillation frequency of the tool path TP; adjusting the command pose of the surgical tool 20; pausing the movement of the surgical tool 20; moving the surgical tool 20 toward or away from an anatomical structure; generating errors; and / or generating or modifying the virtual haptic settings of the surgical tool 20, as described above.

[0182] For example, a machine learning model (MLM) can substantially characterize a first density level of a first region of bone and a second (higher) density level of a second region of bone. The control system 60 can then utilize the first and second density levels to generate a customized toolpath TP, designed to sculpt both regions in a manner that improves tool performance or reduces tool overload. For example, based on the corresponding densities, a first portion of the toolpath TP can be defined as removing a first region of bone. The first portion of the toolpath TP can specify a first depth of cut, cutting speed, feed rate, and / or toolpath oscillation frequency. A second portion of the toolpath TP (for removing a second portion) can specify a deeper depth of cut, a slower cutting speed, a slower feed rate, and / or a higher toolpath oscillation frequency.

[0183] In another example, a machine learning model (MLM) can characterize the contact point as an osteophyte. Simultaneously, a virtual skeleton model can identify the contact point as cortical bone. The control system 60 can compare the contact point information to identify differences. The control system 60 can then generate an error or feedback. For example, a notification can be presented on display 38 to prompt the surgeon to remove the osteophyte, and / or the control system 60 can pause the operation of the surgical instruments 20 and / or manipulator 14 until the error is resolved.

[0184] Several embodiments have been described in the foregoing description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive in nature. In view of the foregoing teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.

Claims

1. A surgical system comprising: a robotic manipulator comprising a plurality of links and joints; a surgical tool coupled to the robotic manipulator and configured to manipulate a tissue of a patient; a sensing system configured to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical tool to interact with the tissue; obtain measurements of the displacement, the velocity, and the interaction force from the sensing system in response to the interaction of the surgical tool with the tissue over time; and implement a machine learning model configured to: receive and process the measurements to estimate stiffness and damping parameters of the tissue; and monitor changes in the estimated stiffness and damping parameters to predict a tissue characterization; and wherein the control system is configured to control the robotic manipulator and / or the surgical tool based on the tissue characterization.

2. The surgical system of claim 1, wherein the machine learning model comprises a first neural network comprising an input layer, hidden layers, and an output layer, wherein: the input layer of the first neural network is configured to receive the measurements; and the output layer of the first neural network is configured to output the estimated stiffness and damping parameters.

3. The surgical system of claim 2, wherein the machine learning model comprises a first long short-term memory (LSTM) coupled to the input layer of the first neural network, wherein the first LSTM comprises a recurrent neural network (RNN) and is configured to: receive the measurements over time; modify weights of the RNN to learn long-term dependencies among the measurements and selectively filter the measurements; and provide the filtered measurements to the input layer of the first neural network.

4. The surgical system of claim 2, wherein the estimated stiffness and damping parameters are lower-dimensional embeddings compared to the measurements.

5. The surgical system of claim 2, wherein the machine learning model comprises a second neural network comprising an input layer, hidden layers, and an output layer, wherein: the input layer of the second neural network is configured to receive the estimated stiffness and damping parameters; and the output layer of the second neural network is configured to output the tissue characterization.

6. The surgical system of claim 5, wherein the machine learning model comprises a second LSTM coupled to the input layer of the second neural network, wherein the second LSTM comprises a second RNN and is configured to: receive the estimated stiffness and damping parameters from the output layer of the first neural network over time; ​ ​ modify weights of the second RNN to learn long-term dependencies among the estimated stiffness and damping parameters and selectively filter the estimated stiffness and damping parameters; and provide the filtered estimated stiffness and damping parameters to the input layer of the second neural network.

7. The surgical system of claim 6, wherein the second LSTM is further configured to receive measurements of the velocity of the surgical tool over time.

8. The surgical system of claim 5, wherein the tissue characterization is represented by a vector of higher dimensionality compared to the estimated stiffness and damping parameters.

9. The surgical system of claim 1, wherein the tissue characterization comprises a parameter of the tissue.

10. The surgical system of claim 9, wherein the parameter of the tissue comprises one or more of: geometry, size, shape, depth, and thickness.

11. The surgical system of claim 9, wherein the parameter of the tissue comprises one or more of: density, stiffness, hardness, softness, smoothness, and roughness.

12. The surgical system of claim 1, wherein the tissue characterization comprises one or more of: stratification of the tissue and identification of the tissue being an embedded object.

13. The surgical system of claim 1, wherein the tissue characterization comprises a tissue type.

14. The surgical system of claim 13, wherein the tissue type comprises one or more of: bone, osteophyte, cartilage, soft tissue, muscle, ligament tissue, tendon tissue, blood vessel, healthy tissue, and malignant tissue.

15. The surgical system of claim 13, wherein: the machine learning model is configured to monitor changes in the estimated stiffness and damping parameters to predict a first tissue type and to predict a second tissue type; and the control system is configured to: record positions of the surgical tool in response to interactions of the surgical tool with the first tissue type and the second tissue type over time; and utilize the recorded positions to register the first tissue type and the second tissue type to positions on the tissue.

16. The surgical system of claim 15, wherein the control system is further configured to: generate a virtual boundary, the virtual boundary to delimit a first region of the tissue comprising the first tissue type and a second region of the tissue comprising the second tissue type; register the virtual boundary to the tissue; and utilize the virtual boundary to constrain movement or operation of the robotic manipulator and / or the surgical tool relative to the virtual boundary.

17. The surgical system of claim 15, wherein: the first tissue type is bone and the second tissue type is soft tissue; or the first tissue type is healthy tissue and the second tissue type is malignant tissue.

18. The surgical system of claim 13, wherein: the machine learning model is configured to monitor changes in the estimated stiffness and damping parameters to predict that the tissue type is bone; and the control system is configured to: record positions of the surgical tool in response to interactions of the surgical tool with the bone over time; and utilize the recorded positions to generate a 3-D surface model of the bone and register the 3-D surface model to the bone.

19. The surgical system of claim 1, wherein the control system controls the robotic manipulator and / or the surgical tool based on the tissue characterization by being configured to perform one or more of: modify a tool path of the surgical tool; modify a feed rate of the surgical tool; modify a cutting speed of the surgical tool; adjust a commanded pose of the surgical tool; pause movement of the surgical tool; move the surgical tool toward or away from the tissue; and / or generate or modify a virtual haptic setting of the surgical tool.

20. The surgical system of claim 1, wherein: the control system is configured to calculate a deformation of the tissue based on the measured values of displacement and interaction force; and the machine learning model is configured to further receive and process a value of the deformation to estimate stiffness and damping parameters of the tissue.

21. The surgical system of claim 1, wherein the control system is configured to: record positions of the surgical tool in response to interactions of the surgical tool with the tissue over time; and utilize the recorded positions to register the tissue characterization to positions on the tissue.

22. The surgical system of claim 1, wherein: the surgical tool is configured to cut the tissue; and the interaction force is applied to the surgical tool based on interactions of the surgical tool with the tissue during cutting of the tissue.

23. The surgical system of claim 1, wherein the sensing system includes at least one force / torque sensor configured to measure an interaction force applied to the surgical tool, wherein the at least one force / torque sensor is coupled to one or both of: a distal link of the robotic manipulator, and the surgical tool.

24. A method of operating a surgical system, the surgical system comprising: a robotic manipulator comprising a plurality of links and joints; a surgical tool coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool; and a control system coupled to the robotic manipulator and the sensing system, the method comprising the control system performing the steps of: controlling the robotic manipulator for moving the surgical tool to interact with the tissue; obtaining, from the sensing system, measurements of the displacement, the velocity, and the interaction force in response to the surgical tool interacting with the tissue over time; and implementing a machine learning model configured to receive and process the measurements to estimate stiffness and damping parameters of the tissue and monitor changes in the estimated stiffness and damping parameters to predict a tissue characterization; and controlling the robotic manipulator and / or the surgical tool based on the tissue characterization.

25. A non-transitory computer readable medium for use with a surgical system, the surgical system comprising: a robotic manipulator having a plurality of links and joints; a surgical tool coupled to the robotic manipulator to manipulate tissue of a patient; and a sensing system to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool, the non-transitory computer-readable medium comprising instructions configured to, when executed by one or more processors: control the robotic manipulator to move the surgical tool to interact with the tissue; obtain, from the sensing system, measurements of the displacement, the velocity, and the interaction force in response to the surgical tool interacting with the tissue over time; and implement a machine learning model configured to: receive and process the measurements to estimate stiffness and damping parameters of the tissue and monitor changes in the estimated stiffness and damping parameters to predict a tissue characterization; and control the robotic manipulator and / or the surgical tool based on the tissue characterization.

26. A surgical system comprising: a robotic manipulator comprising a plurality of links and joints; a surgical tool coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical tool to interact with the tissue; obtain, from the sensing system, measurements of the displacement, the velocity, and the interaction force in response to the surgical tool interacting with the tissue; input the measurements into a machine learning model configured to predict an interaction characterization of the surgical tool; and control the robotic manipulator and / or the surgical tool based on the interaction characterization.

27. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool; and ​ a control system coupled to the manipulator and the sensing system and configured to: obtain, from the sensing system, measurements of the displacement, the velocity, and the interaction force in response to an interaction of the surgical tool with the tissue; and input the measurements into a machine learning model configured to predict a characterization of the tissue.

28. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate a tissue of a patient; and a control system configured to: obtain sensed measurements during an interaction of the surgical tool with the tissue; and apply the sensed measurements to a machine learning model configured to characterize a feature or parameter of the tissue.

29. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate a tissue of a patient; and a control system configured to: obtain sensed measurements during an interaction of the surgical tool with the tissue; and apply the sensed measurements to a machine learning model configured to characterize a type of the tissue.

30. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate a tissue of a patient; and a control system configured to: obtain sensed measurements during an interaction of the surgical tool with the tissue; and apply the sensed measurements to a machine learning model configured to distinguish between different tissue types.

31. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate a tissue of a patient; and a control system configured to: obtain sensed measurements during an interaction of the surgical tool with the tissue; and apply the sensed measurements to a machine learning model configured to characterize a parameter of the tissue, wherein the parameter comprises one or more of: geometry, size, shape, depth, and thickness.

32. A surgical system comprising: a manipulator comprising a plurality of links and joints; a surgical tool coupled to the manipulator and configured to manipulate a tissue of a patient; and a control system configured to: obtain sensed measurements during an interaction of the surgical tool with the tissue; and applying the sensed measurements to a machine learning model configured to characterize an interaction type of the surgical tool, wherein the interaction type comprises soft tissue interaction, bone tissue interaction, or non-contact interaction.

33. A surgical system comprising: a robotic manipulator comprising a plurality of links and joints; a surgical tool coupled to the robotic manipulator and configured to manipulate tissue of a patient; a sensing system configured to measure: a displacement of the surgical tool, a velocity of the surgical tool, and an interaction force applied to the surgical tool; and a control system coupled to the robotic manipulator and the sensing system and configured to: control the robotic manipulator to move the surgical tool to interact with the tissue; obtain, from the sensing system, measurements of the displacement, the velocity, and the interaction force in response to the interaction of the surgical tool with the tissue; input the measurements into a machine learning model configured to predict a characterization of a foreign object located on or embedded within the tissue; and control the robotic manipulator and / or the surgical tool based on the characterization of the foreign object.

34. The surgical system of claim 33, wherein the foreign object comprises one or more of: a surgical implant, another surgical tool, a piece of material, a surgical sponge, or a foreign fluid.

Images