Surgical tracking systems and methods employing machine learning to identify tracking errors

Abstract

The present disclosure relates to surgical tracking systems and methods employing machine learning to identify tracking errors. Tracking systems and methods employing machine learning to identify tracking errors are provided. A tracker is coupled to a surgical object. A localizer is configured to detect a pose of the tracker and generate tracking data related to the tracker. A controller obtains the tracking data from the localizer and inputs the tracking data into a machine learning model configured to identify an error related to the detected pose of the tracker. The controller generates feedback to address the error. The machine learning model can be used for error monitoring of dynamically moving tracked surgical objects and improve accuracy of robotic registration and / or anatomical registration.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 795,825, filed April 28, 2025, the entire contents of which are hereby incorporated by reference. Background Technology

[0003] In surgical settings, surgical objects are typically tracked by tracking systems for several reasons. For example, surgical instruments, robotic systems, and anatomical structures can be tracked to provide navigation guidance of the instruments or robots relative to the anatomical structures. Tracking systems are typically optical tracking systems, such as cameras, that detect the tracker attached to the surgical object. The tracked surgical object usually moves dynamically within the operating room, making highly accurate tracking essential.

[0004] While highly accurate tracking is required in surgical settings, tracking is inherently prone to errors. One type of tracking error relates to non-systematic or externally related factors, such as occlusion of the tracker's line of sight, which reduce the tracker's detectability. Another type of tracking error is systematic error, where the tracker may be fully visible, but the detected pose is still inaccurate. This type of systematic tracking inaccuracy can arise from various causes, such as unpredictable or nonlinear motion of the tracker.

[0005] Conventional tracking systems may fail to accurately detect or characterize such systematic errors, especially when the tracking error exhibits unpredictable or nonlinear behavior. Within the range of detected errors, conventional tracking systems typically compensate for the error by suspending system operation when the tracking error exceeds a default error threshold. For example, some systems suspend operation if the tracker's detected speed exceeds a predetermined speed. However, such mitigation schemes are one-size-fits-all approaches that fail to consider several factors, such as the type of tracker, the type or context of tracker movement, the source of the error, or the context of the surgical procedure. Therefore, conventional tracking systems cannot assess whether the tracking error is acceptable given the circumstances or context in which the error occurred. Consequently, conventional tracking systems are more likely to indiscriminately label tracking errors, thereby interfering with surgical procedures. Furthermore, conventional tracking systems are not well-suited to providing intelligent feedback to address errors because such systems do not understand the nature of the error and instead blindly rely on default error thresholds. Summary of the Invention

[0006] The present invention presents a series of concepts in a simplified form, which are further described in the detailed embodiments below. The present invention is not intended to limit the scope of the claimed subject matter, nor is it intended to identify key or essential features of the claimed subject matter.

[0007] According to a first aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; and input the tracking data into a machine learning model configured to identify errors associated with the tracker.

[0008] According to a second aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; and input the tracking data into a machine learning model configured to identify errors associated with the tracker and predict the sources of the errors.

[0009] According to a third aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; and input the tracking data into a machine learning model configured to identify errors associated with the tracker and predict suggestions to resolve the errors.

[0010] According to a fourth aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical instrument; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors associated with the tracker and estimate actions on the surgical instrument to resolve the errors.

[0011] According to a fifth aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; and input the tracking data into a machine learning model configured to identify errors associated with the tracker and predict that the errors should be ignored.

[0012] According to a sixth aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors associated with the tracker and control the tracker to resolve the errors.

[0013] According to a seventh aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a robotic manipulator including a robotic arm formed by links and joints; a locator coupled to and movable with the robotic arm, and configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model, the machine learning model being configured to identify errors associated with the tracker and control the robotic arm to adjust the pose of the locator to resolve the errors.

[0014] According to an eighth aspect, a surgical system is provided, the surgical system comprising: a tracker coupled to a surgical object; a head-mounted device including a display configured to be positioned in front of the eyes of a wearer of the head-mounted device; a locator coupled to the head-mounted device and movable with the head-mounted device, and configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model, the machine learning model being configured to identify errors associated with the tracker and provide feedback to resolve the errors.

[0015] According to a ninth aspect, a surgical system is provided, the surgical system comprising: a head-mounted device including a display configured to be positioned in front of the eyes of a wearer of the head-mounted device; a locator coupled to and movable with the head-mounted device, and configured to detect the posture of an object (such as a human hand) in an operating room and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors related to the detected posture of the object and provide feedback to resolve the errors.

[0016] According to a tenth aspect, a surgical tracking system is provided, the surgical tracking system comprising: a tracker coupled to a surgical object; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors related to the detected pose of the tracker; and generate feedback to resolve the errors.

[0017] According to an eleventh aspect, a surgical system is provided, the surgical system comprising: a tracker coupled to an anatomical structure; a robotic manipulator coupled to the anatomical structure; a locator configured to detect the pose of the tracker and generate tracking data; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors associated with the tracker and control a robotic arm to adjust the pose of the anatomical structure to resolve the errors.

[0018] According to a twelfth aspect, a surgical system is provided, the surgical system comprising: a robotic manipulator including a robotic arm configured to support a surgical instrument and move the surgical instrument relative to an anatomical structure; and a surgical tracking system comprising: a manipulator tracker coupled to a component of the robotic manipulator; an anatomical structure tracker coupled to the anatomical structure; a locator configured to detect the pose of the manipulator tracker and the pose of the anatomical structure tracker, and to generate tracking data associated with the manipulator tracker and the anatomical structure tracker; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors associated with the detected pose of one or both of the manipulator tracker and the anatomical structure tracker; and generate feedback to resolve the errors.

[0019] According to a thirteenth aspect, a surgical system is provided, the surgical system comprising: a robotic manipulator including a base and a robotic arm coupled to the base, wherein the robotic arm supports an end effector; and a surgical tracking system comprising: a manipulator tracker coupled to the robotic manipulator; a registration tracker configured to be attached to the end effector; a locator configured to detect the pose of the manipulator tracker and the registration tracker, and to generate tracking data associated with the manipulator tracker and the registration tracker; and a controller coupled to the robotic manipulator and the locator, and configured to: obtain the tracking data from the locator; input the tracking data into a machine learning model configured to identify errors associated with the detected pose of one or both of the manipulator tracker and the registration tracker; and generate feedback to resolve the errors.

[0020] According to a fourteenth aspect, a surgical system is provided, the surgical system comprising: a robotic manipulator including a robotic arm configured to support a surgical tool and move the surgical tool relative to an anatomical structure, wherein the surgical tool includes a TCP; and a surgical tracking system comprising: a manipulator tracker coupled to a component of the robotic manipulator; an anatomical structure tracker coupled to the anatomical structure; and a locator configured to detect the pose of the manipulator tracker and the pose of the anatomical structure tracker, and The system generates tracking data associated with the manipulator tracker and the anatomical structure tracker; and a controller coupled to the locator and configured to: obtain the tracking data from the locator; calculate an expected error for the manipulator tracker and the anatomical structure tracker based on the tracking data; calculate an expected true transformation of the manipulator tracker and the expected anatomical structure tracker based on the expected error; calculate a measured TCP value and a true TCP value relative to the coordinate system of the anatomical structure based on the tracking data and the expected true transformation; calculate the error between the measured TCP value and the true TCP value; and determine whether the error exceeds a threshold.

[0021] A computer-implemented method is provided to perform any steps implemented by a controller of any one or more of the foregoing aspects. A non-transitory computer-readable medium or computer program product is provided, comprising instructions configured, when executed by one or more processors, to implement any feature of one or more of the foregoing aspects.

[0022] Any aspect described above can be combined, in whole or in part. Any aspect above can be used, in whole or in part, with any of the following implementations:

[0023] The locator can achieve optical, electromagnetic, inertial, radio frequency, or machine vision-based tracking. The tracker may include tracking elements detectable by the locator. The tracker can be coupled to any surgical object, such as surgical instruments; anatomical structures; head-mounted displays; links of a robotic arm of a robotic manipulator; adjustable arms coupled to a base of the robotic manipulator; end effectors coupled to the robotic manipulator; and / or couplers located between the robotic arm of the manipulator and the end effector supported by the robotic arm. The tracker may include a manipulator tracker, an anatomical structure tracker, an tool tracker, or a registration tracker used during the registration process.

[0024] The controller can acquire identification data of the tracker. The controller can input the identification data into the machine learning model so that the machine learning can identify the error. The identification data may include the geometry of the tracking element of the tracker. The tracking data may include the position and / or orientation of the tracker. The tracking data may include the velocity and / or acceleration of the tracker. The tracking data may also include the velocities of manipulator trackers and anatomical structure trackers. The controller can also calculate the expected error using the tracker type associated with each tracker.

[0025] The machine learning model includes an (artificial) neural network, such as a deep learning neural network. The neural network includes an input layer, hidden layers, and an output layer, wherein the input layer is configured to receive the tracking data, and the output layer is configured to identify the error. The machine learning model may include at least one Long Short-Term Memory (LSTM) network coupled to the input layer of the neural network, wherein the at least one LSTM includes a Recurrent Neural Network (RNN) and is configured to: receive the tracking data over time; modify the weights of the RNN to learn long-term dependencies between the tracking data and selectively filter the tracking data; and provide the filtered tracking data to the input layer of the neural network. The machine learning model may calculate a predicted error for one or more trackers based on the tracking data, the identification data, and any one or more of the tracker types.

[0026] The error may be related to inaccuracies in the pose detected by the tracker. The machine learning model may be pre-trained based on baseline real data derived from laser tracking of the tracker or other trackers. The machine learning model may identify the error based on a comparison between the tracking data and the baseline real data. The machine learning model may establish an error threshold or range that dynamically varies based on the tracking data. The machine learning model may identify the error in response to the error exceeding the error threshold or range. The machine learning model may predict the source of the error. The source may be any one or more of the following: unintended movement of the surgical object; unintended movement of the locator; or incorrect calibration data of the locator or the tracker. The machine learning model may estimate the impact of the error on the accuracy of one or more of the following: the locator, the robot manipulator, or the anatomical structure. The error may include the difference between the measurement tool center point (TCP) value and the true TCP value, and the controller may use the registration transformation and kinematic transformation of the manipulator derived from the robot registration process to calculate the error.

[0027] The controller can provide feedback, such as notifications or warnings, to inform the user of the error. The controller can provide feedback to counteract the error by selectively adjusting or replacing the tracking data. The controller can provide feedback by filtering the tracking data. The controller can provide feedback by controlling the tracker. The controller can provide feedback by adjusting the operation or positioning of the locator. The controller can provide feedback during robot registration or anatomical registration by: filtering tracking data using the machine learning model; comparing filtered and unfiltered tracking data with kinematic data or a virtual model; presenting multiple registration results on a display; and prompting the user to select one of the results. The controller can also provide suggestions to correct the error and can input force data from surgical probes into the machine learning model to identify errors associated with unwanted tissue interactions. Attached Figure Description

[0028] The advantages of this disclosure will become readily apparent as it is better understood by taking into account the accompanying drawings and the following detailed description.

[0029] Figure 1 It is a perspective view of a surgical system based on one implementation method.

[0030] Figure 2 This is a block diagram of an example control system used to control a surgical system.

[0031] Figure 3 It is a functional block diagram based on an implementation method that uses a machine learning model to identify tracking errors and generate feedback to resolve tracking errors.

[0032] Figure 4A An example is illustrated by dynamically monitoring the relationship between robot manipulators, anatomy, and locators to identify tracking errors, based on one example.

[0033] Figure 4B It uses machine learning models to identify tracking errors (such as...) during dynamic error monitoring. Figure 4A The functional block diagram is shown below.

[0034] Figure 4C This is a function block diagram of another implementation for identifying tracking errors during dynamic error monitoring.

[0035] Figures 5A to 5D This illustrates the various types of manipulator trackers that can be involved in the robot registration process.

[0036] Figure 6 It is a functional block diagram based on one implementation method that uses a machine learning model to identify tracking errors during robot registration.

[0037] Figures 7A to 7C An example is given of the anatomical registration process involving the tracked anatomical region, the virtual model of the anatomical structure, and the tracked pointer.

[0038] Figure 8 It is a functional block diagram based on one implementation method that uses a machine learning model to identify tracking errors during anatomical registration. Detailed Implementation

[0039] I. Example System Overview

[0040] refer to Figure 1 An example of a surgical / medical system 10 is shown. As will be described below, system 10 can be used in conjunction with a surgeon or a patient for surgical / medical purposes.

[0041] For surgical procedures, system 10 can be used to treat the surgical site or target anatomical structure (A) of patient 12, such as bone or soft tissue. Figure 1 In the middle, patient 12 is undergoing a surgical procedure. Figure 1 The anatomical structures (A) in the document include the femur (F) and tibia (T) of patient 12. Surgical procedures may involve tissue removal or other forms of treatment. Treatment may include cutting, coagulation, tissue ablation, other in-situ tissue treatments, etc. In some examples, the surgical procedure involves partial or total knee or hip replacement surgery, shoulder replacement surgery, spinal surgery, or ankle surgery. In some examples, system 10 is designed to remove material to be replaced by a surgical implant, such as hip and knee implants, including single-compartment, double-compartment, multi-compartment, or total knee implants. Some of these types of implants are illustrated in U.S. Patent Application Publication No. 2012 / 0330429 entitled “Prosthetic Implant and Method of Implantation,” the disclosure of which is hereby incorporated by reference. The system 10 and techniques disclosed herein can be used to perform other procedures (surgical or non-surgical procedures) or in industrial or other applications utilizing robotic systems.

[0042] The surgical / medical system 10 may include a manipulator 14 or a robotic manipulator. The manipulator 14 may be a collaborative surgical robot. Collaboration means that the surgeon / patient interacts directly with the manipulator 14 by applying forces to it. The manipulator 14 has a base 16 and a plurality of links 18 and joints (J). A manipulator trolley 17 supports the manipulator 14 such that the manipulator 14 is secured to the manipulator trolley 17. The links 18 collectively form one or more arms of the manipulator 14. The manipulator 14 may have a tandem arm configuration (e.g., Figure 1 (As shown), parallel arm configuration, or any other suitable manipulator configuration. In other examples, more than one manipulator 14 may be used in a multi-arm configuration. In other configurations, manipulator 14 may include part or all of an adjustable arm (AA) with manually adjustable passive joints. The user may optionally lock the joints of the adjustable arm (AA) in place. In other cases, joint motors may be used to actively drive some or all of the joints of the adjustable arm (AA).

[0043] Manipulator 14 may include passive (manually articulated) joints, such as planar-extending joints. For example, a passive joint may be the most distal joint attached to a robotic arm that includes multiple active joints. A planar-extending joint may support a tool, such as a saw blade, to allow a user to manually move the saw blade along a plane. In this case, the tool is mechanically constrained to the plane by the mechanical constraints of the passive joint, while the orientation of the plane is actively constrained by the active joints (J) of the robotic arm. In other cases, one or more joints (J) of manipulator 14 may be controlled to constrain the tool within a specific degree of freedom while allowing the tool to move freely passively in other degrees of freedom or in other ways, such as rotation about a point or selected linear motion.

[0044] Manipulator 14 can exhibit kinematic redundancy, where the tool position is defined by up to 6 DOFs, but the robotic arm can operate over more than 6 DOFs. This redundancy allows manipulator 14 to utilize zero-space control, whereby the joints (J) of manipulator 14 can change position while maintaining the orientation of the end effector 22. Zero-space control can also be used to avoid singularities.

[0045] exist Figure 1 In the example shown, the manipulator 14 includes multiple joints (J) and multiple joint encoders 19 located at the joints (J) for determining the position data of the joints (J). For simplicity, in Figure 1Only one joint encoder 19 is illustrated, but other joint encoders 19 can be similarly illustrated. According to one example, the manipulator 14 has six joints J1-J6 that implement at least six degrees of freedom (DOF) of the manipulator 14. However, the manipulator 14 can have any number of degrees of freedom and any suitable number of joints J, and can have redundant joints. The manipulator 14 does not necessarily require the joint encoder 19, but can alternatively or additionally utilize a motor encoder present on the motor at each joint J. Furthermore, the manipulator 14 does not necessarily require rotary joints, but can alternatively or additionally utilize one or more translational joints. Any suitable combination of joint types is contemplated.

[0046] The base 16 of the manipulator 14 is part of a fixed reference coordinate system that provides a reference coordinate system for the manipulator 14 or, in general, other components of the system 10. The origin of the manipulator coordinate system MNPL is defined at the fixed reference point on the base 16. The base 16 can be defined relative to any suitable part of the manipulator 14, such as one or more of the links 18. Alternatively or additionally, the base 16 can be defined relative to the manipulator trolley 17, such as at the location where the manipulator 14 is physically attached to the manipulator trolley 17. In one example, the base 16 is defined at the intersection of the axes of joints J1 and J2. Thus, although joints J1 and J2 are actually moving components, the intersection of the axes of joints J1 and J2 remains a virtual fixed reference attitude that provides both fixed position and orientation references and does not move relative to the manipulator 14 and / or the manipulator trolley 17.

[0047] In other examples, the manipulator 14 may be a handheld manipulator, wherein the base 16 is the base portion of the tool (e.g., the portion held by the user's hand) and the tool tip is movable relative to the base portion. The base portion has a tracked reference coordinate system, and the tool tip has a tool tip coordinate system calculated relative to the reference coordinate system (e.g., calculated via motor and / or joint encoders and forward kinematics). Since the orientation of the tool tip relative to the path can be determined, the movement of the tool tip can be controlled to follow the path. The handheld manipulator 14 may be as described and shown in US20230255701 entitled "Systems and Methods for Guiding Movement of a Handheld Medical Robotic Instrument," the entire disclosure of which is hereby incorporated by reference.

[0048] Manipulator 14 and / or manipulator trolley 17 house manipulator controller 26 or other types of control units. Manipulator controller 26 may include one or more computers, or any other suitable form of controller that directs the movement of manipulator 14. Manipulator controller 26 may have a central processing unit (CPU), graphics processing unit (GPU) and / or other processors, memory, and storage. Manipulator controller 26 is loaded with the software described below. Processor may include one or more processors for controlling the operation of manipulator 14. Processor may be any type of microprocessor, multiprocessor, and / or multicore processing system. Manipulator controller 26 may additionally or alternatively include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuits, and / or other suitable hardware, software, or firmware capable of performing the functions described herein. The term processor is not intended to limit any embodiment to a single processor. Manipulator 14 may also include a user interface UI with one or more displays and / or input devices (e.g., buttons, keyboard, mouse, microphone (voice-activated), gesture control devices, touchscreen, etc.).

[0049] Tool 20 is coupled to manipulator 14 and is movable relative to base 16 to interact with anatomical structures (A) in certain modes. Tool 20 is a physical and surgical tool and, in some implementations, is an end effector 22 supported by manipulator 14 or forms part of it. Tool 20 or end effector 22 can be grasped by a user. One possible arrangement of manipulator 14 and tool 20 is described in U.S. Patent No. 9,119,655 entitled “Surgical Manipulator Capable of Controlling a Surgical Instrument in Multiple Modes,” the disclosure of which is hereby incorporated by reference. Manipulator 14 and end effector 22 may be arranged alternatively. End effector 22 may be as shown in U.S. Patent Application Publication No. 2014 / 0276949, filed March 15, 2014, entitled “End Effector of a Surgical Robotic Manipulator,” which is hereby incorporated by reference.

[0050] Tool 20 may include an energy applicator 24 designed to contact and remove tissue from patient 12 at a surgical site. In one example, energy applicator 24 is a drill 25. Drill 25 may be substantially spherical and include a spherical center, radius (r), and diameter. Alternatively, energy applicator 24 may be a drill bit, saw blade, ultrasonic vibrating tip, etc. Tool 20 and / or energy applicator 24 may include any geometric features such as perimeter, circumference, radius, diameter, width, length, volume, area, surface / plane, range of motion envelope (along any one or more axes), etc. Geometric features may be considered to determine how tool 20 is positioned relative to tissue at the surgical site to perform the desired treatment. In some embodiments described herein, for convenience and ease of illustration, a spherical drill with a tool center point (TCP) will be described, but it is not intended to limit tool 20 to any particular form. In other examples, tool 20 does not include energy applicator 24. For example, tool 20 can be a grooving cutting guide for sawing, a guide tube for receiving another tool, or a tool guide, etc.

[0051] The end effector 22 may include a tool controller to control the operation of the end effector 22, such as controlling power to the tool (e.g., to a rotary motor of tool 20), controlling movement of tool 20, controlling flushing / suction of tool 20, etc. The tool controller may communicate with manipulator controller 26 or other components. Manipulator controller 26 controls the state (position and / or orientation) of tool 20 (e.g., TCP) relative to a coordinate system such as manipulator coordinate system MNPL. Manipulator controller 26 may control the (linear or angular) velocity, acceleration, or other kinematic derivatives of tool 20. In one example, the tool center point (TCP) is a predetermined reference point defined at energy applicator 24. TCP has a known or computable (i.e., not necessarily static) attitude relative to other coordinate systems. The geometry of energy applicator 24 is known in or defined relative to the TCP coordinate system. TCP may be located at the center of the sphere of drill 25 of tool 20, such that only one point is tracked. TCP may be defined in several ways depending on the configuration of energy applicator 24. Manipulator 14 may employ a joint / motor encoder or any other non-encoder position sensing method to determine the TCP's attitude. Manipulator 14 may use joint measurements to determine the TCP's attitude and / or employ techniques for directly measuring the TCP's attitude. Control of tool 20 is not limited to a center point. For example, any suitable primitives, meshes, etc., may be used to represent tool 20.

[0052] The manipulator 14 can operate in multiple modes. In free mode, the manipulator moves the end effector 22 in response to force applied by the user on the manipulator 14. In manual mode, the manipulator moves the end effector 22 in response to force applied by the user, but constrains the end effector 22 relative to a tactile object or virtual boundary / constraint. In guided manual mode, the manipulator moves the end effector 22 in response to force applied by the user, but constrains the end effector 22 to a predetermined tool path. In semi-automatic or automatic mode, the manipulator automatically moves the end effector 22 along a predetermined tool path, and optionally under the supervision of the surgeon.

[0053] The surgical / medical system 10 may include a tracking system 32 or a navigation system. The tracking / navigation system 32 tracks the movement of various surgical objects. Such objects include, for example, manipulators 14, tools 20 / end-effectors 22 and anatomical structures (A) (e.g., femur F and tibia T), or patient limbs or any other surgical tools / instruments / devices. The tracking / navigation system 32 tracks these surgical objects to acquire state information for each object relative to the (navigation) locator coordinate system LCLZ. Transformations can be used to transform coordinates in the locator coordinate system LCLZ to the manipulator coordinate system MNPL and / or vice versa. An example of the tracking / navigation system 32 is described in U.S. Patent No. 9,008,757, filed September 24, 2013, entitled “Navigation System Including Optical and Non-Optical Sensors,” which is hereby incorporated by reference.

[0054] The tracking / navigation system 32 includes a cart assembly 34 housing a navigation controller 36 and / or other types of control units. A navigation user interface (UI) communicates operationally with the navigation controller 36. The navigation user interface includes one or more displays 38. The tracking / navigation system 32 can use the one or more displays 38 to display a graphical representation of the relative status of the tracked object to a user. The navigation user interface UI also includes one or more input devices to input information into the navigation controller 36 or otherwise select / control certain aspects of the navigation controller 36. Such input devices include interactive touchscreen displays. However, input devices may include any one or more of the following: buttons, keyboard, mouse, microphone (voice-activated), gesture control devices, etc.

[0055] The tracking / navigation system 32 also includes a locator 44 coupled to the navigation controller 36. In one example, the locator 44 is an optical locator and includes a camera unit 46. The camera unit 46 has a housing 48 that houses one or more optical sensors 50. The locator 44 may include its own locator controller 49 and may also include a camera VC.

[0056] The tracking / navigation system 32 includes one or more trackers. In one example, the trackers include a pointer tracker PT, one or more manipulator trackers 52A, 52B, a first patient tracker 54, and a second patient tracker 56. Figure 1 In the illustrated example, the manipulator tracker is rigidly attached to the end effector 22 (i.e., tracker 52A), the first patient tracker 54 is coupled to the femur F of patient 12, and the second patient tracker 56 is coupled to the tibia T of patient 12. Patient trackers 54 and 56 may be securely attached to a segment of bone or coupled to the exterior of the leg. A pointer tracker PT is securely attached to a pointer or probe P, which is used to calibrate the anatomical structure (A) to the locator coordinate system LCLZ. Pointer P may be handheld or optionally coupled to an adjustable arm (AA), such as... Figure 1 As shown. The pointer P may include a force / torque sensor configured to measure the force / torque applied to the pointer tip. The pointer P may also include, for example, a controller housed within the pointer body to process the measurement results from the force / torque sensor. The measurement results from the pointer P can be transmitted to the control system 60 using a wired or wireless connection.

[0057] Manipulator trackers 52A and 52B can be attached to any suitable component of manipulator 14 other than end effector 22, such as base 16 (i.e., tracker 52B), or any one or more links 18 of manipulator 14. Trackers 52A, 52B, 54, 56, and PT can be fixed to their respective components in any suitable manner. For example, trackers can be rigidly fixed, flexibly connected (fiber optic), or not physically connected at all (ultrasound), as long as there is a suitable (supplementary) method to determine the relationship (measurement result) between the respective tracker and the associated object.

[0058] Any one or more of the trackers may include an active tracking element 58. The active tracking element 58 may include a light-emitting diode (LED). Alternatively, trackers 52A, 52B, 54, 56, and PT may have passive tracking elements, such as reflectors that reflect light emitted from camera unit 46. Other suitable tracking elements not specifically described herein may be utilized.

[0059] Positioner 44 tracks trackers 52A, 52B, 54, 56, and PT to determine the state of each tracker 52A, 52B, 54, 56, and PT, which corresponds to the state of the object to which it is attached. Positioner 44 can perform known triangulation techniques to determine the state of trackers 52A, 52B, 54, 56, and PT and the associated object. Positioner 44 provides the state of trackers 52A, 52B, 54, 56, and PT to navigation controller 36. In one example, navigation controller 36 determines the state of trackers 52A, 52B, 54, 56, and PT and transmits it to manipulator controller 26. As used herein, the state of an object includes, but is not limited to, data defining the position and / or orientation of the tracked object, or equivalent quantities / derivatives of position and / or orientation. For example, the state can be the attitude of the object and may include linear velocity data and / or angular velocity data, etc.

[0060] The navigation controller 36 may include one or more computers, or any other suitable form of controller. The navigation controller 36 may have a central processing unit (CPU), a graphics processing unit (GPU) and / or other processors, non-transitory memory, and storage. The processor may be any type of processor, microprocessor, or multiprocessor system. The navigation controller 36 is loaded with software. For example, the software converts signals received from the locator 44 into data representing the position and orientation of the object being tracked. The navigation controller 36 may additionally or alternatively include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuitry, and / or other suitable hardware, software, or firmware that can implement the functions described herein. The term processor is not intended to limit any implementation to a single processor.

[0061] The tracking / navigation system 32 may have any other suitable configuration for tracking surgical objects. For example, the tracking / navigation system 32 and / or the locator 44 may be radio frequency (RF) based. For example, the tracking / navigation system 32 may include an RF transceiver coupled to the navigation controller 36. The tracker may include an RF tracking element (e.g., a transmitter or transponder) attached thereto. The RF tracking element may be passive or actively powered. The RF transceiver transmits RF tracking signals and generates status signals to the navigation controller 36 based on the RF signals received from the RF tracking element. The navigation controller 36 may analyze the received RF signals to correlate with relevant statuses. The RF signals may have any suitable frequency. The RF transceiver may be positioned at any suitable location to effectively track the object using the RF signals. Furthermore, the RF tracking element may have any suitable structural configuration, which may be largely compatible with... Figure 1 The trackers shown are different: 52A, 52B, 54, 56, and PT.

[0062] In another example, the tracker can use inertial sensing for detection, such as using accelerometers, gyroscopes, magnetometers, MEMS sensors, etc. The tracking / navigation system 32 can extract inertial sensor information from the tracker using any suitable communication scheme (such as IR, RF, EM communication, etc.).

[0063] In yet another example, the tracking / navigation system 32 and / or the locator 44 are electromagnetic. For example, the tracking / navigation system 32 may include an EM transceiver coupled to the navigation controller 36. The tracker may include an EM tracking element attached thereto, such as any suitable magnetic tracker, electromagnetic tracker, inductive tracker, etc. The EM tracking element may be passive or actively powered. The EM transceiver generates an EM field and generates a status signal to the navigation controller 36 based on the EM signals received from the EM tracking element. The navigation controller 36 may analyze the received EM signals to correlate with the relevant status. Similarly, such navigation system examples may have... Figure 1 The tracking / navigation system 32 shown is configured with different structural configurations.

[0064] Alternatively, the tracking / navigation system 32 and / or the locator 44 utilize computer / machine vision. Here, the locator 44 includes a vision camera to identify the patient's anatomical structures (A) and / or any other surgical objects in the operating room using computer / machine vision technology. This technology may be used in conjunction with a tracker that includes optically detectable tracking elements such as QR codes, barcodes, or reference arrays. Alternatively, the technology may be tracker-free. The computer / machine vision system utilizes a camera to detect the movement of objects by continuously capturing images, viewing the images, and detecting the movement of the objects. The computer / machine vision system may also identify and characterize surgical objects. The computer / machine vision system may utilize any suitable image processing algorithm to determine such distance information, including but not limited to depth map generation, segmentation, edge detection, color analysis, blob detection, pixel analysis, pattern recognition, machine learning models, etc. These algorithms may be executed by any one or more of the controllers described herein.

[0065] In another example, the tracking / navigation system 32 and / or the locator 44 are ultrasound-based. For example, the tracking / navigation system 32 may include an ultrasound imaging device coupled to the navigation controller 36. The ultrasound imaging device images any surgical object (or tracker) and generates status signals to the navigation controller 36 based on the ultrasound images. The ultrasound images may be 2D, 3D, or a combination of both. The navigation controller 36 can process the images in near real-time to determine the object's status. The ultrasound imaging device can have any suitable configuration and may differ from, for example, […]. Figure 1 The camera unit 46 shown.

[0066] Tracking / navigation system 32 may have any other suitable components or structures not specifically listed herein. Furthermore, any techniques, methods, and / or components described above with respect to the illustrated tracking / navigation system 32 may be implemented or provided as any other example for the tracking / navigation system 32 described herein. Positioner 44 may be supported by cart assembly 34 (as shown), or may be located on any other device / support, including being attached to and movable with manipulator 14, or attached to a movable / adjustable / robotic arm other than manipulator 14.

[0067] refer to Figure 2 System 10 includes a control system 60, which, among other components, includes a manipulator controller 26, a navigation controller 36, and a tool controller 21. Control system 60 also includes one or more software programs and software modules. Software modules may be part of one or more programs that run on manipulator controller 26, navigation controller 36, tool controller 21, or any combination thereof to process data to assist control system 10. Software programs and / or modules include computer-readable instructions stored in non-transitory memory 64 on manipulator controller 26, navigation controller 36, tool controller 21, or combinations thereof, executable by one or more processors 70 of controllers 21, 26, 36. Memory 64 may be any suitable memory configuration, such as RAM, non-volatile memory, etc., and may be implemented locally or from a remote database.

[0068] The control system 60 may include any suitable configuration of input devices, output devices, and processing devices appropriate for implementing the functions and methods described herein. The control system 60 may include a manipulator controller 26, a navigation controller 36, or a tool controller 21, or any combination thereof, or may include only one of these controllers. These controllers may be accessible via, for example... Figure 2 The wired bus or communication network shown communicates wirelessly or otherwise. Control system 60 may also be referred to as a controller. Control system 60 may include one or more microcontrollers, field-programmable gate arrays, systems-on-a-chip, discrete circuits, sensors, displays, user interfaces, indicators, and / or other suitable hardware, software, or firmware that can perform the functions described herein.

[0069] like Figure 1 and Figure 2As shown, the surgical system 10 may also employ a head-mounted device (HMD). The HMD includes a display 90, an HMD controller 92, a camera 94, a sensor 96, and an optional tracker 98. The display 90 is positioned in front of the wearer's eyes. The display 90 may be transparent or a digital display. The HMD can combine or overlay computer-generated images with or onto a real-world view in an augmented / mixed reality manner. The camera 94 can detect surgical objects in the operating room and can therefore be used as an example of a locator 44. The sensor 96 can detect HMD movement or the wearer's posture to provide input commands. The tracker 98 may be attached to the HMD and can be detected by the locator 44. The tracker 98 can be used for any tracking modality, such as computer vision or optical tracking. The HMD controller 92 can communicate with and control any of the described components of the HMD.

[0070] II. Techniques for using machine learning models to identify tracking errors

[0071] This article continues to refer to Figures 3 to 8 Systems, computer-implemented methods, computer program products, and techniques for using machine learning models to identify tracking errors in tracking / navigation system 32 are described. The techniques described herein, including machine learning models (MLMs), can be implemented by any one or more of the described controllers (e.g., navigation controller 36, manipulation controller 26, tool controller 21, HMD controller 92, etc.), which will be described below as “controllers”.

[0072] The tracking error can be identified relative to any one or more of the described trackers (52, 54, 56, 98), such as those trackers coupled to any surgical object described, including but not limited to: pointer P, components of manipulator 14, HMD, anatomical structure (A), or any other surgical tool / instrument. Regarding manipulator 14, the tracker may be coupled to: any one or more links 18 of the robotic arm of manipulator 14; an adjustable arm coupled to the base 16 of manipulator 14; an end effector 22 coupled to manipulator 14; and / or a coupler located between the robotic arm of manipulator 14 and the end effector 22 supported by the robotic arm. Examples of manipulator trackers may be similar to those described in U.S. Patent Application No. 18 / 198,938 entitled “Robotic System With Improved Configurations For BaseTracker”; examples of link trackers may be similar to those described in U.S. Patent Application Publication No. 20230277256 entitled “Robotic System Including A Link Tracker”; and examples of trackers located on couplers may be similar to those described in U.S. Patent Application Publication No. 20220039898 entitled “Robotic Surgical System Including A Coupler For Connecting A Tool To A Manipulator And Methods Of Using The Coupler”, the entire contents of each of these references are hereby incorporated by reference.

[0073] The techniques described herein can identify tracking errors of various types of trackers used in robot-assisted joint replacement surgery, including robot tracking, dynamic bone tracking, and bone registration. These trackers can have various shapes and sizes and can be used for tracking using any of the described modalities (e.g., RF, EM, inertial, machine / computer vision, ultrasound, or any combination thereof).

[0074] Tracking error can be related to inaccuracies in the detected attitude of any one or more of the trackers described. In other words, the tracker's attitude is detected, but the detected attitude is inaccurate. For example, tracking error may be caused by suboptimal tracking accuracy due to the tracker's position and / or orientation attitude or movement. Specifically, orientation movement of the tracker often introduces nonlinearity by causing the tracker's face to move away from the direct line of sight of the locator 44. Consequently, such orientation movement can cause inaccuracies in the optical tracking system. Tracking error can also be identified when the tracker is partially detected or when the tracker is not detected. In other examples, tracking error may be caused by a variety of other reasons, such as partial line-of-sight obstruction, incorrect calibration data or tracker operation, accidental collisions or movements of the tracker, locator 44, manipulator 14, etc.

[0075] A. Tracking error identification and compensation

[0076] refer to Figure 3 This paper describes a block diagram of a method 200 for tracking error identification and compensation performed by a controller using a machine learning model (MLM). The machine learning model (MLM) can advantageously identify complex and nonlinear systematic tracking errors in a localization system that might otherwise be difficult to capture using conventional techniques. This benefit is enhanced by utilizing supplementary data, such as tracker identification and surgical object data, which will be described below. When tracker identification data is provided as input, the machine learning model (MLM) can make higher-order predictions related to tracker errors, such as errors unique to the specific tracker being detected, and can generalize this learning experience to identify errors associated with other tracker types not previously handled. When a surgical object is provided as input, the machine learning model (MLM) can determine the complex relationship between tracker errors and the surgical environment or surgical context in ways previously unattainable. The machine learning model (MLM) can quantify errors, estimate the impact of errors, selectively ignore errors based on dynamic error thresholds / metrics / ranges, and even predict the source of errors. Several feedback schemes are also envisioned to resolve or compensate for errors.

[0077] Method 200 includes: at step 202, the controller obtains tracking data from the locator 44. In one example, the tracking data is obtained based on the locator 44 detecting the pose of the tracker. The tracking data may include the tracker's pose (position and / or orientation), the tracker's velocity, and / or the tracker's acceleration. The tracking data may be acquired discretely or continuously over any specified duration. In some cases, the tracking data may be based on other circumstances, such as when the locator 44 partially / incompletely detects or fails to detect the tracker's pose. The described method 200 can be used in any of the examples or use cases described herein, such as for robot registration, anatomical registration, dynamic error monitoring, or for tracking any one or more trackers in a surgical workspace for any other reason.

[0078] Optionally, at step 204, the controller may acquire tracker identification data. The tracker identification data is associated with a specific tracker being detected by the locator 44. The tracker identification data specifies information that helps identify the tracker. In one example, the tracker identification data includes information about the tracking elements of the tracker. As described, the locator may implement various tracking modes (e.g., optical, electromagnetic, inertial, radio frequency, or computer / machine vision-based tracking). The tracker includes tracking elements that can be detected by the locator 44 for a given tracking mode. The tracker identification data may include the tracking mode, configuration, or geometry of the tracking elements. For example, the geometry of the tracking elements may define how the tracking elements are arranged relative to each other and relative to the body of the tracker (or the surgical object to which the tracker is coupled). The tracking element configuration may specify characteristics such as the emission wavelength / frequency, emission power, intensity, or other operating parameters of the tracking elements. The tracker identification data may also include a serial number of a specific tracker or the geometry of the tracker body. The locator 44 may acquire the tracker identification data by initializing the tracker or by tracking the tracker. In other cases, tracker identification data may be obtained from other sources, such as navigation system software or memory, or in response to the user entering the type of tracker into the software.

[0079] In some cases, at step 206, the controller may further acquire surgical object data. This data specifies any information that can be used to identify the surgical object being tracked by the tracker. Surgical object data may include an identifier or type of the surgical object being tracked. For example, an identifier may specify that the surgical object being tracked is the distal femur, the link (L2) of manipulator 14, or the pointer P tool, etc. Surgical object data may include sensor data. For example, if the surgical object is a tool / robot that includes sensors, sensor measurements may be included in the surgical object data. Such sensors may include any sensors suitable for the surgical object. For example, when pointer P is equipped with a force sensor, the sensor data may include force measurements. When the surgical object being tracked is manipulator 14, the sensor data may include kinematic data from the encoder, force / torque measurements from force / torque sensors attached to the robot arm, etc. Additionally, surgical object data may include posture / velocity / acceleration data associated with the surgical object, which may be obtained from sensor data, positioner 44, or any other source such as an external sensing system.

[0080] At point 208, optionally, the tracking data (as well as tracker identification data and surgical subject data) can be fed into at least one Long Short-Term Memory (LSTM) network. The LSTM can be part of a machine learning model (MLM) or separate from it. One LSTM can exist for all input data, or multiple LSTMs can exist for different types of data or subsets of data. One or more LSTMs can be implemented using recurrent neural networks (RNNs) and configured to receive input data over time. The LSTM modifies the weights of the RNN to learn long-term dependencies between the input data and selectively filters the input data. The LSTM then feeds the filtered input data to the neural network (NN) of the machine learning model (MLM).

[0081] The neural network (NN) of a machine learning model (MLM) is an artificial neural network that includes an input layer (IL), hidden layers (HL), and an output layer (OL). The input layer (IL) is configured to receive input data (i.e., tracking data, and optional identification data and surgical subject data). The output layer (OL) is configured to label tracking errors.

[0082] Neural networks (NNs) can be any suitable type of neural network architecture, including but not limited to: deep learning neural networks, deep reinforcement learning neural networks, recurrent neural networks (RNNs), multilayer perceptrons (MLPs), feedforward neural networks, generative adversarial networks (GANs), convolutional neural networks (CNNs), etc.

[0083] Input layer (IL) nodes can receive input data directly from the corresponding data source (202, 204, 206), or they can receive filtered input data from one or more LSTMs. The input layer (IL) can have any suitable number of nodes. A single input node can receive all input data, or nodes can be provided for each type of input data. In another example, input nodes can exist for various subsets of each data type. For example, for tracking data, there could be one input node for position data, another for velocity data, and yet another for acceleration data.

[0084] Hidden layers (HLs) are networked between the input and output layers and are configured to transform the input data by modifying weights and biases and implementing activation functions. Hidden layers (HLs) enable deep learning to learn complex relationships between input data. Hidden layers (HLs) transform input data into a data representation suitable for the output layer. Neural networks (NNs) can include any number of hidden layers (HLs).

[0085] At the output layer (OL) of the neural network (NN), the machine learning model (MLM) can identify, predict, classify, or represent the tracking error (shown at 212). The machine learning model (MLM) can produce a variety of results depending on its configuration or training.

[0086] In one example, a machine learning model (MLM) quantifies tracking inaccuracies. The MLM generates a difference (discrepancy) between the tracking data and baseline real data. The MLM can be pre-trained based on baseline real data derived from laser tracking from a tracker or other tracker. Such training data can be derived from a training process where an array of spherical mounted reflectors (SMRs), which are pre-calibrated precision optical components, is attached to a robotic arm or other manually movable device. The laser tracker system then fires a laser beam at the SMR each time it is automatically / manually moved to various poses. The SMR reflects the laser beam back to the laser tracker system, and distances / angles are measured at each specific SMR pose to establish baseline real data. Simultaneously, a positioner 44 of a certain type is located in the workspace and measures the position of a measuring element (e.g., a reference point in the case of an optical system), which has been calibrated relative to the SMR pose using a CMM or similar method. The tracking data from the positioner 44 is then compared with the laser tracking data to determine the differences between the tracking data and the baseline real data for several poses in the positioner's workspace. Machine learning models (MLMs) can use any suitable format to characterize the difference between the tracking data and the baseline real data, such as using the difference relative to each of the six degrees of freedom of the tracker pose (i.e., the three position DOFs and the three orientation DOFs).

[0087] In another example, a machine learning model (MLM) can estimate the impact of tracking errors on the surgical object to which the tracker is attached. For instance, the MLM can estimate the impact on the control of manipulator 14 or other tools attached to the tracker to provide dynamic error monitoring, which can provide, for example, a risk assessment scheme during a robotic resection procedure. Examples of such dynamic error monitoring will be described below.

[0088] Machine learning models (MLMs) can also intelligently and selectively assess whether tracking errors should be ignored by establishing error thresholds / ranges / metrics that dynamically change based on the input data (tracking data, identification data, surgical object data) and / or the identified errors. The error threshold / range / metric can change for each instance of the tracking data. The MLM can identify errors in response to them exceeding the dynamically changing error threshold / range / metric. In other cases, the error threshold / range / metric can be a user-defined preference. This threshold / range / metric can be adjusted during the procedure. Different error thresholds / ranges / metrics may exist based on the expected accuracy for each step of the surgical procedure. Threshold selection can be context-dependent and adjusted based on the stage of the procedure and the expected accuracy. In some cases, the error threshold / range / metric can be automatically customized based on the maximum expected error for a specific surgical step or task (such as performing a specific resection cut). Removing large areas of bone may have a high threshold, making errors more acceptable. Precise cuts near the planned boundaries may have a lower threshold for acceptable errors. For example, a machine learning model (MLM) can dynamically change the error threshold / range / metric based on the distance to the critical boundary. In another example, the expected cutting error at the TCP can be mapped to one or more active constraint zones and reactive constraint zones (e.g., for TKA plane cutting). In the active constraint zone, system 10 should strictly follow the surgical plan by keeping the tool precisely held on a specific cutting plane. Because high accuracy is required in the active constraint zone, the machine learning model (MLM) can set a lower and fixed error limit to prevent inter-cutting errors. For errors mapped to the reactive constraint zone, the machine learning model (MLM) can set a higher threshold because it does not affect accuracy before penetrating the boundary. In this case, the threshold can be set lower to avoid unintended penetration. The error threshold / range / metric can be varied according to any suitable mathematical function, such as a linear function (e.g., a spring constant) or a nonlinear function. Examples of active and reactive constraint schemes that can be utilized in conjunction with the techniques described herein may be similar to those described in U.S. Provisional Patent Application No. 63 / 737,587, filed December 20, 2024, entitled “Systems And Methods Employing Machine Learning Models To Improve Surgical Robotic Performance,” the entire contents of which are hereby incorporated by reference.

[0089] Machine learning models (MLMs) can also predict the sources of error. These sources can be unintended movement of the surgical object or of the locator 44. Unintended movement may be caused by accidental collisions or movement of the locator 44 or the surgical object. A primary source of error may be a change in the tracker's orientation or movement of the locator 44, which in turn may lead to unintended orientation changes in the tracker used in the application. These orientation changes can be normal or unintended, both of which affect tracking accuracy. Any orientation error introduced at the locator will amplify the TCP error as it propagates along the kinetic chain. Specifically, rotational errors will induce additional translational errors at the TCP due to the lever effect, where small angular deviations will result in large positional shifts over distance. The magnitude of this translational error depends on the tracker's position within the kinetic chain: the farther the tracker is from both the TCP and the anatomical structure tracker, the more significant the impact on positional accuracy. Therefore, inaccuracies in tracker orientation directly translate into increased cutting errors, especially when the tracker is mounted at a point far from the resection area. In other examples, the source of error may be incorrect calibration data for the locator 44 or the tracker. In other examples, the source of error might be line-of-sight occlusion between locator 44 and the tracker. Such occlusion could originate from an object (such as a human) located between locator 44 and the tracker, or it could be due to debris on the tracker's optical tracking elements. The source could also be a malfunction of a component characteristic of locator 44 or the tracker. For example, one of the tracking elements might not be emitting a signal, or the tracker battery might be low, failing to provide sufficient power to the LED tracking elements. In another example, the source of error might be that the tracker is outside the range or field of view of locator 44. By learning the complex relationships between input data (tracking data, identification data, surgical subject data), machine learning models (MLMs) can accurately predict the source of error, eliminating potentially time-consuming guesswork or troubleshooting in the operating room.

[0090] At point 214, the controller can generate feedback to address tracking errors. Depending on the situation, this feedback can be provided in many ways.

[0091] In one example, the controller and / or machine learning model (MLM) can provide feedback to offset or eliminate errors by selectively adjusting, permuting, or filtering tracking data. This compensation may include adjusting, permuting, or filtering any of the six degrees of freedom values ​​of the tracking data that identify the tracker's pose. Such compensation schemes can be useful during robot or anatomical registration processes, which will be described below. This selective adjustment, permutation, or filtering of tracking data can be done almost instantaneously (e.g., immediately during tracking) or in batches or at discrete times (e.g., after tracking or after the robot or anatomical registration process is complete).

[0092] In another example, feedback may include a notification or warning to inform the user of an error. The warning or notification may be provided in any suitable manner, including auditory, visual, tactile, or a combination thereof. In one example, the notification or warning is provided on the user interface (UI) or display 38 of the tracking system 32, or as a computer-generated extended reality element on the display 90 of the HMD. In another example, the notification or warning is provided on the surgical object being tracked. For example, a surgical instrument or manipulator 14 being tracked and manipulated by the user may vibrate or provide an auditory / visual warning. Alternatively, the tracker itself may provide a warning, for example, via an error indicator light or display on the tracker. The notification or warning may inform the user of the error as well as more advanced information determined by a machine learning model (MLM). For example, a predicted source of the error or a suggestion of corrective action to address the source of the error may be presented. The notification may display “Warning – anatomy tracker error detected from leg movement: Stabilize femur using leg holder”. If errors are selectively ignored, users can be informed of this to provide transparency regarding the integrity of the tracking data. The degree of inaccuracy of the error can also be provided to the user.

[0093] Other information that can be displayed may include a comparison between the raw tracking data and the tracking data modified by machine learning. For example, if the machine learning model (MLM) selectively adjusts, permutates, or filters the tracking data, the user interface (UI) or display 38, 90 may provide output that allows the user to understand the differences between the raw tracking data and the tracking data filtered by machine learning. Examples of such output will be described below in the context of robot or anatomical registration processes.

[0094] The controller and / or machine learning model (MLM) can also provide feedback by enabling some type of control over the surgical object being tracked. For example, if manipulator 14 or a surgical tool is being tracked, its operational functions may be constrained, reduced, or aborted in response to an identified error. If the surgical tool is a drill, tap, or screwdriver, the tool may be prevented from further rotational actuation in response to an error. Similarly, manipulator 14 can be controlled in various ways in response to identified errors, such as by adjusting the command posture of the TCP, deviating from the tool path, following a new tool path, changing the tool path direction, reducing / stopping its feed rate (tool path speed), reducing / stopping the cutting speed of end effector 22 (e.g., saw, cutting drill, etc.), implementing new or modified tactile constraints or virtual boundaries on end effector 22, pulling end effector 22 away from the surgical site (to a safe position), etc.

[0095] In another example, the tracker itself can be controlled to address the error. For example, the operation of the tracker or tracking element can be modified by changing the signal strength, frequency, channel, power, modulation, firing order, tracking mode, etc. If the tracker is attached to a movable device (such as manipulator 14 or an active adjustable arm), manipulator 14 or the adjustable arm can be controlled to change its orientation, thereby changing the orientation of the tracker accordingly to address the error.

[0096] In other implementations, the positioner 44 can be controlled to compensate for errors. For example, the orientation of the positioner 44 can be adjusted. If the positioner 44 is attached to a movable device (such as manipulator 14 or an active adjustable arm), the manipulator 14 or the adjustable arm can be controlled to change its orientation to correspondingly change the orientation of the positioner 44 to compensate for errors. Alternatively, the sensing elements of the positioner 44 can be replaced to compensate for errors. For example, the exposure time of the sensing elements of the positioner 44 can be increased, or the region of interest covering the tracker in its field of view can be dynamically “magnified.” The positioner 44 can also selectively increase the amount of sensing elements used to compensate for errors and increase the tracking frequency or modify image compression.

[0097] Additionally, the anatomical structure (A) can be manipulated to correct for errors. For example, the anatomical structure (A) can be adjusted according to guidance provided on displays 38, 90. This guidance can indicate the current position of the anatomical structure (A) based on tracking data, and also indicate the desired position of the anatomical structure (A). Displays 38, 90 can present graphics or templates to assist the user in moving the anatomical structure (A) to the desired position. An example system that can provide such guidance may be similar to the system described in U.S. Patent No. 11,166,781 entitled “System, Method and Software Program For Aiding In Positioning Of A Camera Relative To A Tracker In A Surgical Environment,” the entire contents of which are hereby incorporated by reference. In another example, a robotic manipulator can be attached to the anatomical structure (A). Positioner 44 can detect the pose of tracker 56 attached to the anatomical structure (A) and generate tracking data. The controller inputs the tracking data into a machine learning model (MLM) configured to identify errors associated with tracker 56. Based on this error, the controller can control the robot arm to adjust the posture of the anatomical structure (A) to resolve the error.

[0098] B. Dynamic error monitoring

[0099] Although a method 200 for tracking error identification and compensation has been described, this section will now describe an example use case of the technique, namely dynamic error monitoring, which refers to... Figure 4A , Figure 4B , Figure 4C Methods 200 and 300 are provided.

[0100] For dynamic error monitoring, manipulator tracker 52 is coupled to a component of robot manipulator 14, and anatomical structure tracker 56 is coupled to an anatomical structure (A). In the example shown, manipulator tracker 52 is coupled to the base 16 of manipulator 14 via an adjustable support arm, and anatomical structure tracker 56 is coupled to the patient's tibia. However, as described above, manipulator tracker 52 may be coupled to any component of manipulator 14, such as link 18, end effector 22, or a coupler between the robot arm and end effector 22. Similarly, depending on the nature of the program, anatomical structure tracker 56 may be coupled to any corresponding anatomical structure.

[0101] One aspect of dynamic error monitoring is utilizing the techniques described above to predict dynamic errors during surgical procedures, such as robotic joint replacement, caused by anatomical motion and / or the movement of the locator 44. This monitoring aspect can be used as a surveillance technique to protect the patient from unwanted robotic movements caused by performance degradation of the locator 44, as dynamic anatomical motion can lead to considerable orientation changes in the anatomical tracker 56. When projected onto another coordinate system, orientation tracking errors introduce significant additional translational errors on top of existing errors. By identifying the tracking errors, the controller can estimate or predict TCP inaccuracies or cutting errors at the end effector 22. Similarly, movement of the locator 44 affects the accuracy of both the manipulator tracker 52 and the anatomical tracker 56.

[0102] To achieve this dynamic error monitoring technology, such as Figure 4B As shown, the locator 44 detects the posture of the manipulator tracker 52 and the anatomical structure tracker 56, and generates tracking data (202a, 202b) associated with each tracker 52, 56, which may include posture and velocity / acceleration. As described above, tracker identification data (204) for each tracker 52, 56 can be acquired. Alternatively, surgical object data (206) (in this case, kinematic data from the manipulator 14) can be acquired using a joint encoder. Using forward kinematic calculations, the tracking system 32 can determine the TCP relationship between the base 16 of the manipulator 14 and the end effector 22. Similarly, using the tracking data, the following transformations can be calculated: (1) between the base 16 of the manipulator and the manipulator tracker 52; (2) between the manipulator tracker 52 and the locator 44; (3) between the locator 44 and the anatomical structure tracker 56; and (4) between the anatomical structure tracker 56 and the target anatomical structure (A). Based on these different transformations, the relationship between the locator 44 and the base 16 and / or the relationship between the TCP and the target anatomical structure (A) can be determined.

[0103] The controller obtains real-time tracking data from the locator 44, along with optional kinematic data and tracker identification data, and feeds the data into a machine learning model (MLM). The MLM utilizes this information to define an interactive pre-state formed by the anatomical state and the end effector state. The MLM is configured to identify errors (212) related to the detected pose of one or both of the manipulator tracker and the anatomical tracker 56. The identified errors can be actual / predicted / estimated cutting errors at the TCP.

[0104] Machine learning models (MLMs) intelligently and selectively evaluate whether tracking errors should be addressed or ignored. Specifically, conventional tracking systems typically compensate for tracking errors by halting system operation when the error exceeds a default error threshold, implementing this approach indiscriminately for any tracker and any situation. In contrast, MLMs offer a customized approach to handling tracker errors, taking into account several important factors such as the type of tracker, the type or context of tracker movement, the source of the error, or the context of the surgical procedure. MLMs can selectively assess whether tracking errors are acceptable given the circumstances or situation in which the error occurs. Therefore, using this approach, tracking system 32 is less likely to label tracking errors, thus minimizing interference with the surgical procedure. To achieve this, MLMs can establish error thresholds / ranges / metrics that dynamically vary based on the input data (tracking data, identification data, surgical object data) and / or the identified errors. The error thresholds / ranges / metrics can vary for each instance of the tracking data. Machine learning models (MLMs) can identify errors in response to errors exceeding dynamically changing error thresholds / ranges / metrics.

[0105] To further illustrate, a machine learning model (MLM) can estimate the localization error for a given anatomical structure and robot end effector state by establishing a real-time risk assessment factor by projecting the localization error onto the tool center point (TCP) measured in the anatomical coordinate system of the manipulator 14. For example, in Figure 4A The diagram illustrates measured and actual tracker measurements relative to locator 44, where the measured tracker measurements indicate those including tracking errors. In the illustrated example, tracking errors occur at anatomical structure tracker 56 and manipulator tracker 52 due to unintended movement of locator 44. Using the described transformation, the effects of tracking errors can be projected onto TCP, and risk assessment can be performed based on this projected error. Machine learning models (MLMs) can establish real-time risk assessment factors by quantifying dynamic errors, such as those caused by anatomical structure motion or locator 44 movement during robot-assisted joint replacement surgery. Utilizing these techniques, MLMs can make various estimates of the impact of errors on the system. For example, MLMs can: estimate the impact of dynamic anatomical motion on anatomical structure localization; estimate the impact of locator movement on robot tracking performance; estimate the impact of locator movement on anatomical structure localization; estimate the impact of robot arm / robot cart movement on robot tracking performance; or predict dynamic errors caused by locator movement, robot cart / robot arm movement, and dynamic anatomical motion.

[0106] The machine learning model (MLM) can selectively determine whether a real-time risk assessment factor exceeds a dynamic threshold / range / metric. If the real-time risk assessment factor exceeds the dynamic threshold / range / metric, the controller generates feedback (214) to address the error, such as by generating a warning or notification using the techniques described above. Such warnings can be provided to avoid cutting errors, reduce surgical risks, and protect patients from unwanted cuts caused by locator performance degradation due to tracker detection inaccuracies. The controller can also present the expected TCP error in real time on either of displays 38, 90. If the TCP error exceeds the dynamic threshold / range / metric, the source of the error can also be presented, such as: locator movement, anatomical structure movement, robot cart movement, or excessive movement of the robot arm with significant orientation changes. Such notifications or warnings can provide valuable insights into preventative measures that the surgeon can take to minimize the impact of positioning errors on TCP accuracy. Other forms of feedback are envisioned, such as those described above, such as providing commands to modify or abort the operation of robot manipulator 14.

[0107] If the real-time risk assessment factor is less than the dynamic threshold / range / metric, the controller may selectively ignore the error. Therefore, the controller may allow movement of the locator 44 and / or the target anatomy as long as the expected dynamic error does not exceed the accuracy requirements.

[0108] Figure 4C Method 300 provides another implementation of dynamic error monitoring technology, and can be referenced... Figure 4AThe illustrated setup and transformations are used to further understand the method. Starting with manipulator (robot) tracker 52, at step 302, the controller acquires manipulator tracker data. As described above, this data may include tracking data from locator 44, including the attitude and velocity of manipulator tracker 52, as well as tracker identification or configuration / type data. At 304, the controller may use the raw tracking data from locator 44 to calculate the attitude (or distance) and orientation of manipulator tracker 52 relative to the coordinate system (CS) of locator 44. At 306, using the calculated attitude (or distance) and orientation of manipulator tracker 52 (determined at 304), and optionally the velocity and tracker identification / type data of manipulator tracker 52 (from 302), the controller then calculates the expected error of manipulator tracker 52. In one implementation, using any of the techniques described above, the controller implements a machine learning model (MLM) to calculate the expected error. The machine learning model (MLM) may be a deep neural network or any other suitable model type. At 308, the controller uses the expected error of the manipulator tracker 52 to calculate the expected true transformation of the manipulator tracker 52. The expected true transformation represents the position that the manipulator tracker 52 would expect to be located in the coordinate system of the locator 44 if there were no error.

[0109] Regarding the anatomical tracker 56, at 312, the controller acquires anatomical tracker data. As described above, this data may include tracking data from the locator 44, including the pose and velocity of the anatomical tracker 56, as well as tracker identity or configuration / type data. At 314, the controller uses the raw tracking data from the locator 44 to calculate the pose (or distance) and orientation of the anatomical tracker 56 relative to the coordinate system (CS) of the locator 44. At 316, using the calculated pose (or distance) and orientation of the anatomical tracker 56 (determined at 314), and optionally the velocity and tracker identity / type data of the anatomical tracker 56 (from 312), the controller then calculates the expected error of the anatomical tracker 56. In one implementation, using any of the techniques described above, the controller implements a machine learning model (MLM) to calculate the expected error. At 318, the controller uses the expected error of the anatomical tracker 56 to calculate the expected true transformation of the anatomical tracker 56. The expected true transformation indicates that, if there are no errors, the anatomical structure tracker 56 is expected to locate its position in the coordinate system of the locator 44.

[0110] At 310, the controller obtains a robot registration transformation indicating the relationship between the manipulator tracker 52 and the manipulator's base 16. This can be obtained using any suitable robot registration process, which will be described in detail below. At 312, the controller further obtains kinematic data from the manipulator 14 to determine the relationship between the TCP and the base 16. Using the calculated expected true transformation (308, 318), tracking data (302, 312), robot registration transformation (310), and kinematic transformation (320), the controller calculates the measured TCP value and the true TCP value (at 322) relative to the coordinate system of the anatomical structure. Thus, the tracking error can be projected onto the robot's TCP. The controller can then calculate the error between the measured TCP value and the true TCP value at 324, indicating the error caused by the dynamic motion of either tracker 52, 56. Optionally, filtering or smoothing of the calculated error can be implemented at 326 to avoid overreacting feedback. At 330, the controller determines whether the calculated error exceeds a first threshold. If no, the controller will conclude at 328 that there is no tracking error. If yes, the controller may compare the computer error to a second threshold at 332. If the calculated error is below the second threshold, the controller may provide feedback at 334 via notification or warning to inform the user of the error using any of the techniques described in this document. This feedback may indicate the existence of a potential error that has not yet reached the threshold error level, giving the surgeon an opportunity to take corrective action before identifying the threshold error. If the calculated error is above the second threshold, the controller may take more aggressive action at 336, such as by providing a threshold error indicator and / or modifying control of manipulator 14 (e.g., terminating manipulator 14), or by providing any other feedback described above.

[0111] Dynamic error monitoring techniques offer several advantages. These described techniques replace conventional hard-coded constraints, which are insufficient for complex error detection. Dynamic error monitoring techniques can generalize to a wider range of operating conditions and scenarios and are well-suited for simultaneously and accurately evaluating inputs from multiple disparate data sources, such as locators, anatomical structures, and manipulator rotation and translation. These solutions also offer the added benefit of tolerating noise generated by sensors and the environment.

[0112] C. Robot registration

[0113] This section describes another example of using the error identification techniques described above to enhance the robot registration process. Robot registration is performed prior to surgical procedures and involves aligning / registering the coordinate system of manipulator 14 with that of locator 44. Accurate registration of manipulator 14 is essential to ensure accurate tracking of manipulator 14 relative to the patient, thereby enabling accurate execution of the procedure. However, robot registration is particularly sensitive to tracking errors due to the susceptibility of locator 44 to orientation changes during data collection. Such errors can occur when the tracker involved in the registration process does not consistently face locator 44 or undergoes unexpected movement (including nonlinear rotation).

[0114] refer to Figures 5A to 5D Depending on the specific configuration of the manipulator trackers located on manipulator 14, the robot registration process and results can vary from system to system. As will be described below, these manipulator trackers can be located on various components of manipulator 14, such as, but not limited to, base 16, link 18, end effector 22, etc.

[0115] exist Figure 5A In this design, the manipulator tracker is a base tracker (BT) located on an adjustable arm coupled to a base 16 of the manipulator 14. Since the arm can be adjusted in various user-configurable poses, a robot registration process must be used to determine the relationship between the base tracker (BT) and the base 16. The user then attaches a registration tracker (RT) to the end effector 22 according to a predetermined relationship. An example of a registration tracker (RT) may resemble the end effector tracker (47) of U.S. Patent No. 8,010,180 entitled “Haptic Guidance System And Method” (see [link to relevant documentation]). Figure 6 B), the entire contents of which are hereby incorporated by reference. The registration tracker (RT) may include coupling devices (such as grippers) attached to a tool extending from the end effector 22. The registration tracker (RT) may include registration surfaces of the TCP adjacent to the tool to define predetermined relationships. The registration tracker (RT) is used only during the robot registration process and is removed thereafter. Once the registration tracker (RT) is attached, the system 10 instructs the user to perform the robot registration process by moving the TCP for registration motion, such as by moving the TCP along the edges and vertices of a cubic tactile object that constrains the TCP to the vertices and edges. During this registration motion, both localization data and kinematic data of the TCP are acquired simultaneously. Specifically, localization data is acquired from the registration tracker (RT), and kinematic data about the TCP is acquired from forward kinematic calculations. These data points are paired and compared until an accuracy threshold is met.

[0116] Figure 5B The example involves using a link tracker (LT) associated with one of the links 18 of manipulator 14, instead of a base tracker (BT). The link tracker (LT) can be in a fixed relationship relative to manipulator 14, or it can be in a user-defined or custom position relative to manipulator 14. For example, the link tracker (LT) can be attached to any position on one or more links 18 without prior knowledge of the reference transformation of the nearest joint relative to manipulator 14. During the robot registration process, this missing transformation is calculated, which can later be used in conjunction with the kinematic use of manipulator 14 to determine the position of the link tracker (LT) relative to robot base 16. Even when the link tracker (LT) is in a fixed relationship relative to manipulator 14, there are factors that can affect that relationship, such as the mechanical compliance of the links or joints of the robot arm. Therefore, the robot registration process should still be performed to accurately determine the relationship between the link tracker (LT) and base 16. Regardless of whether the link tracker (LT) is fixed or custom-defined, the robot registration process similarly includes: attaching the registration tracker (RT) to the TCP, performing registration motion on the manipulator 14, and acquiring kinematic / localization data pairs until sufficient accuracy is achieved to define the relationship between the link tracker (LT) and the base 16. Several intermediate transformations (such as between the link tracker (LT) and the link 18 of the manipulator 14) can be used to establish the relationship with the base 16.

[0117] exist Figure 5C In the example, manipulator 14 includes an end effector tracker (EET), i.e., no base tracker or link tracker. The end effector tracker (EET) may be incorporated into the body of end effector 22, may be attached to end effector 22, or may be located on a coupler between end effector 22 and the distal flange of the robot arm. Such a coupler may be similar to the coupler described in U.S. Patent Application Publication No. 20220039898 entitled “Robotic Surgical System Including A Coupler For Connecting A Tool To A Manipulator And Methods Of Using The Coupler,” the entire contents of each of these references are hereby incorporated by reference. The robot registration process similarly includes attaching a registration tracker (RT) to the TCP, moving manipulator 14 to perform registration movements, and acquiring kinematic / positioning data pairs until sufficient accuracy is achieved to define the relationship between the end effector tracker (EET) and the base 16. Several intermediate transformations (such as between the end effector tracker (EET) and the distal flange of the manipulator 14) can be used to establish a relationship with the base 16.

[0118] Figure 5D Examples are similar to Figure 5C The example simply adds a link tracker (LT). Here, the end effector tracker (EET) and the link tracker (LT) are used together. The robot registration process similarly includes: attaching the registration tracker (RT) to the TCP, moving the manipulator 14 to perform registration motion, and acquiring kinematic / positioning data pairs until sufficient accuracy is achieved to define the following relationships: link tracker (LT) to base 16; end effector tracker (EET) to base 16; and / or a combination of link tracker (LT) and end effector tracker (EET) relative to base 16. Several intermediate transformations (such as between the link tracker (LT) and link 18 of manipulator 14 or between the end effector tracker (EET) and the distal flange of manipulator 14) can be used to establish the relationship with base 16.

[0119] While various forms of manipulator tracking for robot registration have been described, we will now describe how the machine learning techniques described above can enhance robot registration for any given variant. The proposed compensation scheme during the robot registration process significantly improves the accuracy of the robot registration transformation, especially when the manipulator tracker moves dynamically. The robot registration transformation obtained from the proposed compensation scheme improves the robustness of the system in taking camera motion into account without affecting cutting accuracy.

[0120] refer to Figure 6 The method includes obtaining tracking data from a registration tracker (RT) and a manipulator tracker, which is acquired by the locator 44 during the robot registration process. Again, the manipulator tracker may include a pedestal tracker (BT), a link tracker (LT), an end effector tracker (ET), or any combination thereof. Supplementary data regarding tracker identification is available. The controller also acquires information about the robot's kinematics. The tracking data (and optional supplementary data) is fed into a machine learning model (MLM). The machine learning model (MLM) identifies errors associated with the detected pose of one or both of the manipulator tracker and the registration tracker and can generate feedback to address these errors.

[0121] In one example implementation, such as Figure 6As shown, tracking data can optionally be provided to a coarse filtering layer before being fed as input to a machine learning model (MLM). The coarse filtering layer filters the tracking data by ignoring tracking data that fails certain checks. One check is a visibility check, where the controller filters out tracking data that fails to include all visible markers from both the manipulator tracker and the registration tracker (RT). The second check is a standard deviation (STD) check, which is applied in static data collection scenarios. Here, multiple data points are collected for a single pose, and the controller calculates its standard deviation. If the standard deviation exceeds a predetermined threshold, indicating that the pose is not static, the data is discarded. The standard deviation check can be used to confirm that the locator 44 and the manipulator 14 have remained stationary. The third check is a quality check, where the controller evaluates the quality of the tracking data to ensure that it meets a predefined threshold before being used in the registration process. Only data that exceeds this quality threshold is retained for further use.

[0122] The filtered tracking data is then fed into a machine learning model (MLM) to improve the accuracy of robot registration. In this example, the MLM provides a higher-level filtering layer to the tracking data to reduce, eliminate, or compensate for tracking errors, for example, using any of the techniques described above. The MLM can identify or predict expected errors for a particular tracker configuration used in the robot registration process and compensate for those expected errors before using them in the robot registration process. The MLM can modify the tracking data in several ways to reduce / eliminate expected errors, such as replacing the tracking data with more accurate tracking, changing the values ​​of the tracking data, or omitting inaccurate tracking data. The MLM can filter the tracking data continuously or at discrete times. For example, the MLM can filter the tracking data in real time as it is received during the robot registration process. Alternatively, the MLM can filter the tracking data after all tracking data has been acquired during the robot registration process.

[0123] To identify robot registration transformations, pairwise matching of tracking / kinematic data points is performed. In this case, the controller may obtain more than one robot registration transformation. One type of transformation may be based on pair-point matching according to data filtered by a machine learning model. This transformation can be considered a machine learning-based robot registration transformation. A second type of transformation may be based on pair-point matching according to data directly from the output of the filtering layer (i.e., without any filtering by the machine learning model). This transformation can be considered a standard robot registration transformation. The controller may present the results of the standard and machine learning-based robot registration transformations to the user on displays 38, 90 in any suitable format. The controller may present statistical information about standard registration and machine learning-based registration, or a comparison between standard registration and machine learning-based registration. For example, displays 38, 90 may provide accuracy statistics, such as an overall accuracy score for each registration process. Based on the information provided, the controller may prompt the user to select one of the transformations. For example, if the user finds that machine learning-based registration provides more accurate results, the user may select that result.

[0124] In another example, a verification process can be performed after robot registration. Using the results of this verification process, the controller can automatically determine which registration method (standard or ML-enabled) to use. In one example, the verification process may involve tracking a registration tracker (RT) as manipulator 14 is moved to one or more predetermined locations (e.g., aiming an end effector at a floor / ceiling). In another example, a pointer P can be used to touch a notch or checkpoint relative to manipulator 14 in a known location (such as on a surgical instrument). In yet another example, both locator 44 and manipulator 14 can be moved, and statistical measurements can be presented to the surgeon to help select a registration method that is more robust to locator movement. Locator movement can be performed manually or automatically, for example, if the locator is mounted to a separate manipulator.

[0125] D. Anatomical registration

[0126] This section describes yet another example of using error labeling techniques to enhance the anatomical registration process. Refer to Figure 7 and... Figure 8Anatomical registration is performed prior to surgical procedure and involves aligning / registering the coordinate system of a virtual anatomical model (AM), which includes aspects of surgical planning, with the target anatomical structure (A) being tracked and thus with the coordinate system of the locator 44. The anatomical structure (A) being tracked can be any anatomical structure, such as any bone, organ, etc. In the example shown, the anatomical structure being tracked is a bone being tracked by the anatomical structure tracker 56, specifically the pelvic bone. The virtual model (VM) includes aspects of surgical planning, such as the planned location of implants, the planned area / volume of the anatomical structure to be cut, virtual constraint boundaries, etc. The virtual model (VM) is derived from patient imaging data, such as CT data, which can be segmented to form the virtual model (VM). Accurate registration of the anatomical structure (A) is necessary to ensure that the manipulator 14 can be accurately tracked relative to the patient, thereby enabling accurate execution of the procedure. However, anatomical registration is susceptible to the tracking errors described above, because the tracker involved in anatomical registration may not be fully aligned with the locator 44, or may experience unexpected movement or a wide range of orientation changes, such as nonlinear rotation.

[0127] refer to Figures 7A to 7C This provides an example of performing an anatomical registration process using a pointer or probe tool (P).

[0128] exist Figure 7A In this configuration, the anatomical structure tracker 56 is directly coupled to the anatomical structure A. Since the anatomical structure tracker 56 is manually attached by the user, an anatomical registration process must be used to determine the relationship between the anatomical structure tracker 56 and the anatomical structure (A). Therefore, the user utilizes a pointer / probe tool (P) equipped with a tool tracker (PT), in this case a pointer tracker, which has a known and fixed relationship relative to the tip of the probe tool (P). The position of the tip of the probe tool (P) can be accurately determined in the coordinate system of the locator 44. An example of the probe tool (P) and tool tracker (PT) may resemble the probe and tracker of U.S. Patent No. 8,010,180 entitled “Haptic Guidance System And Method” (see Figure 7), the entire contents of which are hereby incorporated by reference. Once the locator 44 is able to detect the tool tracker (PT), the system 10 commands the user via displays 38, 90 to perform the anatomical registration process by moving the tip of the probe tool (P) into contact with the target anatomical surface. Then, the target surface of the anatomical structure is digitized using a probe tool (P). Figure 7A In this context, the target surface is the acetabulum of the pelvic bone. However, depending on the procedure, the target anatomical surface can vary.

[0129] The probe tool (P) can be used in various workflows. For example, the tracking system 32 may require the user to touch the tip of the probe tool (P) to discrete, identified landmark points. Alternatively or concurrently, region-based registration techniques may be utilized, where the system allows the user to touch individual points within a specific region determined by the tracking system 32. In another example, the user may drag / sweep / grab the tip of the probe tool (P) across / over / through a target anatomical surface, thereby providing continuous acquisition of points or lines. A preliminary (coarse) anatomical registration procedure may exist, followed by a second (fine) anatomical registration procedure. Such point / region and initial / fine registration processes may be similar to the process described in U.S. Patent No. 10,485,450 entitled “Systems and methods for intra-operative pelvic registration,” the entire contents of which are hereby incorporated by reference.

[0130] The tracking system 32 may include an input device, such as a foot pedal or a button on the probe tool (P), to command the acquisition of any point when the user determines that the acquisition time is appropriate. In other examples, acquisition may be automatically triggered based on supplemental sensing. For example, when the probe tool (P) includes a force sensor or other tissue detection sensor, measurements from the sensor may indicate the confidence level of the tool tip contacting the appropriate tissue (e.g., cortical bone) to trigger automatic acquisition of the point.

[0131] During this registration process, positioning data is acquired from the tool tracker (PT). Based on the relationship between the tool tracker (PT) and the tool tip, the positions of the acquired points can be correlated with the coordinate system of the locator 44. Simultaneously, positioning data is acquired from the anatomical structure tracker 56, and the acquired points are correlated with the coordinate system of the anatomical structure tracker 56, and thus with the coordinate system of the locator 44. The digitized points can then be mapped to a virtual model (VM), such as... Figure 7B As shown. These tracking data points are paired and compared until an accuracy threshold is met. Then, the virtual model (VM) can be registered to the physical anatomical structure (A), such as... Figure 7C As shown.

[0132] Depending on the specific techniques and tracking methods used during registration, the anatomical registration process may differ from that shown. The error compensation techniques described herein can be used in conjunction with any other registration method. For example, instead of the probe tool (P), any other tracked instrument including a tool tip can be used to acquire points, such as a tool attached to manipulator 14, or a tool tip attached to a motorized or robotic handheld tool (such as a drill, screwdriver, tap, awl, saw, milling cutter, cutting drill, etc.). Additionally, “image-free” anatomical registration can be used when a virtual model (VM) of the anatomical structure is not available. Instead, a virtual surface of the anatomical structure (A) is derived intraoperatively from the point cloud acquired using the probe tool (P), and surgical planning can be made intraoperatively relative to the virtual surface. In some image-free registration techniques, the anatomical structure may not necessarily be tracked. In other examples, anatomical registration may involve using computer / machine vision tracking to derive the surface of the anatomical structure (e.g., using a depth map), where the device performing the computer / machine vision tracking can be tracked by locator 44. Such vision techniques can be implemented via an HMD or via locator 44 attached to a trolley. In other implementations, any visual tracking system can track the user's hand and fingers, and the user can acquire points by touching the target anatomical surface with their fingers. Alternatively, anatomical registration can be performed using an ultrasound imaging device tracked by locator 44. Anatomical registration can be performed using any suitable tracking modality.

[0133] While various forms of tracking for anatomical registration have been described, we will now describe how the machine learning techniques described above can enhance anatomical registration for any given variant. The proposed compensation scheme during the anatomical registration process significantly improves the accuracy of the anatomical registration transformation, especially when the tool tracker (PT) and / or the anatomical structure tracker 56 moves dynamically. The anatomical registration transformation obtained from the proposed compensation scheme improves the robustness of the system in taking camera motion into account without affecting cutting accuracy.

[0134] refer to Figure 8 The method includes acquiring tracking data from an anatomical structure tracker 56 and a tool tracker (PT), which is obtained by a locator 44 during the anatomical registration process. Supplementary data regarding tracker identification is available. When the probe tool (P) is equipped with a force sensor or tissue sensor, supplementary data from such sensors may also be acquired. The tracking data (and optional supplementary data) is fed into a machine learning model (MLM). The machine learning model (MLM) identifies errors related to the detected pose of one or both of the anatomical structure tracker 56 and the tool tracker (PT) and can generate feedback to address these errors.

[0135] In one example implementation, such as Figure 8As shown, tracking data can optionally be provided to a coarse filtering layer before being fed as input to a machine learning model (MLM). The coarse filtering layer filters the tracking data by ignoring tracking data that fails certain checks. One check is a visibility check, where the controller filters out tracking data that fails to include all visible markers from both the anatomical structure tracker and the registration tracker (RT). The second check is a standard deviation (STD) check, which is applied in static data collection scenarios. Here, multiple data points are collected for a single pose, and the controller calculates their standard deviation. If the standard deviation exceeds a predetermined threshold, indicating that the pose is not static, the data is discarded. The third check is a quality check, where the controller assesses the quality of the tracking data to ensure that it meets a predefined threshold before being used in the anatomical registration process. Only data exceeding this quality threshold is retained for further use.

[0136] Alternatively, a machine learning (ML) model or another machine learning model can participate in the coarse filtering layer by implementing tissue inspection, which is based on sensors that acquire measurements about the tissue being contacted using a probe tool (P). Specifically, a controller can acquire sensed measurements during the interaction between the tool (P) and the tissue and apply the sensed measurements to a machine learning model configured to characterize the type of tissue or distinguish between different types of tissue. If the identified tissue is soft tissue or cartilage, the model can discard or filter out such acquired points. Such techniques may be similar to those described in U.S. Provisional Patent Application No. 63 / 749,842, filed January 27, 2025, entitled “Robotic Surgical Systems And Methods Employing Machine Learning Models To Characterize Tool Interactions,” the entire contents of which are hereby incorporated by reference.

[0137] The filtered tracking data is then fed into a machine learning model (MLM) to improve the accuracy of anatomical registration. In this example, the MLM provides a higher-level filtering layer to the tracking data to reduce, eliminate, or compensate for tracking errors, for example, using any of the techniques described above. The MLM can identify or predict expected errors for a particular tracker configuration used in the anatomical registration process and compensate for those expected errors before using them in the anatomical registration process. The MLM can modify the tracking data in several ways to reduce / eliminate expected errors, such as replacing the tracking data with more accurate tracking, changing the values ​​of the tracking data, or omitting inaccurate tracking data. The MLM can filter the tracking data continuously or at discrete times. For example, the MLM can filter the tracking data in real time as it is received during the anatomical registration process. Alternatively, the MLM can filter the tracking data after all tracking data has been acquired during the anatomical registration process.

[0138] To identify anatomical registration transformations, tracking data from the anatomical structure tracker 56 and the tool tracker (PT) are matched with corresponding points on the virtual model (VM). In this case, the controller can obtain more than one anatomical registration transformation. One type of transformation can be based on point-to-point matching according to data filtered by a machine learning model (MLM). This transformation can be considered a machine learning-based anatomical registration transformation. A second type of transformation can be based on point-to-point matching according to data directly from the output of the filtering layer (i.e., without any filtering by the machine learning model). This transformation can be considered a standard anatomical registration transformation. As described above, point-to-point matching can be performed during the initial registration process and the fine registration process. Anatomical structure registration can be performed in two phases: an initial point-to-point matching phase, which is similar to the robot registration process; followed by a surface matching phase. During the surface matching phase, outliers may exist in the collected data. If the initial surface matching indicates the presence of such outliers, for example, detected by high residual errors or inconsistent point correspondences observed visually, the process can be repeated after applying an outlier removal method (filtering layer II). This iterative optimization helps ensure that only high-quality data contributes to anatomical registration, thereby improving the overall accuracy and robustness of the resulting transformation.

[0139] The controller can present the results of standard and machine learning-based anatomical registration transformations to the user on displays 38 and 90 in any suitable format. The controller can present statistical information about standard and machine learning-based registration, or comparisons between the two. For example, displays 38 and 90 can provide accuracy statistics, such as an overall accuracy score for each registration process. In one case, the controller can present the results of the anatomical registration process on displays 38 and 90 as a 3D graphical representation of a virtual model (VM) of the anatomical structure, with heatmaps applied to the surface of the virtual model. The heatmaps can be colored and can identify the anatomical surfaces or regions being registered. The heatmaps can be used to compare the accuracy between standard and machine learning-based registration. This allows surgeons to visually assess and confirm the improvements achieved by machine learning before final mapping, ensuring higher accuracy. Based on the information provided, the controller can prompt the user to select one of the transformations. For example, if the user finds that machine learning-based registration provides more accurate results, the user can select that result.

[0140] Alternatively, a verification process can be performed after anatomical registration. Using the results of this verification process, the controller can automatically determine which registration method (standard or ML-enabled) to use. In one example, the verification process might involve using a pointer (P) to touch a pit or checkpoint located at a known position relative to the anatomical structure (A).

[0141] E. Extended Reality

[0142] The error labeling techniques described above can also be used with the HMD in surgical settings for extended reality purposes. As described above, the HMD may include a tracker 98 for which a machine learning model (MLM) can be used to label tracking errors. In another example, the HMD may support a locator 44, which may be a camera 94 and / or a visual sensing system. The HMD locator 44 may detect other trackers for which a machine learning model (MLM) can be used to label tracking errors.

[0143] In another example, when locator 44 is mounted on the HMD, locator 44 can detect the pose of any object in the operating room and generate tracking data about that object. The object does not need to be attached to a tracker. For example, the object could be a surgeon's hand, another person's hand, the human body, surgical instruments, a target anatomical structure, manipulator 14, another HMD, etc. When the movement is a hand movement, the hand movement can be used for any purpose, such as providing posture control input to the HMD, manipulating or operating surgical instruments, performing point-and-click anatomical registration, etc. The controller can feed the tracking data into a machine learning model (MLM) that identifies tracking errors associated with the detected pose of the object and provides feedback using any of the techniques described above to address the tracking-related errors.

[0144] Several embodiments have been described in the foregoing description. However, the embodiments discussed herein are not intended to be exhaustive or to limit the invention to any particular form. The terminology used is intended to be descriptive rather than restrictive in nature. In view of the foregoing teachings, many modifications and variations are possible, and the invention may be practiced in ways other than those specifically described.

Claims

1. A surgical procedure tracking system, the surgical procedure tracking system comprising: A tracker coupled to a surgical object; A locator configured to detect the attitude of the tracker and generate tracking data; as well as A controller, coupled to the locator and configured to: The tracking data is obtained from the locator; The tracking data is input into a machine learning model, which is configured to identify errors related to the pose detected by the tracker; as well as Feedback is generated to address the error.

2. The surgical tracking system of claim 1, wherein the controller is further configured to: Obtain the identity data associated with the tracker; and The identity data is input into the machine learning model so that the machine learning model can identify the error.

3. The surgical tracking system as described in claim 2, wherein: The tracker includes a tracking element that can be detected by the locator; and The identification data includes the geometry of the tracking element.

4. The surgical tracking system of claim 1, wherein the tracking data includes the position of the tracker and the orientation of the tracker.

5. The surgical tracking system of claim 4, wherein the tracking data further comprises one or both of the following: the velocity of the tracker and the acceleration of the tracker.

6. The surgical tracking system of claim 1, wherein the machine learning model comprises a neural network.

7. The surgical tracking system of claim 6, wherein the neural network comprises an input layer, a hidden layer, and an output layer, wherein: The input layer is configured to receive the tracking data; and The output layer is configured to identify the error.

8. The surgical tracking system of claim 7, wherein the machine learning model comprises at least one Long Short-Term Memory (LSTM) network coupled to the input layer of the neural network, wherein the at least one LSTM comprises a Recurrent Neural Network (RNN) and is configured to: Receive the tracking data over time; The weights of the RNN are modified to learn long-term dependencies between the tracking data and the tracking data is selectively filtered. as well as The filtered tracking data is provided to the input layer of the neural network.

9. The surgical tracking system of claim 1, wherein the error is related to the inaccuracy of the posture detected by the tracker.

10. The surgical tracking system of claim 1, wherein: The machine learning model is pre-trained based on benchmark real data derived from laser tracking of the tracker or other trackers; and The machine learning model identifies the error based on a comparison between the tracking data and the benchmark real data.

11. The surgical tracking system of claim 1, wherein the machine learning model is configured as follows: Establish an error threshold or range that dynamically changes based on the tracking data; and The error is identified in response to the error exceeding the error threshold or range.

12. The surgical tracking system of claim 1, wherein the machine learning model is configured to estimate the impact of the error on the accuracy of tracking the surgical object.

13. The surgical tracking system of claim 1, wherein the machine learning model is further configured to predict the source of the error.

14. The surgical tracking system of claim 13, wherein the source of the error comprises one or more of the following: Undesirable movement of the surgical subject; Unintended movement of the locator; or Incorrect calibration data for the locator or the tracker.

15. The surgical tracking system of claim 1, wherein the controller provides feedback by being configured to provide notifications or warnings to inform the user of the error.

16. The surgical tracking system of claim 1, wherein the controller provides feedback to counteract the error by being configured to selectively adjust or replace the tracking data.

17. The surgical tracking system of claim 1, wherein: The tracker is an anatomical structure tracker coupled to the anatomical structure; A manipulator tracker is coupled to a robotic manipulator including a robotic arm configured to support a surgical instrument and move the surgical instrument relative to the anatomical structure. The locator is configured to detect the pose of the anatomical structure tracker and the manipulator tracker, and to generate tracking data associated with the anatomical structure tracker and the manipulator tracker; and The controller is configured to: The tracking data is obtained from the locator; The tracking data is input into the machine learning model to identify the error associated with the detected posture of one or both of the anatomical tracker and the manipulator tracker. as well as Feedback is generated to address the error.

18. The surgical tracking system according to any one of claims 1 to 16, wherein: The tracker is a manipulator tracker coupled to a robot manipulator; and The registration tracker is configured to be coupled to an end effector, which is attached to a robotic arm supported by a base of the robot manipulator; The locator is configured to detect the attitude of the manipulator tracker and the attitude of the registration tracker, and to generate the tracking data associated with the manipulator tracker and the registration tracker; and The controller is configured to: The tracking data is obtained from the locator; The tracking data is input into the machine learning model to identify the error associated with the detected pose of one or both of the manipulator tracker and the registration tracker; as well as Feedback is generated to address the error.

19. The surgical tracking system as claimed in any one of claims 1 to 16, wherein: The tracker is a tool tracker coupled to a surgical probe configured to interact with anatomical structures. An anatomical structure tracker, coupled to the anatomical structure; The locator is configured to detect the pose of the tool tracker and the pose of the anatomical structure tracker, and to generate the tracking data associated with the tool tracker and the anatomical structure tracker; and The controller is configured to: The tracking data is obtained from the locator; The tracking data is input into the machine learning model to identify the error associated with the detected pose of one or both of the tool tracker and the anatomy tracker; as well as Feedback is generated to address the error.

20. The surgical tracking system of any one of claims 1 to 16, wherein the tracker is coupled to one or more of the following surgical objects: Surgical instruments; Anatomical structure; Head-mounted display devices; Links in the robot arm of a robot manipulator; An adjustable arm coupled to the base of a robot manipulator; An end effector coupled to a robot manipulator; as well as A coupler located between the robotic arm of the manipulator and the end effector supported by the robotic arm.

Images