Surgical navigation system and method for displaying augmented reality image

The UWB and IMU-based surgical navigation system with AR technology addresses the challenge of aligning virtual and real-world images in surgeries, ensuring precise bone cuts and implant placements, enhancing surgical procedure accuracy and safety.

JP2024156960AActive Publication Date: 2024-11-06TECHMAH MEDICAL LLC
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2024133293
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2016-09-07
Filing Date
2024-08-08
Publication Date
2024-11-06
Estimated Expiration
2037-03-14

AI Technical Summary

Technical Problem

Current surgical navigation systems face challenges in accurately aligning virtual objects with real-world environments during surgical procedures, leading to potential misalignment of bone cuts and implant placement, which can negatively impact the outcome of the procedure.

Method used

A surgical navigation system utilizing ultra-wideband (UWB) and inertial measurement units (IMU) for precise tracking, combined with augmented reality (AR) technology, to create a dynamic orthopedic implant that is patient-specific and kinematically evaluated, allowing for real-time guidance and feedback during surgeries like arthroplasty.

Benefits of technology

The system provides highly accurate real-time tracking and guidance, ensuring precise bone cuts and implant placements, improving the success rate and safety of surgical procedures by aligning virtual and real-world images consistently.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024156960000001_ABST
    Figure 2024156960000001_ABST
Patent Text Reader

Abstract

To provide a surgical navigation system.SOLUTION: A surgical navigation system includes: a first tracker configured so as to be attached to an anatomical structure of a patient; a second tracker configured so as to be attached to surgical equipment including a tactile motor; a third tracker attached to at least one of a projector and a graphical display configured so as to display an augmented reality image, and attached to a first camera; and a processor connected communicably to the first tracker, the second tracker, and the third tracker. The processor is connected communicably to the surgical equipment, and generates a command to trigger the tactile motor in nearly real time using data output from the first tracker and the second tracker.SELECTED DRAWING: Figure 27
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 308,176, filed March 14, 2016, entitled "Ultra-wideband positioning for wireless ultrasound tracking and communication," and U.S. Provisional Patent Application No. 62 / 384,521, filed September 7, 2016, entitled "Ultra-wideband positioning for wireless ultrasound tracking and communication," the disclosures of each of which are incorporated herein by reference.

[0002] Technical Field The present disclosure is directed to dynamic orthopedic implants, methods of making same, methods of manufacturing same, and tracking devices used to fit the dynamic implants as part of a surgical procedure, as well as tracking surgical instruments, implants, patient anatomy, and a surgeon as part of implementing a surgical plan that may include displaying augmented reality scenes over real world images of the patient, surgical tools, and implants as part of surgical navigation. Summary of the Invention [Means for solving the problem]

[0003] A first aspect of the present invention is to provide a method for designing an orthopedic implant comprising: (a) iteratively evaluating possible shapes for a dynamic orthopedic implant using actual anatomical shape considerations and kinematic shape considerations; and (b) selecting a dynamic orthopedic implant shape from one of the possible shapes, wherein the selected dynamic orthopedic implant shape satisfies predetermined kinematic and anatomical constraints.

[0004] In a more detailed embodiment of the first aspect, the orthopedic implant includes at least one of a tibial implant and a femoral implant. In yet another more detailed embodiment, the method further includes collecting dynamic imaging data prior to iteratively evaluating possible shapes of the dynamic orthopedic implant. In a further detailed embodiment, the dynamic imaging data is fluoroscopic data. In yet another detailed embodiment, the dynamic imaging data is subjected to an image correction process to reduce distortion. In a further detailed embodiment, the dynamic imaging data is subjected to a feature extraction process to establish bone margins. In a further detailed embodiment, the dynamic imaging data is subjected to an initialization process to estimate bone pose. In another more detailed embodiment, the dynamic imaging data is subjected to a sequential shape and pose estimation process to generate a three-dimensional virtual model of the bone. In yet another more detailed embodiment, the sequential shape and pose estimation process utilizes input from a statistical shape model creation process. In yet another more detailed embodiment, the dynamic imaging data is segmented and classified as part of the shape and pose estimation process.

[0005] In yet another more detailed embodiment of the first aspect, the dynamic imaging data is utilized to generate a plurality of bone models that change position relative to one another through a range of motion. In yet another more detailed embodiment, the method further includes constructing a virtual anatomical model using the dynamic imaging data. In a further detailed embodiment, the virtual anatomical model includes an anatomical joint that includes at least two bones. In yet another more detailed embodiment, the anatomical joint includes at least one of a shoulder joint, a knee joint, a hip joint, and an ankle joint. In a more detailed embodiment, the anatomical model includes soft tissue. In a more detailed embodiment, the soft tissue includes a ligament. In another more detailed embodiment, the predetermined kinematic constraints are obtained from predicting normal kinematics. In yet another more detailed embodiment, the normal kinematics are extracted from a kinematic database. In yet another more detailed embodiment, the method further includes establishing implant geometric constraints for the orthopedic implant.

[0006] In a more detailed embodiment of the first aspect, the method further includes establishing manufacturing constraints for the orthopedic implant. In yet another more detailed embodiment, the method further includes establishing a surgical plan to effectuate implantation of the orthopedic implant. In a further detailed embodiment, the surgical plan includes instructions for guiding the surgical navigation system to make the predetermined bone cuts. In yet another more detailed embodiment, the instructions include orientation information indicating an orientation of the predetermined bone cuts relative to the patient's bone. In a further more detailed embodiment, the instructions include position information indicating a position of the predetermined bone cuts relative to the patient's bone. In a further more detailed embodiment, the instructions include attachment of a position tracker to the patient's bone. In another more detailed embodiment, the instructions include attachment of a position tracker to the patient. In yet another more detailed embodiment, the surgical plan includes instructions for guiding the surgical navigation system to make a series of predetermined bone cuts.

[0007] A second aspect of the present invention is to provide a patient-specific orthopaedic implant with a bearing surface optimized for the patient's anatomy, the bearing surface being virtually generated and kinematically evaluated prior to manufacturing.

[0008] In a more detailed embodiment of the second aspect, the orthopedic implant includes at least one of a tibial implant and a femoral implant. In yet another more detailed embodiment, the patient's anatomical shape was collected using dynamic imaging data generated in response to patient motion. In a further detailed embodiment, the dynamic imaging data is fluoroscopic data. In yet another more detailed embodiment, the dynamic imaging data is subjected to an image correction process to reduce distortion. In a more detailed embodiment, the dynamic imaging data is subjected to a feature extraction process to establish bone margins. In a more detailed embodiment, the dynamic imaging data is subjected to an initialization process to estimate bone pose. In another more detailed embodiment, the dynamic imaging data is subjected to a sequential shape and pose estimation process to generate a three-dimensional virtual model of the bone. In yet another more detailed embodiment, the sequential shape and pose estimation process utilizes input from a statistical shape model creation process. In yet another more detailed embodiment, the dynamic imaging data is segmented and classified as part of the shape and pose estimation process.

[0009] In yet another more detailed embodiment of the second aspect, the dynamic imaging data is utilized to generate a plurality of bone models that change position relative to one another through a range of motion. In yet another more detailed embodiment, the implant further includes constructing a virtual anatomical model using the dynamic imaging data. In a further detailed embodiment, the virtual anatomical model includes an anatomical joint comprising at least two bones. In yet another detailed embodiment, the anatomical joint includes at least one of a shoulder joint, a knee joint, a hip joint, and an ankle joint. In a more detailed embodiment, the anatomical model includes soft tissue. In a more detailed embodiment, the soft tissue includes a ligament.

[0010] A third aspect of the present invention is to provide a surgical navigation system comprising: (a) a first ultra-wide band and inertial measurement unit; (b) a second ultra-wide band and inertial measurement unit; (c) a processor communicatively coupled to the first ultra-wide band and inertial measurement unit and the second ultra-wide band and inertial measurement unit; and (d) a graphical display communicatively coupled to the processor, wherein the graphical display is configured to display an augmented reality image that is changeable in at least one of position and orientation when the graphical display is repositioned relative to at least one of the first ultra-wide band and inertial measurement unit and the second ultra-wide band and inertial measurement unit.

[0011] In a more detailed embodiment of the third aspect, the second ultra-wideband and inertial measurement unit, the processor, and the graphical display are integrated as part of a user-wearable helmet. In yet another more detailed embodiment, the helmet includes a visor onto which the graphical display projects the augmented reality image. In a further detailed embodiment, the system further includes a camera. In yet another detailed embodiment, the augmented reality image is generated by a projector. In a further detailed embodiment, the projector comprises a laser projector.

[0012] A fourth aspect of the present invention is to provide a method of planning a surgical procedure, the method including: (a) generating instructions enabling the generation of a dynamic orthopedic implant, the dynamic orthopedic implant being generated as a result of iteratively evaluating possible surface bearing shapes using actual anatomical shape considerations and kinematic shape considerations; (b) generating instructions for generating at least one of a tactile guide and a virtual guide, the instructions being patient-specific; and (c) generating navigation instructions facilitating implantation of the dynamic orthopedic implant, the navigation instructions including simultaneously tracking at least a portion of the patient and surgical tools using a combination of ultra-wideband and inertial measurement units. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram of components of an example tracked UWB-enabled ultrasound probe in accordance with the present disclosure. [Diagram 2] FIG. 1 is a flow diagram for optimizing the position of a reference unit using Position Dilution of Precision (PDOP). [Diagram 3] 1A and 1B are a pair of diagrams illustrating an exemplary design of a UWB and IMU hybrid tracking system in accordance with the present disclosure. [Figure 4] FIG. 1 is a block diagram for the processing and fusion algorithms of the UWB system and the IMU system. [Diagram 5] 1 is an illustration of a UWB system and an IMU system, optionally rigidly fixed to a mobile platform. [Figure 6] 1 is an augmented reality workflow for joint arthroplasty according to the present disclosure. [Figure 7] 1 is a workflow of a UWB / IMU enabled surgical navigation system according to the present disclosure. [Figure 8]1 is a workflow of an augmented reality and UWB / IMU enabled surgical navigation system according to the present disclosure. [Figure 9] FIG. 1 is a block diagram of an exemplary embodiment of a pre-operative and intra-operative workflow for creating a virtual guide for joint arthroplasty using an AR system and a hybrid tracking system such as IMU and UWB in accordance with the present disclosure. [Figure 10] FIG. 2 illustrates a registration relationship between an augmented reality system, a UWB system, and an IMU system according to the present disclosure. [Figure 11] 13 is an example photograph of a virtual bone cut planning showing an orange 3D model indicating another user's position relative to the augmented reality system as part of collaborative surgical planning. [Figure 12] 13 is another example photograph of a virtual implant placement plan showing an orange 3D model indicating another user's position relative to the augmented reality system as part of collaborative surgical planning. [Figure 13] FIG. 1 illustrates the transformation of different coordinate systems in augmented reality. [Figure 14] FIG. 1 is a block diagram of an exemplary method for a rectangular shape-based marker detection approach. [Figure 15] FIG. 2 is a block diagram of an exemplary method for a two-dimensional feature matching approach. [Figure 16] FIG. 1 is a block diagram of an example method for a three-dimensional model-based object detection approach. [Figure 17] A diagram of a basic forward mapping rendering framework that includes two rendering steps. [Figure 18] FIG. 13 illustrates the use of two sets of rays to align two different coordinate systems in an absolute orientation problem. [Figure 19] FIG. 1 is a block diagram of an example process for tracking bones based on a hybrid UWB and IMU tracking system in real time. [Figure 20]1 is a depiction of an exemplary augmented reality system for use in combination with an IMU system and a UWB system, as well as a head mounted display (AR visor) for displaying information to a user. [Figure 21] 1 is a depiction of an exemplary augmented reality system that utilizes a laser projector to display information to a user. [Figure 22] 1 is a depiction of an example integrated camera and UWB and IMU device that provides the ability to integrate optical tracking into a positioning algorithm. [Diagram 23] FIG. 1 is a block diagram of an example process for registering a head-mounted augmented reality system incorporating a feature matching technique and a hybrid tracking system. [Figure 24] FIG. 1 is a block diagram of an example process for registering a laser projector-based AR system. [Diagram 25] FIG. 1 illustrates the integration of an augmented reality system with a UWB / IMU system. [Figure 26] FIG. 1 is a block diagram of an exemplary process for simulating and evaluating intraoperative kinematics. [Figure 27] FIG. 13 is a block diagram of an exemplary process for providing feedback to a surgical tool. [Figure 28] 1 is an illustration of an augmented reality system placing virtual bones and virtual components (e.g., pin guide locations) on a real-world image of the lateral side of the lower leg. [Figure 29] 13 is a pair of photographs showing a normal real-world field of view, as well as an augmented reality field of view using a custom reference device to register the augmentation system to the patient's space. [Diagram 30] Photographs showing the transfer of anatomical tracking from a custom registration device to rigidly fixed markers for augmented reality surgical navigation. [Diagram 31] 13 is a photograph showing tracking of surgical instruments relative to anatomy using augmented reality to position the hip cup. [Diagram 32]This is a representation of what a user would see through hip navigation and an augmented field of view in the correct orientation. [Diagram 33] 1 is an exemplary oscillating saw with UWB and IMU tracking capabilities. [Diagram 34] A four-in-one mechanically adjustable cutting block that can be utilized for multiple sizes and implant groups. [Diagram 35] An adjustable "smart" cutting block for performing femoral resections in total knee replacement surgery. [Diagram 36] FIG. 11 is a flowchart showing an exemplary process for creating and using a smart cutting guide, which can be actively tracked by a UWB / IMU / AR system and which can be pre-programmed or can wirelessly receive the location of the planned cutting plane. [Figure 37] FIG. 11 is a flowchart showing a further exemplary process for creating and using a smart cutting guide, which may be actively tracked by a UWB / IMU / AR system and which may be pre-programmed or may wirelessly receive the location of the planned cutting plane. [Figure 38] FIG. 13 is a visual depiction of the field of view through an exemplary augmented reality helmet for visual aid to position the distal cutting guide (blue line) per preoperative planning. [Figure 39] 1 is a visual depiction of a field of view through an exemplary augmented reality helmet for positioning a distal cutting block. [Diagram 40] 1 is a visual depiction of the field of view through an exemplary augmented reality helmet looking at an oscillating saw equipped with UWB and IMU sensors for tracking in an augmented system. [Diagram 41] 1 is a visual depiction of a view through an exemplary augmented reality helmet viewing a smart instrument for knee resurfacing. [Diagram 42] FIG. 1 is a flow diagram illustrating a method for ultrasound-based surgical navigation using UWB tracking and virtual guidance. [Diagram 43] 1 is a depiction of a user of an augmented reality system that is able to visualize the proper needle path and anatomical structures via an augmented display. [Diagram 44] FIG. 13 is a depiction of a user of an augmented reality system that allows them to view ultrasound B-mode images that are augmented over the patient's anatomy to aid in anatomical referencing and eliminate the need for the operator to look at a computer monitor. [Diagram 45] A depiction of a user of an augmented reality system being able to see anatomical structures constructed from ultrasound data and displayed on the augmented scene alongside B-mode images. [Diagram 46] FIG. 1 is a flow diagram showing that one direct application of augmented reality and ultrasound is to create a framework for navigating a subchondroplasty procedure. [Figure 47] 1 is a screenshot from the surgical planning software showing the reconstructed patient bones. [Figure 48] 13 is a screenshot from the surgical planning software showing the mechanical axes projected onto the x-ray image. [Figure 49] 13 is a screenshot from the surgical planning software showing the placement of the tibial implant. [Figure 50] 13 is a screenshot from the surgical planning software showing the placement of the talar implant. [Figure 51] 1 is a flow chart for UWB / IMU surgical navigation of total ankle arthroplasty. [Figure 52] FIG. 13 is a flow diagram for using a UWB tracking system and an IMU tracking system in a total knee replacement surgery. [Diagram 53] FIG. 1 is a flow diagram of an exemplary process that utilizes dynamic data over an episode of surgical care to create patient-specific implants and instruments, place the implants, and monitor performance post-operatively. [Figure 54] Includes photographs showing examples of dynamic joint reconstruction. [Figure 55]1 includes images showing the superimposition of patient-specific geometric information onto single-plane digital fluoroscopy. [Figure 56] FIG. 1 is a flow diagram illustrating an exemplary process for generating anatomical information from dynamic image data. [Figure 57] Fluoroscopic images of the geometric calibration grid before and after edge distortion removal are included. [Figure 58] 1 is a depiction of the process flow for generating a 3D model from features of a fluorescent screen perspective image. [Figure 59] FIG. 1 illustrates initialization by a hybrid classifier. [Figure 60] FIG. 13 illustrates a modified KPCA model applied to the knee joint. [Figure 61] FIG. 1 illustrates a process for optimization of pose and shape parameters. [Figure 62] FIG. 13 is a diagram illustrating the relationship between an input space distance and a feature space distance. [Figure 63] FIG. 1 is a series of images showing reconstruction of a patient's anatomy of the femur and tibia using fluoroscopy compared to CT. [Figure 64] It includes a series of fluoroscopic images with border overlap. [Figure 65] Illustrative images of shoulder and hip reconstructions using fluoroscopy are included. [Figure 66] FIG. 2 illustrates how the geometric space is first decomposed by extracting a volume of interest (VOI) as a volume in which a part template may reside. [Figure 67] FIG. 2 illustrates how a volume of interest (VOI) is determined by detection. [Figure 68] FIG. 1 shows that an alternative to using statistical shape transformations is to directly identify features on the image and use so-called And-Or trees for shape identification and transformation. [Figure 69] A series of computer-generated illustrations breaking down the anatomy of the femur into primitive shapes. [Figure 70] 1 is a bone model showing the location of ligaments extracted from imaging data. [Figure 71] FIG. 13 shows tibia and femur contact maps for deep knee flexion. [Figure 72] FIG. 13 illustrates how static and fluoroscopic data can be used to combine kinematics and morphology. [Figure 73] FIG. 11 is a distribution map of two distal femurs showing relative cartilage thickness as part of a dynamic imaging study. [Figure 74] FIG. 1 is a flow diagram for estimating cartilage thickness from dynamic data. [Figure 75] FIG. 13 is a probability distribution map of ligament positions for the distal femur and proximal tibia. [Figure 76] 1 is a pair of models and charts utilized to predict ligament seat position and ligament length envelope changes. [Figure 77] A pair of distal femurs modeled to map predicted cartilage loss. [Figure 78] FIG. 1 is a process flow diagram for creating and using a kinematic training network to identify kinematic patterns. [Figure 79] A knee joint bone model showing medial collapse. [Figure 80] 1 is a knee joint bone model showing an estimation of normal joint alignment and changes in ligament lengths. [Figure 81] 1 is a flow chart of an exemplary process for calculating patient-specific ligament stiffness. [Figure 82] FIG. 1 is a process flow diagram outlining a process for creating a patient-specific dynamic implant. [Figure 83] 13 is a chart showing an exemplary process for utilizing estimations from either healthy kinematic profiles or ligament stiffness as part of creating a patient-specific dynamic implant. [Figure 84] 1 is a depiction of different polyethylene sagittal curvatures and corresponding femoral implants. [Figure 85]1 is a flowchart illustrating an exemplary process for implant shape optimization. [Figure 86] A computer-generated model of a passive knee model. [Figure 87] A computer-generated model of the flexion-driven model with a ligament model that can be customized to fit the implant being created. [Figure 88] FIG. 13 is a flow diagram of an exemplary process for creating and using a network to define optimal CAD parameters for a patient implant that is recoverable from kinematic data. [Figure 89] As an initial guess for the patient-optimized cut box, an approximate representation of the J-curve of the offset implant by a line segment. [Figure 90] 1 is a computer-generated resected femur model using a patient-optimized cutting box. [Figure 91] 13 is a computer-generated resected femur model resectioned using a conventional modified implant cutting box. [Figure 92] 1 is a computer-generated bone model showing the amount of additional bone resected using a conventional modified implant cutting box versus the amount of additional bone resected using a patient-optimized cutting box. [Figure 93] 1 is a flowchart outlining the overall concept of creating dynamic implants and executing surgical plans using a smart cutting guide intraoperatively. [Figure 94] 1 is an illustration of the use of UWB and IMU trackers as part of a post-operative assessment. [Figure 95] FIG. 1 is an exemplary diagram listing patient-specific implants and the parameters used to create the implants. [Figure 96] 1 is an exemplary overview of a process for creating a patient-specific implant from dynamic data. [Figure 97] FIG. 1 is a flow diagram showing the information flow for extracting design parameters from dynamic data and a statistical atlas. [Figure 98]FIG. 13 is a flow diagram showing joint reconstruction from dynamic data. [Figure 99] 1 is a screenshot from a software package showing an example of dynamic joint reconstruction. [Figure 100] FIG. 1 is a process flow diagram of an exemplary process that utilizes dynamic data over an episode of surgical care to create patient-specific implants and instruments, place the implants, and monitor performance post-operatively. [Figure 101] FIG. 13 is a screenshot from a knee planning software application that includes kinematic data as captured by the dynamic imaging process and allows for the development of information regarding the soft tissue envelope throughout the range of motion that will aid in improving ligament balance when developing a surgical plan as opposed to adjustments in the operating room. [Figure 102] 1 is a depiction of a flexion-driven kinematic knee model. [Figure 103] Includes a set of simulation models to analyze patient motion and dynamics for input to and output from the dynamic implant design process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Exemplary embodiments of the present disclosure are described and illustrated below to encompass dynamic orthopedic implants, methods of making same, methods of manufacturing same, and tracking devices used to track the dynamic implants as part of a surgical procedure, as well as surgical instruments, implants, patient anatomy, and the surgeon as part of implementing a surgical plan that may include displaying augmented reality scenes over real-world images of the patient, surgical tools, and implants as part of a surgical navigation. Of course, it will be apparent to those skilled in the art that the embodiments discussed below are exemplary in nature and may be reconfigured without departing from the scope and spirit of the present invention. However, for clarity and accuracy, the exemplary embodiments discussed below may include optional steps, methods, and features that a person skilled in the art would recognize as not necessarily falling within the scope of the present invention.

[0015] Ultra-Wideband Positioning for Wireless Ultrasonic Tracking and Communication Currently, wireless ultrasound probes can communicate ultrasound information and provide real-time location and orientation information as part of a single unit / package. A system is disclosed herein that enables wireless ultrasound probes with integrated communication and incorporates ultrawide band (UWB) technology. An exemplary wireless ultrasound probe, including part of an exemplary system, includes a first UWB unit for ultrasound wireless data streaming and a second UWB unit for real-time location tracking and wireless control / feedback. The wireless ultrasound probe also integrates the two UWB units with an IMU and a microcontroller (MCU) to process the location, orientation, and ultrasound data.

[0016] Ultrasound probe design In accordance with the present disclosure, an exemplary wireless ultrasound probe may include any number (one or more) of ultrasound transducers. As a further example, a wireless ultrasound probe may integrated one or more UWB transceiver units including one or more antennas for positioning / control and real-time data streaming, and one or more inertial measurement units (IMUs) including one or more accelerometers, gyroscopes, and magnetometers. FIG. 1 includes an exemplary block diagram illustrating exemplary components of an exemplary wireless UWB-enabled ultrasound probe.

[0017] As a further example, as shown in FIG. 3, an exemplary UWB unit may include four or more antennas (such as a tetrahedral layout). By having at least four antennas that are not all coplanar with one another, the antennas operate to be able to resolve three-dimensional orientation changes, considering that any three antennas form a unique plane and vertical axis in three dimensional (3D) space. The antennas may be connected to the UWB transceiver using a 1×N radio frequency (RF) switch, where “N” is any integer. As a further example, the UWB unit may include an MCU that operates to control the activation and deactivation of the RF switch, as well as the channel switching of the RF switch. In this manner, each antenna generates a signal that allows for the derivation of location and orientation information associated with each of the antennas. In an exemplary form, determining the location of each of the four antennas results from determining the time it takes for a signal to be transmitted from a central UWB anchor unit and received by each of the four antennas. In other words, determining the position of each UWB tracking unit is accomplished between each tag in the UWB tracking unit and a central UWB anchor unit, which may be connected to a central processing unit (CPU) for processing and generating visual display instructions.

[0018] UWB Positioning An exemplary UWB tracking unit according to the present disclosure may connect multiple antennas to a single transceiver, which allows for ranging, locating, and tracking multiple UWB antennas (targets) utilizing the same UWB transceiver unit. As an example, the antennas in the UWB tracking unit may function as a master role / unit or a peripheral role / unit, with the master role functioning to generate timing references for the peripheral antennas in the system. The antennas in each UWB tracking unit may be arranged in any configuration, provided that there are four or more antennas and that one of these antennas is not on the same plane as the other three antennas. For example, as previously mentioned, a tetrahedral antenna configuration satisfies the aforementioned condition. Regardless of the antenna configuration, the four antennas may connect to a single UWB transceiver, with each antenna serving as a reference point for locating. With a single timing circuit and a single transceiver for feeding UWB pulses to the multiple antennas, this configuration enables clock synchronization between all reference points within the UWB tracking unit. This configuration may greatly improve the flexibility of the installation of the master unit, as well as facilitate the calibration procedure. In short range localization applications, a single master unit may be sufficient to provide adequate positioning data for localization. In wide range localization applications, multiple master units may be used. By way of example, the timing circuits of the master units may be synchronized during operation in either a wired or wireless manner.

[0019] The aforementioned exemplary tetrahedral design may allow for the determination of the position and orientation of each UWB unit. An additional IMU may be used to accurately determine the orientation. The output of both sensor systems is fused such that errors from the UWB and IMU are reduced by fusion and filtering techniques. One example of a method would be to use a recursive Bayesian estimation filter to complement the orientation estimation between the UWB and IMU systems. The IMU system may be inaccurate and subject to magnetic artifacts for which the UWB system may be used to provide correction. In addition, the tetrahedral design of the UWB units may allow for monitoring translation estimation between the antennas and improved translation estimation.

[0020] As part of an exemplary system, one or more UWB tracking units may communicate with one or more external reference units that may be mounted or located at different locations within a user environment (e.g., an operating room during a surgical procedure). In an exemplary form, the reference unit may include one or more antennas in any number of orientations and configurations. The reference unit may also communicate with other reference units and master units via wired or wireless connections. In an exemplary form, the reference unit may behave as a master unit to other UWB tracking units and may also be configured to determine the location of any other reference units or UWB tracking units based on positioning techniques such as received signal strength, time difference of arrival, time of flight, and / or angle of arrival. The placement of the reference units may be arbitrary or may be optimized using estimates based on position dilution of precision (PDOP) (see FIG. 2). A user may input the tracking area of ​​interest, the desired accuracy, and the number of available reference units. The optimization software determines the optimal location of the reference units. The software may also be configured to optimize the reference unit placement in 2D or 3D.

[0021] In another alternative exemplary configuration, the UWB tracking system and / or the IMU tracking system can be rigidly fixed to a mobile platform, which may be in the form of a surgical robotic system, to improve performance without relying on external reference units being placed around the operating room (see FIG. 5).

[0022] UWB communication One exemplary advantage of the exemplary system is wireless position and orientation tracking of the ultrasound probe. A second exemplary advantage of the proposed system is wireless communication between the ultrasound probe and a central processing unit (CPU) using UWB technology. The high throughput available with UWB communication makes the proposed system ideal for transmitting real-time ultrasound acquisition information. The real-time transmission of B-mode data, radio frequency (RF) data, I / Q data, or any other acquired ultrasound signals to the CPU is accomplished by a UWB transceiver having a data throughput capability, preferably at least approximately 6-7 Mbps. Lossless data compression can be utilized to compress the real-time data stream so that at least 20-30 frames per second of RF, IQ, and / or B-mode data can be transmitted wirelessly for real-time processing and display.

[0023] Real-time Virtual Guidance for Joint Replacement Surgery with Augmented Reality and IMU Tracking Arthroplasty is a surgical procedure that restores function to a joint by resurfacing one or more of the bones that comprise the joint. Arthroplasty may include surgical procedures that restore normal limb alignment, design and optimize precise implants, secure implant fixation, and / or provide proper soft tissue balance and stability. As discussed in more detail herein below, the present disclosure provides a virtual cutting guide configured in an AR system to provide one or more computer-generated images that are superimposed on the surgeon's / user's real-world field of view to guide the surgeon during a surgical procedure, such as a joint replacement procedure in an operating room. With the development of augmented reality technology, self-referencing hybrid navigation modules based on UWB and IMU technologies have been used to improve the accuracy of simultaneous translational and rotational tracking of surgical tools and the surgeon / user during a surgical procedure, as well as the assessment of intraoperative and postoperative kinematics of the patient.

[0024] In an exemplary form, the subject matter disclosed herein addresses medical applications of augmented reality as part of a surgical procedure, such as joint arthroplasty, using a head-mounted or handheld (or body-attached) display, a regular computer screen, a mobile device screen, or any combination thereof, for viewing the augmented reality content. Specifically, this exemplary embodiment encompasses systems and methods for guiding a surgical procedure with high-precision computer-generated imagery by placing virtual objects on surfaces in a user's real-world field of view in a consistent manner that may provide situations where the user can no longer distinguish between real and virtual objects.

[0025] Augmented Reality Augmented reality (AR) is a field of computer vision and computer graphics that integrates embedded synthetic information into real-world environments. In an augmented reality experience, an end user is equipped with additional information about the real world around the user from computer-generated images overlaid on the user's real-world field of view. Rather than virtual reality, which immerses the user solely in a synthetic environment, augmented reality allows the user to interact with a synthetic world overlaid on top of the real-world environment. As discussed in more detail herein below, exemplary embodiments have non-surgical applications such as, without limitation, computer games, simulations, engineering, training, and a variety of other applications.

[0026] An exemplary AR system according to the present disclosure may consist of one or more color cameras, one or more depth cameras, one or more image processing units, a central processing unit, one or more network communication systems (such as Bluetooth LE, UWB, or Wifi), and any of the following feedback mechanisms: (a) optical see-through feedback, typically a head-mounted display where information is superimposed on the visor (see FIG. 20); (b) video overlay feedback, which may be a handheld device where the display renders video capture by the camera and information is superimposed on the surface of the video; and (c) spatial projection feedback, which may be achieved via one or more laser projectors where information is projected directly onto the object (see FIG. 21). As used herein, AR vision / tracking system generally refers to hardware and software that operates to track objects within a user's visual volume via computer vision, object recognition, tracking, and motion estimation.

[0027] One of the main problems faced when wanting to provide an augmented reality experience is the registration of projected images to the natural environment. More specifically, it has not been possible to accurately position virtual objects on the corresponding target surfaces in the real world (i.e., the natural environment) and update this positioning as the objects in the natural environment and the user move. However, accurate alignment of real-world objects with objects projected in augmented reality is essential for image-guided surgery, as inaccurate or unstable registration can cause non-alignment of bone cuts, implant placement, and overall negative effects on key performance of the surgical procedure.

[0028] Referring to FIG. 6, the flow diagram on the left shows a process for pre-operatively creating cutting guides, which may include the creation of physical alignment guides, physical cutting guides, virtual alignment guides, and virtual cutting guides that are utilized as part of an augmented reality-enabled surgery. The pre-operative step involves creating a virtual model of the patient-specific anatomical structures that will be subject to the surgical procedure. The virtual anatomical model may be created from static imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), and / or x-ray, or from dynamic imaging modalities such as fluoroscopy or tracker-enabled ultrasound. In an exemplary form, the virtual anatomical model may comprise any bone or group of bones, such as the knee, hip, shoulder, ankle, and / or joints, including osteophyte segments. Additionally, the virtual anatomical model may be used in an augmented reality system to accomplish collaborative surgical planning.

[0029] Collaborative surgical planning may involve a process that connects two or more users over a network, which does not have to be local, via augmented reality hardware (helmets) to create a pre-operative surgical plan. The users creating the surgical plan can view a common virtual anatomical model and implants and adjust the parameters of the surgical plan (implant size and location) until the desired surgical plan is achieved. As changes are made to the virtual anatomical model by a user, these changes are relayed over the network to other users, and each user's visual display is updated accordingly. In a similar manner, using avatars, users can see the location of other users relative to the anatomical model, as will be discussed in more detail herein below.

[0030] IMU / UWB / AR-assisted dynamic planning may include capturing dynamic data of the patient's anatomy via one or more imaging modalities or after imaging using a combination of IMU and / or UWB positioning systems. Communication and data transmission between each subsystem (IMU, UWB, augmented reality (AR)) may be achieved through the use of application program interfaces (APIs) that convert data from each subsystem into a format readable by the other or the CPU. This set of communication through software APIs is a common framework for the integration of all AR / IMU / UWB. This dynamic data may be optionally included in the surgical planning process to establish pre- and post-operative joint status in the planning process.

[0031] The creation of virtual and physical alignment and cutting guides may be created using a combination of preoperative data and agreement on the surgical plan. Each physical alignment guide need not be a cutting guide, but may be a placement guide having one or more surfaces configured to match or align with a patient's unique bone surface. The physical alignment guides can be used to register the AR system to the patient in the operating room by placing it in a unique position on the patient based on the patient-matched shape of the alignment guide. Apart from the physical guides, virtual cutting guides can be created as part of the surgical plan, consisting of virtual parameters of the cutting plane, axes for the surgical axes or pin locations, patient geometry, and implant parameters. The parameters of the virtual cutting guides can be stored and transmitted electronically without the need for a physical manufacturing process that may otherwise be required for physical cutting guides.

[0032] Referring back to FIG. 6, the flow diagram on the right shows the process of an intraoperative workflow as part of utilizing an augmented reality surgical cutting guide or a physical cutting guide. This exemplary workflow includes registration and calibration of applicable tracking systems, which may be AR, UWB, or IMU, or any combination thereof. The registration process ensures that all tracking systems are synchronized to the same coordinate system. A more detailed description of the registration process between the various tracking systems is described in more detail with respect to FIG.

[0033] In an exemplary configuration, the optical tracking system may be integrated with a UWB reference unit, which may assist and / or provide a standard frame of reference for each applicable tracking system to synchronize with (see FIG. 22).

[0034] It should also be understood that the AR system has similar limitations as the optical tracking system, which may only track objects within its region of view and / or its visual volume and line of sight restrictions. It is understood that the AR system may circumvent these limitations when operating in combination with a UWB / IMU system (see FIG. 20), and there may be more than one AR system operating simultaneously. The UWB / IMU system may be attached to the patient, instrument, or any other object so that the position of the patient, instrument, or any other object may be tracked wirelessly even if they are outside the vision tracking volume of the AR system (see FIG. 25). The surgeon and surgical assistant may each have an AR system where information captured by each system is shared wirelessly. For example, if the surgeon's AR vision tracking system has a better view of the patient's joints, it may relay its tracking information to other AR systems that do not have the patient's joints within their vision tracking volumes.

[0035] It should be further appreciated that the AR information display / feedback may be implemented in the form of a head mounted display where information is relayed to the AR visor (see FIG. 20), or as a laser projection system where information is projected directly onto the patient (see FIG. 21). For head mounted displays, there may be indicators superimposed on the AR visor for objects that are currently outside the viewing volume of the AR system.

[0036] After registration and calibration, one or more reference trackers can be attached to the patient and / or the applicable surgical tools. Exemplary reference trackers can each be equipped with an IMU and / or UWB and attached to the patient to dynamically track the position of the patient's bone or other tissue of interest. It is noted that the AR-enabled system of the present disclosure does not require a reference tracker, since a vision tracking system can be utilized to track the patient's bone (or other surgical tissue of interest) and the applicable surgical tools as long as the object is within the viewing volume of the AR camera (see FIG. 15 and FIG. 16). Nevertheless, the utilization of a reference tracker provides additional assurance that the patient's bone(s) are always monitored, even outside the tracking volume of the AR system (see FIG. 25). The use of a reference tracker can reduce the computational load required for computer vision, thereby easing the computational burden and improving the performance of the AR system.

[0037] After attachment of the reference tracker to the patient, the patient registration process may include aligning the physical guide with respect to the patient's anatomy. As discussed above, this exemplary process may provide for the creation of a post-operative planning physical guide that may be registered to coordinates specific to the patient's anatomy during surgery. For example, an exemplary registration method for knee arthroplasty may use a unique guide located on a combination of anatomical features of the patient's bone(s), which may include, but are not limited to, the anterior cortical points, lateral and / or medical condyles for femoral registration, and the upper anterior portion of the tibia, lateral and / or lateral tibial condyles for tibial registration. In this exemplary embodiment, a position and orientation tracker including at least one of an IMU unit and a UWB unit is attached to the physical guide. After positioning the physical guide on the bone in a unique known location, a reading is taken from the tracker to determine the relative position and / or orientation of the physical guide, which uses the position and / or orientation of the guide to calculate the position of the patient's bone in the coordinates of the applicable tracking system. When using an AR system, computer vision may be used to recognize and track the physical guide within the viewing space. The AR system can also be used in combination with a UWB / IMU tracker (see Figures 19, 29, and 30).

[0038] Alternatively or additionally, the exemplary embodiment may include a patient registration step using a tracked stylus or computer vision as part of the virtual registration. In an exemplary form, the virtual placement guide may be registered to the patient using a tracker including at least one of a UWB unit and an IMU unit. The tracker may be implanted as part of a stylus to which the surgeon / user may register anatomical landmarks and / or map the patient's bone surface. The tracking data may be specific anatomical landmarks or a point cloud representative of the patient's bones, which may be used to register the patient to the preoperative surgical plan. In situations where an AR system is used, the AR system may be used as a stand-alone tracking system to accomplish patient registration via computer vision (see FIG. 15, FIG. 16) or in combination with the aforementioned tracker (see FIG. 19).

[0039] With reference to Figures 6 and 7, an exemplary workflow for an embodiment of UWB / IMU-enabled surgical navigation after patient registration is shown in more detail. In Figure 6, reference numeral 8 corresponds to an embodiment of UWB / IMU-enabled surgical navigation described more fully in Figure 7. As shown in Figure 7, in accordance with the pre-operative surgical plan, a patient's bone is resected / cut with a tracked surgical instrument as part of the surgical procedure, corresponding to reference numeral 11. Feedback information of the tracked instrument may be displayed on a stand-alone display unit. By way of example, the exemplary process will be described with respect to an ankle arthroplasty workflow using UWB / IMU-based surgical guidance in Section 1. However, it should be understood that this process may also be used in other surgical procedures, including but not limited to total knee arthroplasty.

[0040] In situations where an actual physical cutting guide is utilized, the cutting guide may incorporate UWB and / or IMU trackers (as with a smart cutting guide) to provide position and orientation information to the user with respect to the cutting guide. If a physical cutting guide is utilized in combination with a position and / or orientation tracker, the information to the user may include guidance information useful as part of a joint resection. In this exemplary embodiment, the physical cutting guide may have single or multiple adjustable cutting slots, each of which may be adjusted manually (see FIGS. 33 and 35) or via motorized hardware (see FIGS. 34, 36) to control a series of pre-programmed or predetermined positions and orientations of the cutting slots based on a surgical plan to create the desired resection of the patient's bone. In an alternative exemplary embodiment, the manually adjustable cutting slots may be blocked or eliminated via robotics or mechatronics implementation if the current resection plane does not align with the plane of the preoperative surgical plan.

[0041] In addition to physical cutting guides incorporating trackers, it is also within the scope of the disclosure for surgical instruments, such as, but not limited to, surgical saws to incorporate feedback mechanisms. In an exemplary form, the feedback mechanism may inform the surgeon / user if the current / projected resection deviates from the preoperative surgical plan. By way of example, the surgical instrument may include a tracker including at least one of a UWB unit and an IMU unit that may wirelessly transmit at least one of position and orientation data to a central computer (see FIG. 33). In this manner, the use of a patient reference tracker 6 discussed with respect to FIG. 6 in combination with the surgical tool tracker may determine the orientation and depth of the current resection relative to the patient's anatomy and the preoperative plan. The computer may send a feedback signal to the instrument if the orientation and depth of the resection deviates from a predetermined tolerance according to the surgical plan, and the computer may send information (i.e., feedback) to the surgical tool indicating that the current orientation and / or depth is outside the tolerance of the surgical plan and that the surgeon / user should reposition the surgical tool to arrive within an acceptable tolerance. The information / feedback to the surgical instrument may be implemented to reduce or disable power to the powered hardware, or trigger a single or multiple haptic motors embedded within the instrument, thereby providing haptic feedback to the surgeon. Alternatively or additionally, the instrument may incorporate a visual display that provides instructions to the surgeon on corrective actions required to reposition the surgical instrument to achieve a portion of the surgical procedure consistent with the preoperative plan. As an example, a surgical instrument may include a powered gyro to slightly deflect the instrument, and additionally provide haptic directional feedback to the surgeon to tilt the instrument in a particular direction.

[0042] As part of an exemplary embodiment, after preparation of the patient tissue to receive the implant, one or more orthopedic implant trials, each including position and orientation tracking capabilities, may be utilized as part of step 16. As previously discussed, each trial may include one or more of a UWB unit and an IMU unit that wirelessly provide feedback regarding at least one of the position and orientation of the implant trial. In this manner, the central computer may receive information from the implant trial trackers and verify the positioning of the implant (when using bone tracking with a physical tracker or a vision tracker) or provide information that the implant trial is out of position relative to the patient's bone. Ultimately, an implant trial that fits the patient and provides kinematics and ligament balance consistent with the preoperative plan is utilized, at which point the surgeon moves to the implant placement step.

[0043] The implant placement process, identified by reference number 18 in FIG. 7, involves the selection and placement of the orthopedic implants to be implanted over their useful life. By way of example, each implant may include one or more of a UWB unit and an IMU unit that wirelessly provide feedback regarding at least one of the implant's position and orientation. In this manner, the central computer may receive information from the implant tracker and verify the positioning of the implant (when using bone tracking with a physical tracker or a vision tracker) or provide information that the implant is out of position relative to the patient's bone. Ultimately, the implant is accurately positioned and secured to the patient's anatomy.

[0044] After the implant is secured to the patient's anatomy, the surgeon can try the implant out within its range of motion. For the tracker(s) associated with the implant, the central computer records position and / or orientation information output from the tracker and correlates this to calculate the position of the implant. This implant tracking can be compared to the pre-operative kinematics and / or pre-operative kinematic plan to assess whether the implant's range of motion is within the intended limits.

[0045] Turning now to Figures 6 and 8, an exemplary workflow for an embodiment of AR and UWB / IMU enabled surgical navigation after patient registration is shown in more detail. In Figure 6, reference numeral 10 corresponds to an embodiment of AR and UWB / IMU enabled surgical navigation described more fully in Figure 8. As shown in Figure 8, in the context of a physical cutting guide, the cutting guide can incorporate at least one of a UWB unit and an IMU unit to provide position and orientation data for the cutting guide, which can be utilized with a reference tracker on the patient's bone to provide guidance during the surgical procedure. The cutting guide can have single or multiple adjustable cutting slots, which can be adjusted manually (see Figures 34 and 36) or via motorized hardware that controls the position and orientation of the cutting slot to produce the desired resection (see Figures 35 and 37). In another alternative exemplary embodiment, the manually adjustable cutting slot can be blocked / closed if the current resection plane does not match or fall within a predetermined tolerance of the plane required by the preoperative surgical plan. The AR system can aid in resection by providing a virtual resection plane on the AR visor where the user can manually adjust the cutting slot of the cutting guide as illustrated in FIG. 34.

[0046] In step 13 of FIG. 8, an exemplary surgical instrument such as a bone saw can incorporate multiple feedback mechanisms to inform the surgeon that the current resection has deviated from the surgical plan. The exemplary instrument can include a tracker including at least one of a UWB unit and an IMU unit that can wirelessly transmit data to a central computer. When used in combination with the reference tracker 6 of FIG. 6, the orientation and depth of the current resection can be determined. If the orientation and depth of the resection deviates from the pre-defined tolerances of the pre-operative surgical plan, the computer can send feedback to the instrument. The feedback can be implemented to reduce or disable power to the motorized hardware or trigger a single or multiple haptic motors to provide haptic feedback to the surgeon. Additionally or alternatively, feedback can be displayed on a display associated with the surgical instrument to guide the surgeon in repositioning the instrument to reach the proper orientation and / or position consistent with the pre-operative surgical plan. The instrument can also include a motorized gyro to slightly deflect the instrument and provide haptic directional feedback to the surgeon to tilt the instrument. In this exemplary embodiment, the AR system, in combination with the above-mentioned technology, may assist by providing warnings, feedback, and / or directional information regarding the resection plane orientation and resection depth superimposed on the AR visor or computer screen, or on the laser projected onto the patient and instrument (see FIG. 27).

[0047] As shown as step 14 in FIG. 8, the AR system may provide virtual resection and / or resurfacing guidance showing the location and orientation of areas where joint resection and / or resurfacing may be required. For example, a knee implant may require multiple resection cuts to the patient's distal femoral joint and proximal tibial joint. The AR system may show all the resection planes required so that the surgeon can make adjustments before resecting any bone. The AR system may display the required resections procedurally as it guides the surgeon during each resection step (see FIG. 40).

[0048] If physical cutting guides are available, the exemplary AR system can provide virtual pin locations to assist the surgeon in cutting pin location holes as part of the visual display on the AR visor or computer screen, or the laser projected onto the patient. In this exemplary embodiment, the cutting guides can be generic or patient-specific. In either case, the AR visor can virtually display the resection that will be produced by the current pin placement so that the surgeon can make adjustments before resecting the bone. The AR system can also procedurally display the pin locations for each cutting guide required during each procedure (see FIGS. 28, 38, 39).

[0049] After resection of the patient's bone(s), the exemplary process flow proceeds to step 17 where the AR system may operate in combination with trial components enabled in at least one of the UWB unit and the IMU unit and / or utilize a reference tracker on the patient's bone(s) to enable orthopedic implant trial placement. The AR may overlay information regarding the placement of the trial components during motion, the patient's range of motion, and the contact area of ​​the implant (see FIG. 26 ).

[0050] Once the implant trial sequence is complete and the implant range of motion and contact are verified, the AR system can work in combination with one or more surgical instruments (enabled by at least one of the UWB unit and the IMU unit) and bone reference trackers to virtually provide placement information for the implant on an AR visor or computer screen, or with a laser projected onto the patient. For example, the AR system can utilize position and orientation information from the trackers on the patient's bones and surgical instruments to provide visualization of the placement of the implant by superimposing a virtual bone and virtual acetabular cup onto the patient in the context of a total hip arthroplasty (see FIGS. 29-32).

[0051] After implantation of the orthopedic implant, the surgeon can try the implant in its range of motion, as identified by reference number 20 in FIG. 8. The AR system can work in combination with reference trackers on the patient's bone and implant to perform a kinematic verification where the surgeon can manually manipulate the bone and / or joint. The AR can overlay information regarding the placement of the implant during motion, the implant's range of motion, and the contact area of ​​the implant (see FIG. 26).

[0052] Turning to FIG. 9, a block diagram illustrates an exemplary embodiment of a pre-operative and intra-operative workflow for creating a virtual fixture for arthroplasty using an AR system and a hybrid tracking system such as IMU and UWB in more detail than that shown in FIG. 6. In the pre-operative workflow, the patient-specific anatomy must be extracted or constructed by using several imaging modalities such as computed tomography (CT), magnetic resonance imaging (MRI), dual plane X-Ray radiograph, single plane fluoroscopy, or a combination thereof. After the calibration process, a reconstructed 3D virtual model of the patient-specific anatomy is created. Using the reconstructed virtual bone model, virtual cutting guides and implants can be created in a pre-planning process.

[0053] In the intraoperative workflow component, the patient may be tagged with multiple rigid units attached to the patient's anatomy. In an exemplary embodiment, the term "unit" may be understood as one or more sensors for measuring the position and orientation of a target that is wirelessly connected to a self-referencing hybrid navigation system based on IMU and UWB technologies. After positioning the one or more rigid units on the patient, camera calibration may be performed to provide registration between the real world and the virtual world provided by AR. In other words, the camera registration process may include registration between different coordinates of the virtual coordinate system and the real world coordinate system using a target tracker that may be selected from one of the following trackers: marker detection, feature matching, 3D model-based object detection, self-referencing hybrid navigation system, or any combination thereof. After registration and virtual guide creation, the virtual guide may be rendered using one or more augmented reality modules to enable depiction of the virtual guide aligned with the patient's anatomy in the real world. In this manner, the virtual guide may be visually overlaid on the patient's anatomy, either by overlaying an image on a head-mounted or handheld (or body-attached) display, or via a projection system, which may consist of a laser or projector to overlay the virtual guide. When the implant components are sized and placed on the virtual bone model in the software, the virtual implants can be used to assess intra- and post-operative kinematics.

[0054] With reference to Fig. 10, a more detailed description of the technique utilized to establish a registration relationship between the tracking unit and the AR system is disclosed. Registration of the IMU unit and the UWB unit can be achieved through a registration-matching motion, where the surgeon / user rotates the tracker in a known orientation, or the unit can be rotated in any orientation until sufficient information is collected, or by robotics or mechatronics, and registration software running on a central computer can determine a transformation so that the UWB and IMU coordinates match each other. Registration of the UWB and the AR system may be achieved by either registering the AR coordinate system to the UWB coordinate system, or vice versa.

[0055] In a first exemplary method for registering a tracking unit with an AR system, it is assumed that the AR system is mobile and that the camera associated with the AR system can be freely positioned. As part of the registration process, each camera of the AR system can capture a single or multiple registration targets, which can be images or objects, provided that the position of each camera has been previously measured in UWB coordinates. The registration of the UWB unit and the AR system can be completed by matching the coordinates of the registration targets. In an alternative exemplary embodiment, an object such as a stylus that can be recognized by the AR system can incorporate a UWB unit and / or an IMU unit, and a user may arbitrarily manipulate a pointer in front of the camera of the AR system, or the user may use the stylus to capture an activity with the camera of the AR system while collecting points from a known object, or the pointer can be manipulated via robotics or mechatronics while the activity is captured by the camera of the AR system. The position of the stylus, which can be represented as a 3D point or 3D point cloud collected via the AR system and the UWB unit during these activities, can be used together to determine the transformation between the coordinates of the two tracking systems. In another alternative exemplary embodiment, a device may have the AR system and the UWB and / or IMU units tightly attached together. A user may manipulate or walk around with the device, or the device may be manipulated via robotics or mechatronics such that the position of the device can be established and represented as a 3D point or 3D point cloud, or the like, and the transformation between the coordinates of the two tracking systems can be determined.

[0056] In a first exemplary method for registering a tracking unit with an AR system, it is assumed that the AR system is non-directional and that each camera of the AR system is fixed. In this exemplary method, an object such as a stylus that is recognizable by the AR system may incorporate a UWB unit and / or an IMU unit, and a user may arbitrarily manipulate the stylus in front of the camera of the AR system, or the user may use the stylus to capture an activity with the camera of the AR system while collecting points from a known object, or the stylus may be manipulated via robotics or mechatronics while the activity is captured by the camera of the AR system. The position of the stylus, which may be represented as a 3D point or 3D point cloud collected via the AR system and the UWB unit and / or IMU unit during these activities, may be used together to determine the transformation between the coordinates of the two tracking systems. In another alternative exemplary embodiment, the stylus may not be recognizable by the AR system providing the second image, or the object may be recognizable. In this case, the user may "paint" or collect specific registration points on the image or object. The position of the stylus, which can be represented as a 3D point or a 3D point cloud collected via the AR system and the UWB unit and / or IMU unit during these activities, can be used together to determine the transformation between the coordinates of the two tracking systems.

[0057] Finally, registration between the IMU unit and the AR system may be achieved by embedding the IMU unit in an object recognizable by the AR system. A user may arbitrarily manipulate an object in front of the camera of the AR system, or the user may be asked to position the object in different orientations while the activity is captured by the camera of the AR system, or the object may be manipulated via robotics or mechatronics while the activity is captured by the camera of the AR system. The object orientations collected during these activities via the AR system and the IMU unit, which may be expressed as any orientation representation such as Euler angles, direction cosine matrices, or quaternions, may be used together to determine the transformation between the coordinates of the two tracking systems.

[0058] Returning to FIG. 6, a more detailed description of step 6 of collaborative surgical planning is provided herein below. Collaborative surgical planning allows for simultaneous sharing and communication of holographic applications in terms of creating pre-operative surgical plans with other users who may be in the same room or working remotely, when the application is running on each device to view the augmented reality content. The application for collaborative surgical planning may work on multiple devices, such as handheld tablets or mobile phones, head-mounted augmented reality devices, laptops, or desktops. A collaborative framework based on network communication technology allows for support of multiple platforms on different devices connected to the Internet or local networks. One exemplary embodiment of a collaborative process for performing surgical planning may be built on a server and client architecture. The server component as a back-end computer may manage tracking connected and disconnected clients and synchronize messages between clients. For real-time communication with all devices connected to the server, an exemplary communication protocol that may be used is the user datagram protocol (UDP), which relays all messages between the devices. In this exemplary embodiment, message packets consisting of several bytes with a specific format may contain control information and user data (i.e., specific number, position, orientation, input, etc.). As clients, applications running on each device communicate tasks for surgical planning by relaying informative messages between other users over the synchronized network to seamlessly render real-time positioned visualizations. Tasks in the context of surgical applications may include location of reference points on target bones, bone cuts during a typical total joint replacement, implant size and location, implant alignment, etc.

[0059] As shown in Fig. 11 and Fig. 12, the task of bone cutting is designed to identify the desired resection plane associated with the target bone and to fit the implant to its unique position and orientation with respect to the bone structure. To improve communication with other users, the exemplary embodiment allows tracking the position of other users to display a 3D avatar model representing the user's positioning relative to the anatomical model as an anchor. According to the aspect of communicating directly with other users, a group voice chat function may be supported. To improve computational performance, the graphical models for the positioned visualization in the considered task may simply be managed and rendered on each device and then updated by receiving messages over the network with any changes in the current field of view.

[0060] 13, the conversion of different coordinate systems in AR is indeed important. In the AR system according to the present disclosure, the real world is represented by a scene frame F s and camera frame F c The real world including the scene and the virtual scene frame F vs and the virtual camera frame F vc Without loss of generality, F s and F vs are equivalent, i.e., F s =F vs In order to accurately project a virtual object into the current world frame, we assume that F c and F vc The cameras must be located at the same position and share the same camera parameters (e.g., focal length, field of view, skew, etc.). In order to accurately view virtual models such as virtual guides and medical tools superimposed on the real environment, F c and F vc and are precisely aligned before the synthesis process that creates the augmented reality display. c About the World Frame F sIt is necessary to estimate the unknown parameters of the transformation E, which defines the pose in the world frame. In the process of camera pose estimation in the world frame (e.g., registration), inaccurate or unstable pose estimation (position and orientation) can cause an overall negative impact on the performance (in the case of surgery, with respect to the surgical procedure itself). In the following sections, we provide a comprehensive description of several developed techniques for accurately inserting virtual objects, such as virtual guides, into a real-world environment.

[0061] In the context of an AR system utilizing a camera and a projector, an exemplary process for calibrating a camera to a projector is described herein below. This exemplary process involves approximating a camera model based on a pinhole camera model, which can be defined by a 3×4 three-dimensional perspective projection matrix. The two-dimensional projected pixels are expressed as m=[u,v]. T and the three-dimensional point is M=[X, Y, Z] T Then, the following corresponding points in homogeneous coordinates are

[0062]

number

[0063] teeth,

[0064]

number

[0065] and

[0066]

number

[0067] The 3×4 transformation matrix E between the world coordinate system and the camera coordinate system is defined as follows:

[0068]

number

[0069] where R and t are the rotation and translation matrices, respectively. E is also called the extrinsic that defines the position of the camera P. The relationship between a 3D point M and its image projection pixel m is given by:

[0070]

number

[0071] where s is an arbitrary scale factor and K is the focal length in pixels f x and f y , the coordinates of the cardinal points (u0, v0), and the skew parameter γ, which is set to zero here since modern manufacturing techniques for most conventional cameras make it negligible. The 3×4 camera projection matrix, which mixes both intrinsic and extrinsic parameters, is

[0072]

number

[0073] The concentrated camera projection matrix is ​​given by

[0074]

number

[0075] Using this, equation (2) can be redefined as follows:

[0076]

number

[0077] To account for computing depth values, a 4x4 camera projection matrix can be used by adding one row to the 3x4 camera projection matrix.

[0078] As a preliminary step in the intraoperative workflow when using an AR system, a camera (or multiple cameras) is incorporated into a head-mounted or handheld (or body-attached) display device to be calibrated. In general, camera calibration is utilized to find the intrinsics (also referred to as internal parameters) of the camera. In this routine, the calibration targets the camera at a known structure with many individual and identifiable points. By viewing this structure from different angles, the relative position and orientation of the camera for each image as well as the intrinsics of the camera can be calculated. To provide multiple views, a diverse set of images at different orientations and distances is acquired. Specific to head-mounted or handheld AR systems, multiple views of a known planar object (e.g., a chessboard) are acquired with the display device rotating and moving back and forth (instead of moving a target). The detected points on the known planar object are then used to calculate the homography matrix, which is associated with calculating both the individual translation and rotation for each view, and the intrinsic parameters, where intrinsics are defined as the set of parameters that are the same for all views. The camera can be calibrated by refining the intrinsic and extrinsic parameters. The intrinsic parameters calibrated herein can be stored as metadata on a server, computer, smartphone, or tablet. In a special case, when an optical see-through head mounted display provides the availability of access to the position of the camera in world coordinates and the perspective projection of the camera when the photo or video is taken, all the steps of camera calibration can be ignored. In an exemplary embodiment, the perspective projection matrix of the projector model based on the pinhole projector model is defined by the same set of parameters as the pinhole camera model. Therefore, the perspective projection matrix of the projector model can be found by performing the same procedure as the camera calibration.

[0079] In accordance with this disclosure, the following description provides a process and structure for solving the open problem of pose estimation, which estimates the pose of a three-dimensional object from a set of projections of two-dimensional points. The challenge of pose estimation is also known as estimating the extrinsic parameters of the camera. The pose of the camera can be recovered by estimating a homography, which is an invertible mapping of a set of points on the perspective plane. The application of pose estimation can provide the ability to simultaneously estimate both three-dimensional geometry and camera pose from stored objects as references and objects in the current scene as targets. In addition, a UWB unit and / or an IMU unit embedded as part of an AR system can enable the localization of a surgeon in an operating room to be initially identified when applied to medical procedures. As an example, the position of a static reference in the real world as well as any other geometric shape of the reference is predetermined to align an external tracking coordinate system to the AR coordinate system in the rendering framework.

[0080] As discussed in more detail herein below, various exemplary techniques can be utilized to estimate the camera pose, including, but not limited to, two-dimensional marker detection, two-dimensional feature matching, and three-dimensional model-based object detection. These exemplary techniques can also be used to track the position and orientation of a target object or reference in an AR application in which a virtual guide or annotation is overlaid on the surface of a target surface through an AR display.

[0081] As shown in FIG. 14, a block diagram illustrates an exemplary process for a rectangular shape-based marker detection approach that converts real-world and virtual-world coordinate systems. Given a color input image, one exemplary process converts the input image to a black and white color image. After applying basic image processing algorithms to remove noise, the process detects lines or edges in the binarized image to fit the preferred shape of the marker to each cluster of unordered boundary points. The process then finds four candidate corners to fit lines to each side of the marker. As part of this process, some additional filtering is performed to remove undesired corners on obvious non-marker regions. To identify individual markers, the process decodes the detected code in each marker. As the final step of the process, an estimate of the camera pose is generated to render any virtual model of the corresponding target on the surface of the target plane along with the user's real-world view.

[0082] Alternatively, a circular shape-based marker detection approach can be utilized to transform the real-world and virtual-world coordinate systems. Depending on the marker detection algorithm, a different marker is tagged to each of the references. The markers can be designed as circular shapes with unique textures consisting of a combination of binary color patterns. The textures in the markers are used to identify the markers. At the core of marker detection, fiducials on the markers (e.g., edges) are used to estimate the camera pose through multiple frames in different fields of view. Specifically, an exemplary method of marker detection can accomplish some or all of the following process steps: (1) binarizing the color input image, (2) removing non-markers, (3) decoding the detected code in the marker to identify the marker, and (4) estimating the camera pose. As part of binarizing the color input image, an adaptive thresholding approach can be used that uses minimum and maximum values ​​in tiles of a grayscale image that is converted from the color input image and then divided into tiles of a certain dimension in pixels.

[0083] Referring to FIG. 15, a block diagram illustrates an exemplary process for a two-dimensional feature matching technique that transforms real-world and virtual-world coordinate systems. Correspondence between features (e.g., corners, curves, lines, edges, or regions) in the reference and target images is established to estimate a homography matrix that maps from one image to another. Homography is also known as a perspective transformation that operates on homogeneous coordinates. In both input images, the exemplary process detects candidate features among all pixels in the target image. The process then describes each candidate feature with local information around the location of the feature, and then the process encodes the local information of each detected feature into a vector known as a feature descriptor. Once all features are described, the process searches for feature descriptors that are consistent at multiple scales, and then the process estimates the camera pose by aligning the parameters of the homography matrix. By way of example, once the feature descriptors in the reference image are defined, these feature descriptors can be embedded or stored as text or table files in an embodiment in a server, computer, smartphone, or tablet to improve the overall processing time of the feature matching technique.

[0084] Turning to FIG. 16, a block diagram shows an exemplary process for a three-dimensional model-based object detection technique that converts real-world and virtual-world coordinate systems. The three-dimensional model-based object detection technique process deals with a challenging set of non-textured objects (e.g., metal or output from a three-dimensional printer or rapid prototyping machine). To address these challenging issues in non-textured objects, this exemplary process may first generate edge templates as pre-processing from a polygon mesh model of the non-textured target object as the appearance of the model changes for rotational variations in x, y, and z axes. Then, the process rasterizes each of the edge templates in different fields of view. Given a color input image, the process performs basic image processing algorithms to convert the color input image to a grayscale image. Then, the process detects all candidate edges or lines on the grayscale image and projects a set of sample rater points resulting from rasterizing the edge templates onto the edges or lines on the image plane. Once the projection is complete, the process searches in a multi-scale for an edge template that matches with the minimum distance between the edge template and the edge or line detected in the input image. As a final step, the process estimates the homography to accurately position the model on the image plane. All edge templates and sets of sample raster points can be embedded or stored as text or table files in a server, computer, smartphone, or tablet, in embodiments.

[0085] Turning to Fig. 17, we show an example of a rendering framework in a basic forward mapping process. The example process includes two rendering steps. In the first step, a ray TV is described with a user T using two parameters on an arbitrary projection plane of a frustum of a perspective view. It concerns computing an image of a virtual object V from a user position T on that projection plane. The projection is represented by a projection matrix P_T where m_T on the image plane Π_plane is the projection pixel. An intermediate point M on the display frame D needs to transfer pixel m_T to pixel m_p. After transferring pixel m_T along the ray Tm_T to the nearest point on the display frame D, the image of V transferred from Π_plane through the projection center T is extended on the display frame D. In the second step, the image of the extended display is found by transferring point M to pixel m_P via the camera or projector projection matrix.

[0086] The following description describes processes and techniques for rendering a three-dimensional (3D) model as part of an AR display. To render the 3D model as part of a target in the virtual world coordinates of the AR display, a geometric framework based on a pinhole camera or projector model can be considered. Various structures of each component of this exemplary process for rendering an image of a three-dimensional virtual object include: (1) a user T may be moving or stationary; (2) a projector or camera P for creating a computer-generated image on the display may be embedded in a head-mounted or handheld (or body-attached) display device or at any position in the surrounding environment; (3) a display surface D may be planar or non-planar. The geometric framework is based on a user T, a camera P, a virtual world frame F, and a virtual world frame F. vs , and the real-world frame F s The geometric relationship between the frame F vs and F sis simplified as the display frame D in this specification. Such geometric relationships between geometric components are for representing the mapping from the three-dimensional virtual object space to the two-dimensional camera image space for any three-dimensional point V on the virtual object using an intermediate point on the display frame D, and the projected pixel m of the virtual point V p is the ray P C intersects the camera image plane P plane by M, where P c is the center of the camera and M is the intersection of the ray TV and the display frame D.

[0087] When TM = kTV, for the user T, the following camera projection matrix

[0088]

Number

[0089] and the relationship between the virtual point V and its projected pixel m with respect to geometric components such as the display frame D p is defined as follows,

[0090]

Number

[0091] where the following operating factors

[0092]

Number

[0093] indicates the intersection of the ray TV and the display frame D, and ~ indicates equivalence up to scale. When k > 0, it guarantees that the display and the virtual object are on the same side as T. The condition 0 < k ≤ 1 indicates that the virtual object is behind the display frame for the user, and when k > 1, the virtual object is in front of the display frame.

[0094] According to a preferred embodiment of the basic forward or backward mapping, the order of the two subsequent steps can be reversed. In an exemplary form, the steps can be valid for any structure of an AR application. In a first step, a ray TV is represented with the user T using two parameters on an arbitrary projection plane of a frustum of a perspective view. This corresponds to calculating an image of the virtual object V from the user position T on that projection plane. The projection is performed in the form of an image II. plane The projection matrix P on T and m T and denotes the projected pixel. The midpoint M on the display frame is pixel m T m p The light ray Tm T to the closest point on the display frame D along pixel m T When transmitted, II plane The image of V, transferred from the camera to the projection center T, is extended onto the display frame D. In the second step, the point M is projected onto pixel m using the camera projection matrix, which defines the intrinsic and extrinsic parameters of the camera in equation (3). P In three-dimensional computer graphics, the image on the extended display is found by transmitting the P and P P Two transformation matrices such as may be required to define pixel projection, visibility, and view frustum behavior. In this exemplary process, the application-specific coordinate system and both transformation matrices can be embedded as metadata, allowing for locating images captured in the real world.

[0095] As shown in Fig. 20, in an exemplary embodiment for rendering a 3D virtual model on an optical see-through display integrated into a head-mounted AR device, reference points extracted from a target tracker, marker detection, feature matching, 3D model-based object detection, an external tracking system, or any combination of these can be used. By using backward mapping, the following optimal points can be obtained:

[0096]

number

[0097] The pose estimation process to approximate ,may accomplish some or all of the following steps:

[0098]

number

[0099] Let P be the camera image plane. plane It can be seen above that the reference pixel f i where i and k are the indices of the reference pixel and the captured image in different fields of view, respectively. (2) Find the projection pixel

[0100]

number

[0101] , point M i,k is ray T k V i A camera projection model P intersecting with P By projecting the vertices of the three-dimensional model onto the display frame D via i,k (3) Compute an image of user position T to V as described in basic forward mapping;

[0102]

number

[0103] By using the first step, in terms of updating the positions of multiple rays T k V i and (4) repeating the previous step until one of the following conditions is met: (i) the approximate intersection of the multiple rays is no longer updated, (ii) a maximum number of iterations is reached, or (iii) a convergence tolerance is reached. In this process, the projection pixel m i,k may be normalized in the transformed image coordinates. During the alignment process,

[0104]

number

[0105] The virtual model formed by the method may visually indicate the current position of the target object as well as the alignment quality to the user in real time.

[0106] The following discussion provides an example solution for approximating the intersection of multiple rays. Considering a point to be two-dimensional p=(x,y) or three-dimensional p=(x,y,z), the unit vector u is the vector of cosines of vector v with respect to each of the following coordinate axes:

[0107]

number

[0108] In the formula, e j is the unit vector of the jth coordinate axis. The unit vector u gives the direction of the vector v. For a line, the parametric equation can be given by: p(t)=tu+p0, (6) where p0 represents a point passing through the line -∞≦t≦∞. If the domain of t is limited to within 0≦t≦∞, then a ray (e.g., a half-open line) may be denoted as a vector starting at point p0 and point of direction u. Assuming that v1=p1-p0 and v2=p2-p0, the three non-collinear points p0, p1, and p2 may define two separate vectors v1 and v2. Since the two vectors lie on a plane in space, the normal to the plane may be defined as n = v1 × v2. (7) where × is the cross product of vectors v1 and v2. For a plane, the set of points p orthogonal to n is given by: n (p-p0)=0, (8) where a point p0 in the plane is offset from the origin. To calculate the shortest distance from a point to a line, the squared perpendicular distance technique can be used. The squared perpendicular distance D can be given by:

[0109]

number

[0110] where l is the identity matrix, (lu T u) is (lu T u) 2 =(lu T u) T (lu T u)=(lu T u). Considering multiple R rays, we can find a unique solution in least squares for the intersection of the set of R that minimizes the sum of the squared distances.

[0111]

number

[0112] For a quadratic form in p, the least squares distance objective can be defined as

[0113]

number

[0114] To find the minimum of the sum of squared distances, equation (10) is differentiated with respect to p as follows:

[0115]

number

[0116] therefore,

[0117]

number

[0118] where † is the Moore-Penrose pseudoinverse. An exemplary method for estimating a minimal set of R rays that eliminates a substantial proportion of outliers may include some or all of the following steps: (1) Randomly select two rays to find an initial point p with the shortest distance between them. s (2)p s andr R Each ray in l Calculate the distance between the number of rays τ current Count the numbers and fit them to a predefined tolerance ε to get p s and r l Observe normal values ​​at an acceptable distance from (4) τ max <τ current If so, the nearest point p closest =p s and the maximum number of rays τ max = τ current (5) Repeat steps 1 to 5 until the maximum number of iterations is reached (5 iterations). And (7) In equation (13),

[0119]

number

[0120] to the nearest point p closest and ray r l The estimate is made using all identified inliers whose distance between is less than ε.

[0121] The present disclosure also provides a solution to the problem of absolute orientation when aligning two different coordinate systems in the context of using an AR system. The two different coordinate systems refer to a real world coordinate system and a virtual or augmented coordinate system. The following exemplary process can be used to solve the problem of absolute orientation between an aligned reference point in the virtual world coordinate system and a corresponding predefined reference point in the real world coordinate system.

[0122] A rigid transformation between two points p1 and p2 in three dimensions that map to different coordinate systems can be defined as p2=Rp1+p0, where R is a 3×3 orthonormal matrix of rotations and p0 is a vector of translations.

[0123]

number

[0124] and p a =(x a , y a , z a ), which denote the coordinates of the aligned reference point in virtual world coordinates and the coordinates of a predefined point in real world coordinates, respectively, the input for the absolute orientation problem is the conjugate pair

[0125]

number

[0126] To solve the absolute orientation problem, the following points are

[0127]

number

[0128] andp a The rigid body transformation matrix between can be refined by using R and t in equation (14) below:

[0129]

number

[0130] where there are 12 unknown parameters (i.e., 9 elements of the rotation matrix and 3 elements of the translation matrix). Since each conjugate pair generates 3 equations, at least 4 conjugate pairs are needed to find the 12 unknown parameters. In this embodiment, more than 4 conjugate pairs can be used to achieve better accuracy of the results. In simple matrix form, the rigid body transformation that transforms the coordinates of each point in the virtual world coordinates to real world coordinates can be defined as follows:

[0131]

number

[0132] where R(q) is the unit quaternion q and

[0133]

number

[0134] and the location of the origin of the aligned reference point in the world coordinate system. The unit quaternion q is

[0135]

number

[0136] is a vector defined as:

[0137]

number

[0138] and Pa = {p a,1 , p a,2 , ..., p a,n Considering each of the points {\displaystyle \mathbb {R}\right in terms of their aligned references in virtual world coordinates and their corresponding predefined references in real world coordinates, the problem of absolute orientation is to align these two sets of points in space, as shown in Figure 10.

[0139]

number

[0140] and P a The geometric center of the point cloud with respect to is first calculated as follows:

[0141]

number

[0142] Each point is then subtracted from the geometric center.

[0143]

number

[0144] The problem in determining the rotation involves aligning two sets of rays with respect to a geometric center in ray coordinates where the rays are derived from a set of conjugate pairs. The best alignment is approximated in a least-squares sense by calculating the rotation R(q) to be maximized with respect to the scalar product of each ray pair.

[0145]

number

[0146] Using quaternion notation, equation (18) can be written as follows:

[0147]

number

[0148] By using quadratic form notation, equation (19) can be refined so that

[0149]

number

[0150] where q is a column vector. Using the eigenvector for the most positive eigenvalue, the unit quaternion can be maximized in this quadratic form.

[0151]

number

[0152] and r a,i =(x a,i , y a,i , z a,i ), the matrix

[0153]

number

[0154] and N a,i is given by:

[0155]

number

[0156] To simplify the matrix N, the sum over all conjugate pairs of the products of coordinates k and l at the aligned points and the corresponding predefined points, respectively, may be defined as:

[0157]

number

[0158] α kl Using the above, the matrix N is given by:

[0159]

number

[0160] These calculations result in a unit quaternion to represent the rotation that aligns the ray bundle. From the elements of the unit quaternion, the rotation matrix can be found as follows:

[0161]

number

[0162] Once the rotation is determined, the translation part of the transformation in equation (15) can be given by:

[0163]

number

[0164] With reference to FIG. 18, an example of aligning two different coordinate systems using two sets of rays with absolute orientation is shown. For example, considering two sets of points P_M^={p_(M^,1), p_(M^,2), ..., p_(M^,n)} and P_a={p_(a,1), p_(a,2), ..., p_(a,n)}, respectively, in aligned references in virtual world coordinates and corresponding predefined references in real world coordinates, the problem in determining the rotation involves aligning the two sets of rays with respect to the geometric center in ray coordinates where the rays are taken from the set of conjugate pairs {(p_(M^,1), p_(a,1)), ..., (p_(M^,n), p_(a,n))}. The best alignment in the least squares method is approximated by calculating the rotation to be maximum with respect to the scalar product of each ray pair. Once the rotation is determined, the translation portion of the transformation can be estimated by calculating the location of the origin of the aligned reference point in the world coordinate system.

[0165] Turning to FIG. 19, a block diagram of one exemplary process for tracking bones using a UWB and IMU hybrid tracking system in real time is shown. A patient can be tagged with multiple "units" fixed to the patient's anatomy. An embodiment may first register the units from the external tracking system to a virtual world coordinate system in which the coordinates of the units of the external tracking system are predefined. In the virtual world coordinate system, a computer-generated model or image is rendered and overlaid on the surface of the patient's anatomy via a display device (e.g., projector, head-up display, etc.). After registration, the external tracking system may continuously estimate and track the position and orientation of the units in real time and send the estimated coordinates of the target units to a software application of the system that updates and renders the augmented information along with the coordinates of the target units.

[0166] 20 illustrates an exemplary user device of an AR system according to the present disclosure. By way of example, the user device includes a helmet with an integrated camera and a head-up display shown on the visor. The exemplary user device also includes an IMU unit mounted on top of the helmet and a series of UWB units. Although not shown, an on-board battery is included to power the camera, UWB unit, IMU unit, and head-up display.

[0167] FIG. 21 shows an alternative exemplary structure to that which may be used as part of an exemplary AR system. More specifically, the structure includes multiple laser projectors associated with color cameras. Each camera and laser projector is also associated with an IMU unit and a UWB unit. Using information from the IMU unit and the UWB unit, the AR system knows the relative positions of the cameras and laser projectors and updates the image displayed by the laser projectors taking into account the positions of the laser projectors and the IMU unit and the UWB unit mounted on the patient's anatomy.

[0168] FIG. 22 shows an example structure incorporating an IMU unit and a UWB unit along with a camera. In this way, the structure of FIG. 22 allows for the integration of optical tracking into the positioning algorithm. This integration can help refine accuracy, detect undesired motion of the anchor, and speed up the positioning algorithm.

[0169] With reference to FIG. 23, a block diagram of an exemplary process for registering a head-mounted AR system incorporating a feature matching approach and a hybrid tracking system is provided. In this exemplary process, features can be represented as corners, curves, lines, edges, or regions on a target image. From feature matching, a match between a set of features in both a previous image and a current image can be established to estimate a homography or geometric transformation. If at least four matches are found in an image plane, a homography can be estimated by a linear equation. To improve the accuracy of high-precision registration and tracking of the user and patient's anatomical structures during motion in real-time processing, an IMU unit and a UWB unit are integrated into the head-mounted AR system to view augmented information in a display device.

[0170] With reference to FIG. 24, a block diagram of an exemplary process for registering a laser projector-based AR system is provided. A known structure with many individual and identifiable points can be used to calibrate each camera. By viewing this structure from different angles, the relative position and orientation of the camera with respect to each image as well as the intrinsics of the camera can be calculated. After finding the intrinsic parameters of the camera, a robust multi-view tracking approach with multiple cameras can be performed to provide robustness against occlusion. Tracking a moving target in an operating room covered by multiple calibrated cameras with overlapping fields of view can be established by finding a match between the different cameras. For this, an exemplary process for tracking a moving target can use a target tracker including marker detection, feature matching, 3D model-based object detection, a hybrid tracking system, or a combination thereof. By combining the target tracker with the hybrid tracking system, the location of the patient's anatomical structure can be provided by a hybrid tracking system using IMU and UWB technology. A solution to the potential problem of resolution can be found by localizing the surgeon's motion and searching for the best laser projector among multiple laser projectors placed around the operating room.

[0171] Turning to FIG. 25, an exemplary integration of an AR system with a UWB / IMU system is graphically illustrated. The AR camera may consist of a computer vision tracking system whose viewing volume is indicated by a dotted line. The AR camera may include at least one of a UWB unit and an IMU unit (identified as "Tracker" in FIG. 25) to allow the position and orientation of the AR camera to be known as well as to register this position and orientation to global coordinates. In this illustration, AR camera 1 would be able to track tracker C within its viewing volume. Tracker C may also transmit its position and orientation to the AR system, and a sensor fusion algorithm may be used with the AR vision tracking system to improve tracking accuracy. Tracker B is within the tracking volume of AR camera 2, but is invisible to AR camera 1. AR camera 2 may track the position and orientation of tracker B and transmit that data wirelessly to AR camera 1. Alternatively, tracker B may transmit its position and orientation directly to AR camera 1. Tracker A may be tracked even if it is located outside the viewing volumes of both AR cameras. The position and orientation of tracker A can be tracked by both its own UWB and IMU units and transmit its position and orientation data to both AR cameras. It is understood that the positioning system can use a UWB / IMU tracker instead because the AR system can be limited to tracking multiple objects within its viewing volume. The use of a UWB / IMU unit / tracker can reduce the computational load required for the computer vision of the AR system, thereby easing the computational burden and improving the performance of the AR system.

[0172] The AR system may communicate with the tracker locally as part of the helmet if the AR system is local to the helmet, but it is also within the scope of this disclosure to have an AR CPU running an AR software application located remotely from the tracker. In such cases, the AR system needs to communicate with the tracker via a wired or wireless network (or a combination of wired and wireless networks). In an exemplary form, the AR software application runs on a server communicatively coupled to the tracker via a private network. The wireless tracker should be suitable to manage the protocol between the server and the AR application. An exemplary method for the protocol may provide all of the following functions or components (1) data sequencing, (2) data routing, (3) flow control, and (4) error control to efficiently handle errors during data transmission over the network. When the tracker is configured to connect to a server running an AR application that creates a three-dimensional model of a virtual object on a display, the server may continuously provide position and orientation data from the tracker to the AR application. As an example, the tracker may continuously and simultaneously track multiple external surgical tools associated with one or more surgical procedures, as well as the anatomical structures of the user and the patient. The AR application uses the estimated coordinates of the tracked targets and the transformation of Equation (15) to render a three-dimensional virtual model that is overlaid on the corresponding real-world image visible to the user.

[0173] As mentioned above, the tracker may track the motion of a user (e.g., a surgeon) of the AR system. In an exemplary form, the tracker may include one of an IMU sensor, a UWB sensor, a depth measurement sensor (e.g., a range camera, a binocular stereo camera, or a time-of-flight camera), an imaging sensor, a relative measurement sensor (e.g., a three-axis gyroscope, an accelerometer, and a magnetometer), or any combination thereof. As discussed above, the ability to accurately track the user's motion while wearing the AR vision system is important to ensure that the displayed virtual image is properly aligned with the real-world image seen by the user. By way of example, the exemplary AR system may be incorporated as part of a helmet with an optical see-through display or as part of a mobile device screen within the user's field of view (FOV). High-precision registration of the AR system may be achieved by appropriately tracking the user's relative motion, preferably in six degrees of freedom (DOF), (1) three variables of position (x, y, and z), and (2) three angles of orientation (yaw, pitch, and roll).

[0174] As discussed above, the AR system operates to display a virtual image on the surface of a real-world image. As would be envisioned, the usefulness of the virtual image is predicated on registering the virtual image to the real-world image. By way of example, the virtual image may comprise one or more images of a patient's bone(s). Thus, the following description concerns the registration of a virtual patient bone model to an actual patient's bone.

[0175] Each tracked object may be integrated with an external tracking system (tracker) based on IMU and UWB technologies (see Figures 2-4 and the above description of these figures) that is rigidly fixed to the object. To register the virtual bone model to the real patient's bone, (1) a mixture of IMU and UWB technologies may be used to "paint" (select multiple points on the surface of the real bone) and register the selected points to a virtual bone model reconstructed from medical images; (2) a patient-specific registration device may be used to perform registration where the orientation of the device relative to the patient's anatomy is known; and (3) specific reference landmarks may be selected that correspond to landmarks on the virtual bone model.

[0176] Once the virtual bone model is registered to the actual patient's bones by using an external tracking system, the head-mounted AR system of FIG. 20 can be registered by combining the anatomical position estimated by the hybrid tracking system with sensors such as its IMU sensor, UWB sensor, depth sensor, and imaging sensor. To accurately register the AR system to the real-world field of view, a target tracker with a single camera and IMU sensor (e.g., marker detection, feature matching, 3D model-based object detection, or a combination thereof) can be used to determine the surgeon's position and orientation in the operating room and to identify the target. The target tracker-based registration described herein may be replaced with a hybrid tracking system directly integrated or built into the head-mounted or handheld (or body-attached) system.

[0177] As shown in FIG. 21, in another exemplary embodiment of the AR system without a head-mounted or handheld display device, which may consist of multiple cameras, laser projectors, and IMU and UWB technologies, a simultaneous multiple view tracking approach can be performed to handle occlusion under poor lighting, static or moving ambient light sources, or unconstrained indoor environments. There may be two reasons to facilitate multiple cameras: (a) a single camera is limited to provide adequate coverage of the target environment due to its limited field of view; (b) multiple cameras may provide robustness against occlusion. The multiple view tracking approach described herein may be applied to estimate and identify the location and orientation of the target object to overlay a virtual guide / image on the surface of the actual patient's bone in the operating room along with the user's real-world field of view. The problem of consistently and accurately tracking a moving target in the operating room covered by multiple calibrated cameras with overlapping fields of view may be addressed by establishing correspondence between trackers of the same object in different cameras. The correspondence may be to recover the a priori information of the target object. To find matches between different cameras, the target trackers described herein can be used, including marker detection, feature matching, 3D model-based object detection, external tracking systems, or a combination of these. Thus, potential problems in occlusion can be solved by localizing the surgeon's motion in real time and finding the optimal laser projector among multiple laser projectors placed around the operating room.

[0178] After the considered AR system is registered, the rendering module uses the estimated pose to combine the input image taken by the camera sensor in the AR system with the virtual guide, and then renders the augmented image on the display. Then, all the cuts for the surgical procedure are visually assisted by superimposing the proposed cut position and orientation on the anatomical structure. Additionally, the surgical instruments used for cutting can be tracked, and the exact orientation and position of the cut is updated in the AR overlay in real time.

[0179] During surgery, the ability to track the patient's anatomical position and the surgeon's motion from many different sensors in the hybrid tracking system and AR system can be fed back to the localization of the virtual guide to accurately superimpose the virtual scene on the surgeon's viewpoint. In addition, to account for potential registration errors caused by occlusion and tracking impairments in the AR system, trackers (e.g., IMU units and / or UWB units) attached to the patient's bones can be used to track the patient's anatomical position and a central unit as a global reference in the hybrid tracking system. The tracking application software running on the CPU of the AR system can wirelessly communicate with the trackers / units via private network updates so that the trackers provide data to identify the user / surgeon's position in real time.

[0180] After completing the necessary cuts as part of an exemplary surgical procedure, for example, an AR system can be used to evaluate intraoperative kinematics in the following aspects: (1) range of motion, (2) relaxation in extension and flexion (implant translation), (3) ligament distance, and (4) any other clinically relevant kinematic information. Evaluation of intraoperative kinematics with the AR system can be simulated by (1) registering the cutting instrument to the hybrid tracking system once preoperatively and then tracking the instrument to estimate the position and orientation of the cut during surgery, (2) using object recognition techniques with a tracker / unit attached to the patient's anatomy to estimate the geometric transformation of the implanted patient's implant and then accurately superimposing the virtual implant on the resected bone during surgery, and (3) assuming the cut was performed as planned. For postoperative kinematic evaluation, IMU and UWB technologies can be used to track body motion and analyze muscle activity. AR system techniques combined with a hybrid tracking system can also enable visualization of postoperative kinematics at the point of care.

[0181] Turning to FIG. 26, a block diagram of an exemplary process for simulating and evaluating intraoperative kinematics is shown. This exemplary process assumes that the cutting is properly performed in the operating room as planned for joint replacement surgery. The units designed for the hybrid tracking system can be rigidly fixed at any position of the patient's anatomy. To track the position and orientation of the surgical tool, such as the cutting tool, in real-time processing, multiple sensors that communicate wirelessly to the hybrid tracking system are attached to the target tool. The aforementioned exemplary process for detecting and localizing non-textured objects using 3D models can be used to track and identify the patient's implants, which are generally metallic. In this process, features should be indicated by lines or edges. All edge templates and a set of sample rater points loaded in the process of matching features can be embedded or stored as text or table files in a server, computer, smartphone, or tablet. After estimating the poses of all targets, including the cutting tool, the patient's implant, and the patient's anatomy, a rendering of the cutting combined with the virtual components constructed from the corresponding target virtual model and pre-planning can be performed to view the augmented information on a display device in the augmented reality system undertaken. The aforementioned process can simulate and evaluate intra- and post-operative kinematics via an augmented reality module.

[0182] Referring to FIG. 27, a block diagram of an exemplary process for providing feedback to a surgical tool as part of surgical navigation according to the present disclosure is shown. As an example, a CPU / processor collects position and orientation data from a UWB and IMU-enabled surgical tool. The integration of the IMU system, the UWB system, and the AR system is achieved using a software application program interface designed to convert messages from each system into a format readable by the processor. The processing results can be sent back to each system in the framework using the same software tools to convert the processor information into a format readable for each system. The patient's position can be tracked by a UWB and IMU unit / tracker that is rigidly attached to the patient's bone or the AR's vision system. The feedback to the surgical tool may be implemented as enabling or disallowing the connection of a battery to the electronics of the surgical tool. Alternatively or additionally, the feedback may be implemented as enabling haptic feedback (e.g., a haptic motor that creates a tumbling sensation, as discussed above). The feedback may also be implemented as a warning message displayed on a visual element within the AR helmet or surgical tool.

[0183] Turning to Figure 28, an exemplary illustration of a user-wearable device of the AR system is shown. The user-wearable device includes a graphical projector that operates to place virtual bone images and virtual components (e.g., pin guide locations) on the real-world patient anatomy. The user-wearable device may also display projected resection levels and virtual placement of implants on the real-world patient anatomy.

[0184] 29 shows a real-world view of a palpable bone model of a patient's pelvis (shown as "normal") and a second view showing the AR system of the present disclosure projecting a virtual image onto the bone model. As can be seen, the AR system aligns the image to the palpable bone model and includes an image of a virtual reamer shown reaming the acetabulum consistent with the pre-operative surgical plan.

[0185] FIG. 30 is a photograph showing the transfer of anatomical tracking from a custom registration device to rigidly fixed markers for AR surgical navigation.

[0186] 31, a photograph showing tracking of a surgical instrument relative to anatomy using AR to position a hip socket. In this exemplary photograph, visual feedback is provided to the user directly on a computer screen monitor adjacent to a tactile pelvis model, as well as by an AR head-up display as part of an AR helmet.

[0187] Referring to FIG. 32, a dual image representation shows, on the left hand side, the relative position of a user of an AR system according to the present disclosure in the context of a total hip replacement surgical procedure. The right hand representation is what the user would see via a head-up display on the visor of the AR helmet shown in the left hand representation. In this exemplary depiction, a surgeon is performing a total hip arthroplasty using a UWB unit and an IMU unit along with the AR system. The UWB / IMU stylus is an instrument that allows the surgeon to register the patient's bones to the remainder of the applicable tracking system. As discussed above, the surgeon may paint the bone surface or pick out specific anatomical landmarks with the stylus. The UWB / IMU surgical instrument represents a surgical instrument enabled by the UWB unit / IMU unit whose position and orientation are known to the surgical navigation and guidance software. The AR helmet may aid in localization and improve the accuracy of the tracking system, and the AR helmet may also relay information to the surgeon by superimposing virtual objects onto the real world image via the visor of the AR helmet.

[0188] FIG. 33 shows a surgical oscillating saw with a UWB tracking unit and an IMU tracking unit.

[0189] FIG. 34 shows a mechanically adjustable four-in-one cutting block (physical embodiment) that can be utilized for multiple sizes and implant groups according to the present disclosure. The mechanical aspects of this cutting guide may be combined with the programmable tracked aspects of the cutting block of FIG. 35. As an example, the cutting slot in this cutting block may remain closed until such time as the slot is in the correct position as determined by pre-operative planning via data output from the UWB and IMU tracking units attached to the cutting guide. Alternatively, the slot may be manually opened and closed.

[0190] Turning now to FIG. 35, an adjustable "smart" cutting block for performing femoral resections in total knee replacement surgery is shown. The cutting block is designed such that the four (or more) positions of the cutting block are directed by a motorized system that translates along two rails and rotates to match the orientation of the posterior cut, the anterior cut, and the chamfer cut. The instrument is wirelessly programmable via UWB, BLE, BlueTooth, or other communication protocols such that the implant size is stored on the device and the position of the cutting slot is directed by the cutting box of the implant size. Initial positioning of the device may be accomplished using custom metrology or by the use of so-called virtual guides. In an alternative exemplary embodiment, the motor for moving the cutting slot can be off-hardware and accomplished by attachment of an external motor or a standard surgical driver. Although the exemplary cutting block is described as showing at least four cutting positions, the actual position of the cutting slot is patient-specific and can consist of any number of cuts. The programmable aspect of the "smart" block allows for customization of the stop positions, such that each stop position is directed by a defined cutting plane.

[0191] Referring to Figure 36, an exemplary flow chart illustrates a process for utilizing the smart cutting guide of Figure 35, where the smart cutting guide may be actively tracked by a UWB / IMU / AR system. In an exemplary form, the cutting guide may be pre-programmed or may wirelessly receive the location of the planned cutting plane generated as part of pre-operative planning using a virtual model of the patient's actual anatomy. The cutting guide may be actively tracked by the UWB / IMU / AR system and in this manner automatically opens and closes the cutting slot such that the cutting blade may only penetrate the slot when the guide is in the correct position and orientation for cutting the bone.

[0192] FIG. 37 illustrates an alternative exemplary flow chart of a process for utilizing the smart cutting guide of FIG. 35, where the smart cutting guide may be actively tracked by a UWB / IMU / AR system. In an exemplary form, the cutting guide may be pre-programmed or may wirelessly receive the location of the planned cutting plane generated as part of the pre-operative planning using a virtual model of the patient's actual anatomy. The cutting guide may be actively tracked by the UWB / IMU / AR system and operates to automatically reposition the cutting block and open and close the cutting slot. As an example, a pre-operative bone resection plane includes a series of sequential bone cuts. The exemplary smart cutting guide of FIG. 35 is programmed to reposition the cutting block to each progressive bone cut and open the cutting slot only if the block is in the correct position and orientation for each particular bone cut.

[0193] 38, a visual depiction of an AR head-up display for positioning a real-world cutting guide relative to a real-world image of bones according to a pre-operative plan is shown. By aligning the cutting block with the augmented virtual aid, patient-specific procedures can be followed without the need for additional instrumentation. In this exemplary embodiment, a head-up display on the visor of the AR helmet provides feedback regarding deviations in the position of the real-world cutting guide relative to the real-world patient's bones.

[0194] Turning now to FIG. 39, a visual depiction of real-world cutting block / guide positioning using virtual guide pins visible via an AR head-up display is shown. In an exemplary form, the virtual pins define the appropriate location relative to the tracked location. Planned and tracked location information is rendered within the AR display and updated accordingly. In this case, the cutting block may be passive (no electronics) or may include communication and tracking systems, such that the location of the cutting block is known or unknown to the AR system. The cutting block may also include a mechanical stop that prevents cutting through the slots until such time as the guide is in the correct position, as determined by pre-operative planning or intra-operative preference.

[0195] Turning to FIG. 40, an exemplary oscillating saw is equipped with a UWB unit and an IMU unit for tracking within the AR system. FIG. 40 also shows the user's field of view through a head-up display associated with the visor of the AR helmet worn by the user. In the exemplary form, the head-up display shows the AR components of the user's field of view, while the visor is transparent to allow the user to visualize real-world elements such as the patient's bones and the tactile saw. The path of the saw and planned cut can be extended as part of the head-up display associated with the visor of the AR helmet to provide precise information on the cutting parameters. The position and orientation of the cutting blade as well as its path are updated in real time to the patient so that line of sight is never an issue.

[0196] Referring to FIG. 41, an exemplary visual representation of the visor of the AR system helmet during a knee resurfacing procedure is displayed. In an exemplary form, the visor head-up display shows the planned resurfacing position on top of the real-world image of the patient's bone that is not augmented. The resurfacing position is actively displayed to the user on the visor screen or via a holographic display that is visually distinct from the real-world image of the patient's bone. As part of the standard surgical technique for bone resurfacing, a motorized burr or router is used to remove cartilage and bone to create a surface ready for the attachment / attachment of an orthopedic implant. In an exemplary form, the motorized burr or router surgical instrument may be a connected instrument and may include or be configured with an IMU / UWB unit tracking device that communicates the instrument's orientation and position to the CPU or central navigation computer in real time. The CPU or central navigation computer also receives tracking data on the anatomical structure by a reference IMU / UWB device rigidly fixed to the bone or some aspect of visual tracking or a combination of these methods. This information is sufficient to determine where the surgical instrument is located relative to the patient. Once the deburr is moved within the area highlighted for removal via the AR display, the operator can deburr as normal. As an example, if the deburr position is detected outside of the defined area, a switch may prevent motor activation. This prevents accidental over-resection or inaccurate implant placement. In this way, a truly wireless non-gaze feedback system for smart instruments is applied to resurfacing procedures. The IMU / UWB system provides real-time feedback of the deburr orientation and deburring depth.

[0197] Ultrasound and UWB for Surgical Navigation With reference to FIG. 42, the following is an exemplary method for ultrasound-based surgical navigation using UWB tracking and virtual guides in accordance with the present disclosure. In this exemplary embodiment, no pre-operative imaging step is required. Instead, a B-mode ultrasound system with a tracked ultrasound probe is used in the operating room to collect data on the geometry of the bone surface, but before incision. This pre-incision information can take the form of B-mode, RF, or IQ ultrasound data (or any combination of the above) and is converted from ultrasound to a 3D point cloud using signal processing techniques known to those skilled in the art. Once all ultrasound data is collected, the patient's bone model is reconstructed from the ultrasound data and imported into a previously calibrated augmented reality processing system. From the bone model, and optionally the ligament and soft tissue information obtained by the ultrasound system, a default surgical plan is created that can be optionally modified by the surgical team. The surgical plan includes the depth and orientation of the resection, as well as the size classification of the implant. In parallel, the patient's anatomy is registered to the AR system using one of many options, including registration of the point cloud from the tracked probe or visual object recognition by the AR system camera. Prior to registration, a reference fiducial marker or tracked probe is rigidly fixed to real-world anatomy for tracking throughout the entire surgical procedure. A virtual cutting guide is created from the registered anatomy and surgical plan. The AR system displays the virtual cutting guide to the surgeon by rendering the combination of the anatomy and the cutting guide on the surface of the real-world image. When performing the resection prescribed by the cutting guide, tracked surgical instruments can be used so that the position and orientation of the final resection can be verified by the system. After the resection, the AR system can be used to visualize the orthopedic trial implant evaluation and calculate the kinematic performance of the implant, which can be used to update the virtual cutting guide for further refinement if deemed unsatisfactory. If the placement and performance are deemed satisfactory, the surgery can proceed to the final implantation.

[0198] Turning to FIG. 43, the aforementioned AR system may be utilized as part of a surgical needle injection procedure using UWB tracking and virtual guides. In this exemplary embodiment, pre-operative imaging is performed to generate a model of the patient's anatomy (e.g., knee joint). Alternatively, as known to those skilled in the art, a user may utilize an ultrasound probe to generate an ultrasound data representation of the patient's anatomy and generate a point cloud from which a virtual model of the patient's anatomy may be generated. In either method, the user / operator wears an AR helmet including a head-up display that generates a virtual image of the patient's bones and needle, each tracked in 3D using an IMU unit and / or a UWB unit. More specifically, with the left hand, the operator holds a bone tracking device fixedly mounted on the outside of the patient's lower leg, while with the right hand, the needle to be delivered inside the patient's knee joint. As the operator looks through the visor of the AR helmet, the virtual representations of the patient's bones, the knee joint and the needle, are superimposed on the real-world image of the patient's leg in a registered and aligned manner. Considering that the patient's leg and the needle are tracked simultaneously, any changes in position and orientation will be updated simultaneously in real time via the real-world view and the virtual image of the patient's bones and the needle. In this modality, after needle insertion, the operator has a direct and virtual line of sight to know exactly how far the needle has been injected into the joint and where it is located. In this modality, the operator can visualize the proper needle path and anatomical structures by the augmented display. The anatomical structures can be created by reconstruction of the ultrasound data and rendered on the glasses in the proper position after registering the helmet to the ultrasound tracking system.

[0199] As shown in FIG. 44, the AR system described above can be utilized to superimpose a B-mode image of the ultrasound image onto the patient's anatomy to aid in anatomical referencing and eliminate the need for the operator to look at a computer monitor.

[0200] 45, the AR system described above can be utilized to generate virtual anatomical structures that are constructed from ultrasound data and updated based on further ultrasound scans. This can be particularly useful when an operator uses ultrasound imaging to find / locate difficult anatomical landmarks or to assist in visualizing features located near the reconstructed anatomical structure.

[0201] As shown in FIG. 46, one direct application of AR and ultrasound is to create a framework for navigating a subchondroplasty procedure. Preoperative MRI can be used to create an accurate anatomical model as well as identify the location of the subchondral defect to be treated. In the operating room, ultrasound can be used to register the MRI data to the patient by identifying the bony borders. This information can also be used to register an augmentation system to the patient and ultrasound data to navigate the procedure. Finally, the needle path and fixation positions of the subchondroplasty instrument are rendered above the patient via an AR display (such as a helmet head-up display on the visor) and updated in real time by tracking with a UWB unit, an IMU unit, or an electromagnetic sensor to assist the surgeon in injecting the subchondroplasty material.

[0202] UWB and IMU for non-line-of-sight surgical navigation of total ankle arthroplasty Total ankle arthroplasty surgical procedures require many complex instruments and techniques. The procedure can be simplified using wireless tracking and customized registration. The following exemplary system includes software and hardware components to perform wireless tracking of surgical instruments and implant components via ultra-wideband communications against a known preoperative surgical plan and patient anatomy.

[0203] The first step as part of this exemplary process is pre-operative patient imaging. This can consist of 3D imaging techniques such as MRI or CT, or multiple 2D images such as X-ray, ultrasound, or fluoroscopy. The resulting images can be used for the creation of a virtual patient-specific anatomical model. For the ankle, it is generally required that the fibula, tibia, and talus be reconstructed for complete surgical planning. In some cases, it is sufficient to appropriately scale the fibula, calcaneus, and other bony structures that may provide visual references but are not required for specific planning parameters.

[0204] Surgical planning is the process of generating a pre-operative virtual surgical plan that may include sizing, orientation, and positioning of components with a patient-specific virtual surface model. The surgical plan is communicated from the planning application to an intraoperative navigation platform, as well as an optional platform for creating a custom patent anatomical mapper (PAM) for registering the virtual plan to the patient intraoperatively (see Figures 47-50).

[0205] An exemplary navigation system that may be used as part of this exemplary surgical procedure may require a minimum of two tracking devices. Each tracking device may include multiple UWB antennas (e.g., four or more) and an inertial measurement unit that may include accelerometers, gyroscopes, and magnetometers. Thus, the complete position and orientation of the tracking device may be known to the second tracking device by fusion of IMU and UWB data.

[0206] As part of an exemplary navigation system, a UWB tracking unit may connect multiple antennas to a single transceiver, which allows for the ability to range and track multiple UWB antennas (targets) utilizing the same UWB transceiver unit. In an exemplary configuration, the antennas in the UWB tracking unit may function as either a master role or a peripheral role, with the master unit acting as a timing reference for the peripheral units in the system. As a further example, the antennas in the UWB tracking unit may be arranged in any configuration, provided there are four or more antennas and that one of the antennas is not on the same plane as the other three antennas. For example, a tetrahedral configuration would satisfy this condition. In this configuration, the four antennas may connect to a single UWB transceiver, with each antenna acting as a reference point for positioning. With a single timing circuit and a single transceiver to feed UWB pulses to the multiple antennas, this configuration enables clock synchronization between all reference points in the unit. This configuration may greatly improve the flexibility of the installation of the master unit, as well as facilitate calibration procedures. In short range localization applications, a single master system is sufficient to provide adequate positioning data for localization. In wide range localization applications, multiple master systems can be used. The timing circuits of the master units are synchronized during operation, either by wired or wireless methods.

[0207] The tetrahedral design allows for the determination of the position and orientation of each UWB unit. One or more IMUs can be additionally used to determine the orientation more accurately. The output of both sensor systems is fused (see FIG. 4) such that errors from the UWB and IMU(s) are reduced by fusion and filtering techniques. One exemplary method for reducing errors includes using a recursive Bayesian estimation filter to complement the orientation estimation between the UWB unit / system and the IMU unit / system. The IMU system may be subject to magnetic artifacts that are inaccurate and can be compensated for using the UWB system. In addition, the multi-antenna design of the UWB system can be used to monitor and improve the translation estimation between the antennas.

[0208] In another configuration, the UWB tracking system may use single or multiple external reference units that may be mounted or placed at different locations within the operating environment (e.g., operating room) to improve positioning accuracy. The reference units may use single or multiple antennas in different configurations. The reference units may communicate with each other via wired or wireless connections. The reference units may behave as master units to other UWB tracking units, and they may also be configured to perform positioning relative to any other reference units or UWB tracking units based on time difference of arrival, time of flight, angle of arrival techniques. The placement of the reference units may be arbitrary or may be optimized using position dilution of precision (PDOP) based estimation (see FIG. 2). Also, the calculation of PDOP may be configured to optimize the reference unit placement in 2D or 3D.

[0209] In another configuration, the UWB and / or IMU tracking systems can be rigidly affixed to a mobile platform, which may be in the form of a surgical robotic system, to improve performance without relying on placing external reference units around the operating room (see FIG. 5). In this exemplary embodiment, the system can be registered to a single bone anatomy by first rigidly affixing a reference UWB-IMU tracker to the bone. A second tracker can be tracked relative to the reference tracker. Registration of the patient's anatomy to this local coordinate system can be accomplished in a variety of ways.

[0210] One method is to create a patient anatomical mapper (PAM) that has at least one surface that matches the "negative" of the patient's anatomy, and is rigidly attached to a second tracker such that when placed in registration with the patient's anatomy, the position and orientation of the tracker on the PAM is known, and the position and orientation of the PAM on the anatomy is known. Once placed in the registered position and orientation, the system registers the patient's anatomy into a local coordinate system.

[0211] The second method is to palpate or digitize the anatomical surface with a tracking device. The drawback of this method is the lack of PAM. It requires one reference tracker rigidly fixed to the patient's anatomy. A second tracker is used to sample the surface points of the anatomical structure. A sufficient number of surface points are required to obtain a point cloud. This point cloud represents a digitization of the exact anatomical surface and is registered to a previously created virtual surface model. One such point cloud-model registration method is called iterative closest point.

[0212] FIG. 51 provides a process flow diagram for an exemplary total ankle arthroplasty procedure. First, the patient's ankle is imaged using any number of imaging modalities, including, but not limited to, CT, MRI, x-ray, ultrasound, and fluoroscopy. After ankle imaging, a virtual model of the ankle bones is created by obtaining the tibia and talus from the ankle image, while average scale models of the fibula, calcaneus, and navicular are registered. Next, a tibial reference adapter is fixed to the proximal portion of the tibia using bone pins. The reference adapter has a UWB and IMU pod / unit attached to it. The tibial joint is then palpated using a stylus with the IMU and UWB pod attached. A talus reference adapter is then fixed to the proximal tibia using bone pins, and then the IMU and UWB pod is attached to the talus reference adapter. The talus is then palpated using a stylus with the IMU and UWB pod attached. After palpation, the talus cutting guide equipped with UWB and IMU units is positioned on the tibia. The surgeon can then place bone pins to fix the talus cutting guide in place. The talus is then navigated to the correct flexion angle (e.g., 20 degrees) and the correct medial-lateral position. A bone pin is then attached to the talus through hole of the talus cutting guide to lock the ankle angle. The talus is then resected through the slot in the talus cutting guide, after which the talus cutting guide can be removed. Using the same bone pin previously attached to the tibia, the tibial cutting guide is attached to the tibia, and the tibia is resected using the slot in the cutting guide. Finally, the tibia and talus bone fragments are removed, as well as the pins previously attached to the tibia and talus. Note that registration may be performed separately for the tibia and talus. Tracked during the procedure are the cutting blocks required to perform the talus and tibia resections consistent with the preoperative surgical plan.

[0213] UWB and IMU for non-line-of-sight surgical navigation of total knee arthroplasty With reference to FIG. 52, the exemplary navigation system described above may also be utilized to perform navigated total knee replacement / total knee arthroplasty.

[0214] As part of this process, imaging of the patient is taken preoperatively. This can consist of 3D imaging techniques such as MRI or CT, or multiple 2D images such as x-ray, ultrasound, or fluoroscopy. These images are used to create a virtual patient-specific anatomical model. For the knee, at a minimum, the knee portion of the joint (distal femur and proximal tibia) may need to be created.

[0215] After image acquisition, surgical planning can be accomplished. Surgical planning is the process of generating a virtual surgical plan that may include sizing, orientation, and positioning of components with a patient-specific virtual surface model. The surgical plan is communicated from the planning application to an intraoperative navigation platform and an optional platform for creating a custom patent anatomical mapper (PAM) for intraoperatively registering the virtual plan to the patient.

[0216] An exemplary navigation system may require a minimum of two tracking devices. Each tracking device may include multiple UWB antennas (at least four) and an inertial measurement unit that may include accelerometers, gyroscopes, and magnetometers. Thus, the complete position and orientation of the tracking device may be known to the second tracking device by fusion of IMU and UWB ranging data.

[0217] In this exemplary process and apparatus, the UWB tracking unit may connect multiple antennas to a single transceiver, which allows for the ability to range and track multiple UWB antennas (targets) utilizing the same UWB transceiver unit. In one example structure, the antennas in the UWB tracking unit may function as either a master role or a peripheral role, with the master unit acting as a timing reference for the peripheral units in the system. The antennas in the UWB tracking unit may be arranged in any structure, provided that there are four or more antennas and that one of the antennas is not on the same plane as the other three antennas. For example, a tetrahedral structure would satisfy this condition. In this structure, the four antennas may connect to a single UWB transceiver, with each antenna acting as a reference point for positioning. With a single timing circuit and a single transceiver to feed UWB pulses to the multiple antennas, this structure allows for clock synchronization between all reference points in the unit. This structure may greatly improve the flexibility of the installation of the master unit, as well as facilitate calibration procedures. In short range localization applications, a single master system is sufficient to provide adequate positioning data for localization. In wide range localization applications, multiple master systems can be used. The timing circuits of the master units are synchronized during operation, either by wired or wireless methods.

[0218] By connecting multiple antennas to a single transceiver, it enables the ability to create multiple anchors or tags within the same UWB unit. The UWB antenna arrays of both the central unit and the peripheral units may be arranged in any configuration, provided that one of the antennas is not on the same plane as the other three antennas. For example, a tetrahedral configuration would satisfy this condition. The UWB antenna array of the central unit serves as an anchor for the system. For example, a tetrahedral configuration would have four antennas connected to a single UWB transceiver. This creates four anchors in the central unit. With a single clock and a single transceiver to feed UWB pulses to the multiple antennas, this structure enables clock synchronization between all anchors in the unit. This structure may greatly improve the flexibility of anchor installation, as well as facilitate the calibration procedure of the unit. For short range localization applications, a single central system is sufficient to provide adequate anchors for localization. For wide range localization applications, multiple central systems may be used. The clocks of the central units are synchronized during operation, either by wired or wireless methods.

[0219] In an alternative exemplary configuration, the UWB tracking system may use single or multiple external reference units that may be mounted or placed at different locations in the operating room to improve positioning accuracy. The reference units may use single or multiple antennas in different configurations. The reference units may communicate with each other via wired or wireless connections. The reference units may behave as master units to other UWB tracking units, and they may also be configured to perform positioning relative to any other reference units or UWB tracking units based on time difference of arrival, time of flight, angle of arrival techniques. The placement of the reference units may be arbitrary or may be optimized using position dilution of precision (PDOP) based estimation (FIG. 2). Also, the PDOP calculation may be configured to optimize the reference unit placement in 2D or 3D.

[0220] In a further alternative exemplary configuration, the UWB and / or IMU tracking systems can be rigidly fixed to a mobile platform, which may be in the form of a surgical robotic system, to improve performance without resorting to placing external reference units around the operating room (see FIG. 5). In the proposed invention, the system is registered to a single bone anatomy by first rigidly affixing a reference UWB-IMU tracker to the bone. A second tracker is tracked relative to the reference tracker. Registration of the patient anatomy to this local coordinate system can be accomplished in a variety of ways.

[0221] The system is registered to a single bone anatomy by first rigidly affixing a reference UWB-IMU tracker to the bone. A second tracker is tracked relative to the reference tracker. Registration of the patient's anatomy to this local coordinate system can be accomplished in a variety of ways.

[0222] One method is to create a patient anatomical mapper (PAM) that has at least one surface that matches the "negative" of the patient's anatomy, and is rigidly attached to a second tracker such that when placed in registration with the patient's anatomy, the position and orientation of the tracker on the PAM is known, and the position and orientation of the PAM on the anatomy is known. Once placed in the registered position and orientation, the system registers the patient's anatomy into a local coordinate system.

[0223] The second method is to palpate or digitize the anatomical surface with a tracking device. The drawback of this method is the lack of PAM. It requires one reference tracker rigidly fixed to the patient's anatomy. A second tracker is used to sample the surface points of the anatomical structure. A sufficient number of surface points are required to obtain a point cloud. This point cloud represents a digitization of the exact anatomical surface and is registered to a previously created virtual surface model. One such point cloud-model registration method is called iterative closest point.

[0224] With particular reference to FIG. 52, an exemplary procedure for utilizing UWB and IMU trackers is shown. Step 1 begins with exposure of the knee joint through a surgical process. Step 2 involves rigidly affixing a reference tracker / unit to the femur and registering the femur to the tracking system. Image registration is accomplished by digitizing the exposed joint surface. In an alternative exemplary embodiment, registration may be accomplished with a custom-made device shaped to match a unique position and orientation on the femur (bone) that is known preoperatively. The tracker is attached to the custom-made registration device in a known position and orientation such that when placed on the patient's anatomy, the position and orientation of the tracker is known relative to the patient's anatomy. Step 3 involves attaching the tracker to an existing surgical device that will be used to perform the distal knee amputation. The system tracks the position and orientation of the instrument relative to the patient's femur and provides feedback to the surgeon regarding the expected resection depth and orientation. When the position of the cutting block matches the surgical plan, the block is pinned to the patient's anatomy. Alternatively, the tracker can be tied directly to the cutting instrument or bone pin. Step 4 is performed after the distal cut and is similar to step 3, with the difference that the surgical instrument being tracked relative to the anatomy is a cutting block to perform the posterior resection, anterior resection, and chamfer resection. After the femoral resection is performed, step 5 involves the registration of the tibia, which is performed in a similar manner as disclosed in step 2. Step 6 involves tracking the position and orientation of the tibial cutting instrument and pinning it at the desired location after which the resection is performed. Step 7 uses the tracking system to determine and size the rotation of the tibial implant after the resection. This may involve setting the rotation relative to landmarks on the tibia or kinematic landmarks that may be tied to the femoral anatomy. After proper placement, as in step 8, a trial implant can be used with the tracker rigidly fixed to the implant or patient anatomy to test for ideal procedural outcomes such as kinematics or loosening.Note that registration is performed separately for the femur and tibia. Tracked during the procedure are the cutting blocks necessary to perform femoral and tibial resections consistent with the preoperative surgical plan. Alternatives to the described procedure include, but are not limited to, (i) after registration, sizing information can be determined or verified by selecting appropriate sizing landmarks with the tracked probe, and (ii) custom surgical instruments made specifically for the tracking probe can be used in place of traditional instrumentation.

[0225] Dynamic patient-specific implants: Combining morphology and kinematics With reference to FIG. 53, an exemplary process for utilizing dynamic data over an episode of surgical care to create patient-specific implants and instruments, as well as to place the implants and monitor performance postoperatively, is described. The process includes kinematic assessment, which may be performed using fluoroscopy or other dynamic imaging modalities (as opposed to traditional static imaging), which is uploaded to a processing CPU to reconstruct the motion and anatomy from the imaging data. The kinematics and anatomy are used as inputs to a predictive model to determine optimal reconstructed geometry and normal kinematics. The outputted shape parameters are used to construct dynamically created patient-specific implants and associated instruments. The implants and instruments are manufactured to achieve accurate placement and are coupled to smart instrumentation in the surgical plan and operating room. Finally, to monitor performance, postoperative data such as kinematics and range of motion are collected by any number of sensors, including IMU, UWB, or a combination of these, along with the patient's reported outcomes. This data can then be utilized in a feedback loop with a predictive engine to optimize implant parameters for further patients. In this manner, data from the entire episode of care can be utilized to drive the design of the implants and instruments.

[0226] An introduced step, step 21, in the dynamic implant creation process shown in FIG. 53 is dynamic patient imaging. Dynamic patient imaging can be accomplished by the use of static 3D imaging techniques such as MRI or CT followed by active and passive kinematic activity collected by a motion capture system (e.g., optical, EM, UWB, and / or IMU). Imaging can also be accomplished using ultrasound dynamic patient imaging and joint reconstruction as shown in FIG. 54. As shown in FIG. 54, 3D bone information can be captured from ultrasound images (RF data, B-mode images, or IQ data) or other imaging modalities. Coupling an imaging device with tracking sensors on the patient provides the ability to capture the dynamic information necessary to further reconstruct the anatomy as well as create the patient-specific kinematics and soft tissue envelope used for the implant design methodology described.

[0227] Another method for dynamic imaging is to use fluoroscopic imaging. A patient joint model is then reconstructed from this image sequence. Superimposition of patient-specific shape information onto single-plane digital fluoroscopy (Figure 55 top left) to reconstruct the patient joint model. Active contact areas are tracked to extract joint curvature and soft tissue information. An example of 3D points and curvature information extracted from fluoroscopy is seen in the top right image of Figure 55. The points represent the 3D surface geometry of the femur and are linked to the tibial curvature of the articulation region. Within the articulation region, the normal curvature can be extracted by comparison with a database of shape and kinematics, and the closest output from the database can be used to optimize the manufactured component to match the normal profile. The bottom image of Figure 55 illustrates how patellofemoral kinematics can drive a custom patellofemoral groove curvature from the patella articulation surface and groove orientation (also extracted from fluoroscopy or ultrasound data) for optimal patellofemoral tracking.

[0228] After dynamic imaging, a bone and soft tissue reconstruction step 22 is performed. By way of example, the bone and soft tissue reconstruction step may utilize fluoroscopy and may consist of four parts, including but not limited to the following, as shown in FIG. 56: (i) image processing to extract features from the fluoroscopy image; (ii) initialization to estimate the initial pose of the 3D model using a hybrid classifier that integrates k-nearest neighbor (KNN) and support vector machine (SVM); (iii) optimization to determine the optimal pose and shape of the 3D model by maximizing a similarity measure between the 2D x-ray fluoroscopy and the reconstructed 3D surface mesh model. The similarity measure is designed as a novel energy function that includes a border score, a region score, a homogeneity score, and a multi-body registration score; and (iv) 3D shape analysis to represent the training dataset for the 3D surface mesh model using a nonlinear statistical shape model called kernel principal component analysis (KPCA).

[0229] As shown in Figure 56, the creation of anatomical information from dynamic fluoroscopic image data begins with the acquisition of a fluoroscopic image. As part of this image acquisition, the subject / patient may be viewed in any number of positions including deep knee flexion and the end point of contralateral gait. After image acquisition, image processing sub-steps are performed.

[0230] Using a calibration target, distortions can be estimated and removed from subsequent images as part of the image processing sub-steps. An exemplary first step of this procedure is to estimate the geometric distortions of any 2D image. By x-raying a known rectangular grid of metal beads, the 2D spatial transformation can be estimated for each of the small square sub-images bounded by four beads. In a standard technique used in removing geometric distortions, a local bilinear model can be used to model the spatial mapping as well as the gray level interpolation. Once the 2D distortions have been removed, the effective source-to-image planar distance (focal length) can be calculated by a biplanar calibration grid with a known displacement between the planar surfaces.

[0231] Figure 57 shows fluoroscopic images of the geometric calibration grid before and after removal of the geometric distortion. As part of this sub-step, a bilinear transformation can be calculated for each set of four grid points that transforms the image positions of the beads in the image on the left to the regularly spaced grid positions on the right. Clearly, the calibration procedure removes the pincushion distortion so that the grid points lie along straight lines.

[0232] After image processing, an initialization sub-step can be performed to determine the initial pose of the mean model. The initialization is based on a hybrid classifier that combines k-nearest neighbors and support vector machines, as shown in Figure 59.

[0233] Two main methods of reconstruction have been developed to construct 3D patient anatomy from fluoroscopic images, as shown in Figure 56. The first method, Method 1, involves sequential shape and pose estimation, while the second method, Method 2, involves reconstruction using an And-Or-Tree (AoT). A more detailed description of each of these methods follows.

[0234] The 3D reconstruction of the sequential shape and pose estimation is based on a nonlinear statistical shape model, i.e., kernel principal component analysis (KPCA). As shown in FIG. 60, by projecting the training data into a high-dimensional kernel space, the shape of the 3D model can be represented by a vector of shape parameters. As shown in FIG. 61, as part of this first method, an optimization process is performed that determines the shape and pose parameters of the 3D model from an array of monoplane fluoroscopic x-ray images. As shown in Table 1, the optimization is based on a novel energy function that combines border, area, homogeneity, and multi-body registration scores to measure the similarity between the 3D model and the 2D x-ray images. The hybrid energy function does not require time-consuming DRR generation or error-prone 2D segmentation. [Table 1]

[0235] The 3D model is then reconstructed by approximating the prior image since the distribution map between the input space points and the feature space points is not necessarily known. It is preferable to reconstruct the prior image of the corresponding test point based on the distance constraint in the input space. This is achieved by establishing the relationship between the input space distance and the feature space distance as shown in Figure 62.

[0236] Alternatively, reconstruction can be accomplished using AOT techniques, as shown in FIG. 65. First, the geometric space is decomposed by extracting a volume of interest (VOI) as a volume in which part templates may reside. Each VOI is further divided into a set of overlapping sub-volumes that are used as partitioned volumes for placement of part templates. Examples of sub-volumes are shown on the left nodes of FIG. 65. An "And-Or tree" can be generated recursively by dividing a volume and representing the division by pairs of And-Or nodes. An "Or node" connects to all "And nodes" that slice the volume represented by this "Or node" into two sub-volumes. The "Or node" also connects to two sets of leaf nodes whose surfaces are placed on each node either by an inscribed volume or a surface perpendicular to the depth direction. Each "And node" connects two Or nodes, each of which represents one of two smaller sub-volumes that occupy the current sub-volume. The tree starts from a root "Or node" that represents the volume of interest (VoI) and continues to grow until the sub-volumes are divided to a bounded size. Using surfaces as bounded boxes, the appearance of the part template can be further defined. The possible appearances for each part template are also represented by an "And-Or tree", where "And" represents composition and "Or" represents transformation. A layer of "And nodes" decomposes the part template into curve segments. These curves are projected into the image plane. The 3D object template can be transformed into a 2D object template consisting of active curves that should resemble the appearance of the object in the image plane. The deformed active curves are then projected back into the object space as a reconstructed 3D model.

[0237] As shown in Figure 66, the Volume of Interest (VoI) is determined by detection. The shape of the generic model is learned from different known poses by optimizing the obtained information. The template is then projected onto the 2D image plane as an active contour that deforms in the image plane. The leaves of the appearance "And-Or tree" are projected onto the 2D image plane as active contours. At the part level, the template may perform in-plane translations, rotations, which are called 3D deformations. The projected active curves can also be deformed in 2D. Both 2D and 3D deformations are guided by maximizing the obtained information. The 3D model is reconstructed by projecting the deformed active curves back onto the object plane.

[0238] As shown in Figures 67-69, an alternative to using statistical shape deformation is to directly identify features on the image and use so-called And-Or trees for shape identification and deformation (see Hu, Wenze, and Song-Chun Zhu, "Learning 3d object templates by quantizing geometry and appearance spaces," IEEE Transactions on Pattern Analysis and Machine Intelligence 37.6 (2015):1190-1205, the disclosure of which is incorporated herein by reference). In the aforementioned publication, shape parameters for bony anatomical structures are prescribed by AoT structures and identification of these structures in a fluoroscopy frame.

[0239] It is worth noting that for the knee, at a minimum, the knee portion of the joint (distal femur and proximal tibia) needs to be created, however, the same approach can be applied to any joint.

[0240] Referring back to FIG. 53, sub-step 22 of bone and soft tissue reconstruction involves predicting the soft tissue and normal anatomical structures using the dynamic images obtained in sub-step 21. As part of standard implant design or patient-specific implants and instruments, static CT or MRI is relied upon to extract the morphology of the joint. However, the morphology usually changes due to disease or deformity. In the case of osteoarthritis of the knee, cartilage is lost and osteophytes present may change the morphology of the knee. The use of static CT and MRI cannot accurately describe the condition of the ligaments of the joint. For example, the collapse of the medial compartment of the knee in addition to the presence of osteophyte growth changes the dynamic behavior of the medial collateral ligament (MCL) and lateral collateral ligament (LCL). Given these major changes to the soft tissue and bone, extraction of the bone contour becomes difficult, inaccurate and sometimes impossible. In this situation, a statistical atlas of a particular population can be used to accurately predict the location of the ligaments and then predict the original deformed bone shape in addition to dynamically extracting the design parameters and curvatures for this particular patient. Rather than relying on static images to generate patient-specific implants and instruments, the present embodiment uses static images in addition to kinematic data (dynamic data) to generate implants and instruments that are optimized to replicate both the patient's anatomy and kinematics. Prior art patient-specific implants and instruments are at best optimized only to replicate the patient's anatomy, but ignore kinematics or fail to use kinematics as a factor in the final shape of the orthopedic implant.

[0241] Turning to FIG. 70, as part of reconstructing the soft tissue associated with the virtual bone models, the locations of the ligaments are extracted from the imaging data. In particular, surface models of the bones and ligaments are reconstructed from the MRI. The bone models are added to a statistical atlas, and each vertex is flagged as either belonging to an attachment site or not based on its distance to the ligament surface model. Performing this process for multiple subjects allows the creation of a probability distribution map for each ligament attachment site (shown in the top row of the femoral attachment sites for the ACL and PCL in FIG. 70). In the distribution map, each bone atlas vertex is assigned a probability of belonging to an attachment site.

[0242] Referring to FIG. 71, as a further part of the bone and soft tissue reconstruction sub-step 22, femur and tibia contact maps are created during deep knee flexion. For a single subject, both the femur and tibia are assigned corresponding vertices in the respective bone atlases. The pose of the femur relative to the tibia is updated at each flexion angle. At each pose, the femur and tibia vertices that belong to the contact region are determined based on their proximity to the articulating bone. Performing this analysis across multiple subjects allows the creation of a probability distribution map for each bone of the contact region at each flexion angle.

[0243] Figure 72 shows the inputs and outputs associated with an exemplary method for determining cartilage thickness within the contact area during deep knee flexion (as recorded during dynamic imaging substep 21 of Figure 53). Using this method, the contact area is mapped, accurate patient-specific thickness is determined, and this information is linked back to a previously created normal statistical atlas. In this way, kinematics is linked to morphology.

[0244] FIG. 73 shows a patient-specific cartilage distribution map (obtained from sub-step 21 of the kinematic analysis of FIG. 53) showing severe cartilage loss in the medial compartment. Creating a patient-specific implant with this information alone would result in poor implant functionality. Instead, the present embodiment uses statistical methods to estimate the morphology before the deformations are created, thereby allowing the exact patient-specific curvature (pre-pathology) to be extracted. This estimated morphology then results in better implant functionality.

[0245] Referring to FIG. 74, a flow diagram for estimating cartilage thickness from dynamic data according to the present disclosure is shown. Considering multiple postures of an articulating surface, e.g., a knee, contacts for each posture are determined. Contacts are determined primarily using a small percentage of the closest points between each model. For each contact point, the distance between the point and the articulating model is determined. The cartilage thickness can then be estimated as X% of model 1 and Y% of model 2 such that the sum of the two thickness values ​​is equal to the total distance between the surfaces. Calculation of thickness at each contact vertex in each posture provides a set of "known" thicknesses that should be kept constant during the estimation procedure. This set can be considered as convex set 1 in the projection on convex set (POCS) algorithm. Convex set 2 is a cartilage atlas previously calculated from a priori datasets, which may include normal anatomy, specific pathological cohorts (varus, valgus cases of the knee), or a combination of these. According to the POCS algorithm, convex set 1 is projected onto convex set 2 and the result is projected back onto convex set 1, and so on until the result converges. In the described algorithm, the projection onto the atlas updates all vertices that belong to the cartilage. If the result has not converged, the vertices that belong to the cartilage are set to convex set 1 (of "known" thickness) and projected back onto the atlas until convergence is reached. Upon convergence, the cartilage surface of each articulating bone model is exported. This routine allows for an accurate estimation of the cartilage by capturing the complete contact information using the dynamic data.

[0246] Referring back to Figure 53, during bone shape reconstruction, ligament locations are predicted using a ligament shape atlas that has the ability to capture soft tissue loci across populations with the probability of each point being a ligament locus as shown in Figure 74. The calculated ligament insertion points are then used to calculate the ligament length envelope during kinematic activity as shown in Figure 75.

[0247] The basis of the cartilage estimation is a statistical model that includes an average cartilage template and locally deforms the average cartilage template using information from the segmented femoral and tibial models. The average cartilage template is the average cartilage thickness calculated from a database of manually segmented cartilage models. Each thickness value has an index associated with it and corresponds to a point on the bone atlas used to localize the value. When adding the average template to a new bone model, each point on the bone is deflected outward along the normal direction by a distance corresponding to the average thickness from the template at that location. The average cartilage template is only adjusted if there is an overlap of the femoral and tibial cartilage. In this case, the cartilage thickness is reduced by a small factor globally and by a larger factor in the overlapping regions. This process is repeated until there are no more overlapping regions.

[0248] As shown in Figure 76, by using the estimated cartilage distribution map together with the measured joint deformation, the location of cartilage loss can be determined and the amount of cartilage loss can be estimated by projecting the patient's cartilage onto a normal cartilage model. By correcting the amount of cartilage loss, the joint can be returned to its normal alignment. This change in joint alignment directly affects ligament length and laxity. For example, as shown in Figure 78, medial cartilage loss will result in laxity in the medial collateral ligament and increased tension in the lateral collateral ligament. The restoration of normal joint alignment changes in ligament length can be calculated (see Figure 79). Using the soft tissue and morphological information, the closest kinematic model is selected from the normal kinematic database.

[0249] As an example, determining normal healthy kinematics may be through the use of a network of deep neural cells where the network can be trained with movements performed by healthy joints. The network of deep neural cells can take pathological movement inputs and determine optimal healthy kinematics (see FIG. 73).

[0250] With reference to FIG. 80, a flow chart is provided illustrating an exemplary process for calculating patient-specific ligament stiffness as part of the normal kinematic prediction component of FIG. 53. In an exemplary embodiment, a series of passive kinematic motion aspects are recorded, including but not limited to the motion, the velocity of the motion, the acceleration of the motion, and the length of the ligaments during the motion. These aspects are used as inputs for a passive motion model, which also receives inputs regarding the ligament stiffness and mass of the tissue moved during the motion. Using these inputs, the passive motion model predicts the force required to move the tissue. This predicted force is then compared to the measured force to analyze whether the ligament stiffness value should be adjusted so that the predicted force is equal to the measured force. In situations where the predicted and measured forces are not equal or very similar, the ligament stiffness value and / or mass input are optimized and updated to allow for subsequent calculation of the predicted force. This process may be repeated until the predicted and measured forces are within an acceptable tolerance of each other. A more detailed explanation of how the measured forces are generated is discussed below.

[0251] A force actuator attachment is connected to the distal end of the tibia near the ankle, which allows the tibia to be passively extended from approximately 90 degrees of flexion to full extension relative to the femur during fluoroscopic monitoring. The actuator force may be constant during the entire activity. Care should be taken so that during this activity, the femur does not move, but only the tibia extends. Multiple passive trials can be performed by varying the amount of constant force in each trial. Care should be taken so that the activity is performed at a slow speed so that the patient does not feel a sense of imbalance and does not tighten their muscles to impede the activity. The kinematics of the tibia and patella relative to the femur can be calculated from the fluoroscopic images using 2D to 3D reconstruction. An inverse dynamic model of the lower leg should be created to simulate a passive leg-raising activity. This model should incorporate the femur, tibia, and patella, as well as the major ligaments that span the joint (patellar ligament, ACL, PCL, MCL, LCL). The reconstructed bone model can be used along with the seating positions of the soft tissues to set the geometry of the model. The model can start with initial stiffness values ​​of the ligaments, masses, and inertia of the components based on previously published values ​​and can predict the amount of force required to perform an activity. Finally, patient-specific values ​​of these parameters can be calculated by using an optimization algorithm that matches the predicted forces to the actuator forces recorded during multiple trials.

[0252] Returning to FIG. 53, the output from the normal kinematic prediction sub-step 23 can be utilized as input for the patient-specific dynamic creation step 24 and the instrument creation steps 2-4.

[0253] As shown in FIG. 81, a flow diagram is provided to illustrate an exemplary process for creating a patient-specific dynamic implant. There may be at least three potential paths for generating a patient-specific dynamic implant, all of which may rely on a database containing healthy joint data including profile, kinematics, and soft tissue properties. One of the paths may utilize active kinematics performed by the patient in combination with the healthy kinematic database to estimate a patient-specific healthy kinematic profile 23-B and utilize soft tissue properties 23-C. Outputs from sub-steps 23-B and 23-C may be input to an implant shape optimization sub-step 28 to optimize the implant kinematics and shape. Another path may utilize the patient-specific healthy kinematic profile estimation described in 23-B and input it to a deep neuronal network model in sub-step 27 to directly create the implant shape.

[0254] FIG. 82 shows an exemplary process flow for creating patient-specific implants using estimates from either healthy kinematic profiles or ligament stiffness.

[0255] Using the ligament stiffness values ​​obtained previously as described with respect to FIG. 81 and based on feedback from the surgeon, a decision is made as to what type of implant the patient should have. For example, in the context of knee replacement, the decision may be whether the patient should receive a cruciate-retaining, posterior-stabilized, BCS, or BCR implant. Since manufacturing custom polyethylene bearing components is not always a cost-effective business model, it is proposed to have a wide family of polyethylene geometric templates and corresponding femoral templates to capture the statistical shape variations of normal knee anatomy. There should be different families for each type of implant with different sizes of implants within a family to account for the different sizes of bones that correspond to a particular shape. Since it is not possible to perfectly match the properties of the multiple tissues involved in the articulation of the joint (cartilage, meniscus) and still obtain a reliable component, the shape of the articulating surface should be optimized to adequately constrain the motion while distributing the loads to optimize the life of the component. Current implants rely on sagittal and coronal polyethylene curvatures to either constrain kinematics by matching the femoral curvature (supercongruence) or reduce the curvature or reduce constraints (unconstrained) by creating a completely flat articulating surface. The latter is more common in partial joints or when ligaments are retained, as these soft tissues partially constrain the kinematics.

[0256] Optionally, the dynamically designed tibial surface can be optimized for contact over the entire range of motion corresponding to the femoral template group. This allows for precise matching of the surface with the expected loads for added longevity and improved performance. Based on the predicted normal anatomy curvature of the patient, first the implant group, then the best size of the femoral and polyethylene template within the group is selected as the base model. The idea is to have the best model that fits the patient's anatomy and keeps the patient in neutral alignment.

[0257] An example of a pair of different polyethylene sagittal curvatures (top is femoral curvature) shown in FIG. 83. Top left is a less constrained congruent curvature. This design allows more posterior-anterior translation through flexion. Top right is a super congruent design, which constrains translation due to a curvature that more closely mirrors the femoral curvature while increasing the contact area. Custom designs should be optimized based on static non-weight-bearing imaging data. The dynamic implants disclosed herein utilize imaging that allows analysis of weight-bearing kinematics that are used to define the correct curvature based on expected implant loads, contact areas, and desired post-operative kinematics.

[0258] With reference to Fig. 81 and Fig. 84, once the kinematic profile and soft tissue model are identified, sub-step 28 of implant shape optimization may include a forward dynamic flexion-driven simulation of the lower limb. Based on the type of implant, the relevant soft tissue may be retained in the model and the template implant is adapted to the bone using natural joint alignment. For this simulation, the flexion of the femur relative to the tibia should be prescribed while the remaining five degrees of freedom (three translations, axial rotation, abduction / adduction) are predicted. Based on normal data as a reference previously identified as suitable for the patient, optimization may be performed to modify the sagittal and coronal curvature of the femoral implant condyle (and applicable cam) so that the difference between the desired and observed degrees of freedom is minimized. Once the best template model is optimized (as a double check, the simulation is repeated with two implants corresponding to lower and higher sizes to generate the optimal femoral profile for each polyethylene geometric template), the final design can be selected based on a weighted combination of the one that gives the best replication of the normal knee geometry and maintains sufficient contact area with the polyethylene bearing to ensure reduced wear and improved longevity.

[0259] Referring to FIG. 85, from Newton's equations of motion, F=ma (force=mass×acceleration). Thus, if the accelerations (kinematics) of the relevant components are known, the forces driving the motion can be predicted, a process known as inverse dynamics. For the passive model shown in FIG. 85, the geometry would consist of the femur, tibia, and patella assumed as rigid bodies with appropriate ligaments (ACL, PCL, MCL, LCL) connecting the femur and tibia, as well as extensor mechanisms (quadriceps and patellar ligaments). Since the femur is stationary, the system has 12 degrees of freedom (6 for the tibia and 6 for the patella) and 12 outputs can be calculated. Inputs to the model may consist of the translational and rotational kinematics of the tibia and patella relative to the femur, the spring stiffness of the ligaments, and assumed inertial parameters. Thus, the ligaments should be modeled as massless nonlinear springs that only generate load when stretched, and the forces in them should be specified as inputs based on the amount of stretch they exhibit. The quadriceps and patellar ligaments may be modeled as massless systems, and the forces in them can be predicted. Finally, a force can be applied to the distal tibia to simulate the actuator forces. Outputs from the model may be predictions of distal actuator forces and contact forces at the femoral-tibial and patellofemoral joints. In this case, the joint contact forces are not of interest, and the trial-by-trial input kinematics are also measured. Thus, a multivariate optimization can be performed to identify the best spring stiffness parameters and inertia values ​​that most accurately predict the actuator forces.

[0260] Referring to FIG. 86, one embodiment of a kinematic driven model is a flexion driven model. This model is a forward dynamics model and is essentially the opposite of the passive model discussed immediately above. The idea is to predict accelerations given the force and inertia values. The geometry may consist of the femur, femoral implant, tibia, tibial implant, and patella, assumed as rigid bodies with appropriate ligaments (ACL, PCL, MCL, LCL) connecting the femur and tibia, as well as extensor mechanisms (quadriceps and patellar ligaments). The implant components may be assumed to be rigidly fixed to the bone. This model is boundary condition driven. A point on the distal end of the tibia corresponding to the ankle joint should be held fixed, and the head of the femoral head should be pressed down at a constant velocity to simulate knee flexion. The system generates 18 equations (6 for the tibia, 6 for the femur, and 6 for the patella). Considering only the femoro-tibial kinematics are of interest, five degrees of freedom of the femur relative to the tibia can be solved (axial rotation, three translations, and abduction / adduction). All other equations should be suitably constrained. The femoral kinematics can be influenced by the contacting geometry of the implant condyles. The sagittal and coronal curvatures of the femur and implant should be updated using a multivariate optimization algorithm that attempts to match the observed kinematics with the target kinematics of the femur relative to the tibia.

[0261] Turning to FIG. 81 and FIG. 87, a more detailed description of sub-step 27 of the deep learning model follows. In an exemplary embodiment, a process for creating and using a network to define optimal CAD parameters for a patient implant that can be restored from kinematic data can be utilized. Inputs to a network of deep neural cells can include patient geometry, soft tissue data, and kinematic data, all of which are processed by the network to determine the best shape of a patient-specific implant that will help restore the patient's anatomy and achieve optimal post-operative kinematics. Those skilled in the art are familiar with neural cell networks, so a detailed description of the neural cell network has been omitted in the interest of brevity.

[0262] Returning to Figure 53, after implant creation, the process can proceed to manufacturing sub-step 25. The manufacturing process for the patient-specific dynamic knee can encompass the generation or creation of patient-specific implants, patient-specific instruments, and patient-specific resection cutting guides. The optimized knee implant can account for intraoperative bone loss by creating patient-specific resection planes that can pass information to smart cutting guides (described above) that are performed during surgery.

[0263] 89-92, the geometry of the femoral implant cutting box is designed to minimize the amount of bone resected (or re-cut in the case of revision) based on the patient's anatomy and the implant's curvature. Inputs for the design of the implant's cutting box may include the implant J-curve, the patient's anatomy, and the closest conventional implant's cutting box. Due to the unique resection planes that differ from conventional implant resection planes, smart cutting guides (discussed above) can be used to optimize and perform the resection in the operating room (see FIGS. 34-37).

[0264] Returning to FIG. 53, output from manufacturing sub-step 25 provides input to intraoperative navigation sub-steps 5-10.

[0265] As shown in FIG. 93, an exemplary flow chart outlines the overall concept of creating a dynamic implant and executing a surgical plan using a smart cutting guide intraoperatively. Dynamic imaging data is collected and used preoperatively to create the patient anatomy as already outlined in FIG. 56 (Creating the Patient Anatomy). The anatomical and kinematic data is then used to create the dynamic implant articulation as already outlined in FIG. 81-FIG. 84 (Creating the Dynamic Implant). This creation is bounded by the geometric constraints of the manufactured and approved implant. Creation of a patient-specific cutting box as outlined in FIG. 88-FIG. 91 (Creating a Patient-Specific Cutting Box). This patient-specific cutting box design makes it easier to use the smart cutting block discussed above intraoperatively to execute the planned surgical cuts. The cutting box design may be bounded by manufacturing constraints of the box design, such as, but not limited to, the thickness of the implant, and constraints driven by the motion of the smart cutting block. The designed patient-specific implant is then adjusted for manufacturing, and the preoperative surgical plan (location and orientation of the cut) can be used to configure the smart cutting guide. Implant placement information and pre-operative surgical planning can be stored or transmitted to an intraoperative navigation system where a smart cutting guide can be used to facilitate cutting and where a surgical navigation and guidance system is used for implant placement navigation and range of motion verification.

[0266] Returning to FIG. 53, after intraoperative navigation sub-steps 5-10 and the surgical procedure are completed, the exemplary AR system can be utilized post-operatively to assess the patient's condition.

[0267] As shown in FIG. 94, the aforementioned exemplary AR system can be utilized post-operatively in combination with a UWB and / or IMU tracking system to study, monitor, and evaluate a patient's recovery process by quantitative metrics such as range of motion, contact area of ​​the implant, and analysis of muscle activity during motion. In an exemplary form, a set of UWB trackers and IMU trackers can be placed on both sides of a joint for post-operative evaluation. The AR system overlays the position of the implant based on the patient's motion, analyzes muscle activity, and provides an evaluation to the physician in real time. In this exemplary application, the UWB and / or IMU tracking system(s) may be implemented as a belt-fastened device or a body-fitting garment.

[0268] Turning to FIG. 95, an exemplary list of patient-specific implants and parameters used to create the implants is shown. Of particular relevance to the knee are the different curvatures of the medial-lateral femoral condyles, the femoral patellar groove (and patellar button curvature), and the lateral lip of the tibial plateau, which are essential for accurate patient-specific femortibial tracking. These bony structures can be coupled with patient-specific soft tissue and pathology restoration techniques discussed above to create accurate patient-specific knee implants and procedures. The bottom portion of the flow chart outlines the steps for pre-operative imaging and joint reconstruction used to drive the creation of the implant parameters. From these parameters, the implants and plans are created and implemented in the operating room.

[0269] Turning now to Fig. 96, an exemplary process for creating a patient-specific implant from dynamic data is outlined. Pre-operative imaging is used to extract patient-specific design parameters, which are then compared to a database containing kinematic analyses of normal and implant patients. This comparison allows for the creation of corrective estimates of the dynamic data and the derivation of restored anatomical curvature as part of the dynamic implant process discussed above.

[0270] Referring to FIG. 97, an exemplary information flow for extracting design parameters (such as curvature) from the dynamic data and statistical atlas according to the dynamic implant process discussed above is shown.

[0271] Referring to Fig. 98, a flow diagram of an exemplary process for constructing an articular implant from dynamic data is shown, where a patient-specific shape is constructed from analysis of single-plane digital fluoroscopic images. This patient-specific data can be fed back into the dynamic data to extract patient-specific contact and soft tissue envelopes for creating an implant from a single image capture according to the dynamic implant process discussed above.

[0272] Turning to FIG. 99, another example of dynamic joint reconstruction is shown. Here, 3D bone information can be captured from ultrasound images (RF data, B-mode images, or IQ data). The picture is a reconstruction of a knee or shoulder. Coupling the imaging device with a tracker on the patient provides the ability to further reconstruct the anatomical structures as well as capture the dynamic information required to create the patient-specific kinematics and soft tissue envelopes used to capture the implant design methodology described, according to the dynamic implant process discussed above.

[0273] Proceeding to FIG. 100, an exemplary process for utilizing dynamic data over an episode of surgical care to create patient-specific implants and instruments, place the implants, and monitor performance postoperatively is described. The process begins with a kinematic assessment using fluoroscopy or other dynamic imaging modality (as opposed to traditional static imaging), which is uploaded to a processing CPU to reconstruct the motion and anatomy from the imaging data. The kinematics and anatomy are used as inputs to a predictive model to determine optimal reconstructed geometry and normal kinematics. The outputted shape parameters are used to construct dynamically created patient-specific implants and associated instruments. The implants and instruments are manufactured to achieve accurate placement and are coupled to smart instrumentation in the surgical plan and operating room. Finally, to monitor performance, postoperative data such as kinematics and range of motion are collected by any number of sensors including IMU, UWB, or a combination of these, along with the patient's reported outcomes. This data can then be utilized in a feedback loop with a predictive engine to optimize implant parameters for further patients. In such a method, data from the entire episode of care can be utilized to drive the design of the implants and instruments.

[0274] FIG. 101 shows an example screenshot from a knee planning software application that includes the kinematic data as captured by the dynamic imaging process and allows for the development of information regarding the soft tissue envelope throughout the range of motion that will aid in improving ligament balance when developing a surgical plan as opposed to adjustments in the operating room.

[0275] 102 and 103, another simulation step can be performed to model deep knee flexion and results captured with pre-operative fluoroscopy and stored for comparison with the original data observed on the patient during the pre-operative scenario discussed above, including weight-bearing and non-weight-bearing activities that can be used to drive the simulated post-operative performance. Based on the simulation, the femoral shape can be further optimized to ensure that the implant has sufficient contact area with the polyethylene bearing to ensure reduced wear and improved longevity. This model can be established using the patient's kinematics, soft tissue attachment, proposed implant design, and anatomy as inputs. Based on the inputs to achieve the estimated implanted kinematics and dynamics, including implant loading, the output should be optimized throughout flexion. If the achieved results are not acceptable due to looseness or improper loading, the shape or more specifically the curvature parameters can be iteratively optimized until the desired results are achieved. In such a method, femoral, tibial, or patellar implant shape parameters can be optimized based on multiple patient activities.

[0276] From the above description, it should be apparent to one skilled in the art that the methods and apparatus described herein constitute exemplary embodiments of the invention, but that the invention described herein is not limited to any precise embodiment, and modifications can be made to such embodiments without departing from the scope of the invention as defined by the claims. Additionally, it should be understood that the invention is defined by the claims, and that any limitations or elements describing the exemplary embodiments described herein are not intended to be incorporated into the interpretation of any claim element unless such limitations or elements are expressly stated. Similarly, it is understood that the invention is defined by the claims, and that it is not necessary to meet any or all of the specified advantages or objectives of the invention disclosed herein to fall within the scope of any claim, since inherent and / or unexpected advantages of the invention may exist even if they may not be expressly discussed herein.

Claims

1. A surgical navigation system comprising: a first tracker comprising at least one of a first inertial measurement unit and a first ultra-wide band unit, the first tracker configured to output data indicative of changes in position and orientation of the first tracker, the first tracker configured to be attached to a patient's anatomy; a second tracker including at least one of a second inertial measurement unit and a second ultra-wide band unit, the second tracker configured to output data indicative of changes in position and orientation of the second tracker, the second tracker configured to be attached to a surgical instrument including a haptic motor; and a third tracker including at least one of a third inertial measurement unit and a third ultra-wide band unit, the third tracker being configured to output data indicative of changes in position and orientation of the third tracker, the third tracker being attached to at least one of a projector and a graphical display configured to display an augmented reality image, and the third tracker being attached to the first camera; a processor communicatively connected to the first tracker, the second tracker, and the third tracker, the processor configured to utilize a surgical plan and images from the first camera to generate instructions for displaying the augmented reality image, and to update the display in near real time with data output from the first tracker and the third tracker, including changes in orientation and / or position of the augmented reality image; the processor is communicatively connected to the surgical instrument and uses data output from the first tracker and the second tracker to generate instructions for triggering a haptic motor in near real-time. Surgical navigation system.

2. the third tracker, the processor, and the graphical display being integrated as part of a user-wearable helmet. The surgical navigation system of claim 1.

3. the helmet has a visor including the graphical display on which the augmented reality image is displayed, The surgical navigation system of claim 2.

4. the surgical instrument including a motorized gyroscope configured to provide directional feedback; The surgical navigation system of claim 1.

5. the processor uses data output from the first tracker and the second tracker to generate instructions for triggering the motorized gyroscope in near real-time. The surgical navigation system of claim 4.

6. the augmented reality image includes at least one of a virtual orthopedic implant, a virtual guide, and a virtual bone; The surgical navigation system of claim 1.

7. the processor is configured to generate instructions for displaying a kinematic verification of a surgeon manually manipulating a patient's bone. The surgical navigation system of claim 1.

8. a head mounted graphical display including a visor for displaying both virtual and real world images; The surgical navigation system of claim 1.

9. 1. A method for displaying an augmented reality image, comprising: obtaining output data from a first tracker, the first tracker including at least one of a first inertial measurement unit and a first ultra-wide band unit, attached to the patient's anatomy, the output data being indicative of changes in position and orientation of the first tracker; obtaining output data indicative of changes in position and orientation of a second tracker from a second tracker including at least one of a second inertial measurement unit and a second ultra-wide band unit, and obtaining output from a first camera attached to the second tracker that is attached to at least one of a projector and a graphical display configured to display an augmented reality image; a processor communicatively coupled to the first tracker and the second tracker, the processor configured to utilize a surgical plan and output from the first camera to generate instructions for displaying the augmented reality image, and to utilize output data from the first tracker and the second tracker to update the display in near real-time, including changes in at least one of an orientation and a position of the augmented reality image. A method for displaying augmented reality images.

10. obtaining output data from a third tracker, the third tracker including at least one of a third inertial measurement unit and a third ultra-wide band unit, the third tracker being attached to the surgical instrument, the output data being indicative of changes in position and orientation of the third tracker; 10. The method of claim 9.

11. the processor is communicatively coupled to the surgical instrument and uses data output from the first tracker and the third tracker to generate instructions for triggering a haptic motor of the surgical instrument in near real-time. The method of claim 10.

12. the virtual image includes at least one of a virtual guide, a virtual implant, and a virtual bone; 10. The method of claim 9.

Citation Information

Patent Citations

  • Systems and methods for displaying equipment maintenance and operating instructions using augmented reality

    JP2009538487A

  • Computer-assisted surgical instrument tracking system and method

    JP2014533964A

  • Tracking objects with markers

    US20070081695A1

  • Methods and systems for tracking and guiding sensors and instruments

    US20130237811A1

  • Methods and systems for creating virtual and augmented reality

    WO2015192117A1