Systems and methods for sensory augmentation in medical procedures

Abstract

To provide a self-contained surgical navigation system including a head-mounted display device worn by a user during surgery.SOLUTION: The system includes a display generator for generating a visual display on a display device and a sensor suite having at least one tracking camera 3904. The system further includes a support module including a user-replaceable modular battery removably inserted into a housing of the support module, and a processor unit configured to receive data from the sensor suite and calculate a position and orientation of the at least one marker. The support module is electrically coupled to the head-mounted display device and provides power and data to the head-mounted display device. The display device and the support module cooperate to provide the full sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electrical equipment.SELECTED DRAWING: Figure 39

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. patent application Ser. No. 16 / 786,938, filed February 10, 2020, which is a continuation-in-part of PCT application Ser. No. PCT / US2018 / 18330, filed February 15, 2018, which is a continuation-in-part of U.S. patent application Ser. No. 15 / 674,749, filed August 11, 2017, and PCT application Ser. No. PCT / US2017 / 046438, filed August 11, 2017, both of which claim the benefit of priority to U.S. provisional application Ser. No. 62 / 375,483, filed August 16, 2016, the contents of each of which are incorporated by reference in their entirety for all purposes.

[0002] The present invention relates to novel visualization and sensory augmentation devices, systems, methods, and apparatus for positioning, orientation, and situational awareness during medical procedures, including but not limited to surgery, diagnostic procedures, therapeutic procedures, and anesthesia procedures. [Background technology]

[0003] Current medical procedures are typically performed by surgeons or medical personnel with little or no support beyond the instruments necessary to effect change on the patient. For example, an orthopedic surgeon may have some measuring instruments (e.g., rulers or the like) and cutting tools (e.g., saws or drills), but the surgeon is not supported with visual, auditory, and / or tactile input. In other words, the surgeon sees nothing but what he or she is manipulating, hears nothing but normal communications from other participants in the operating room, and senses nothing beyond the normal feedback from grasping instruments or other items of interest in the procedure. Alternatively, large console-based navigation or robotic systems are utilized, with their displays and cameras located outside the sterile field, away from the surgeon. These require the surgeon to repeatedly shift their gaze between the surgical site and the two-dimensional display. Additionally, the remote location of the camera creates line-of-sight issues when drapes, personnel, and / or instruments obstruct the camera's view of markers in the sterile field, and the camera's viewpoint is not suitable for intra-wound imaging. Anatomical registration is typically performed using a stylus with markers to probe so that the markers are visible to the camera. Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention provides visual projection of feedback required for a procedure(s) into the user's field of view without requiring the user to unnaturally move or rotate their head to view an external screen. Augmented or virtual displays appear to the user as a natural extension or enhancement of the user's vision. Furthermore, sensors and cameras within the user's headpiece have the same perspective as the user, minimizing line-of-sight ambiguity issues associated with external cameras. Our 3D mapping of anatomical surfaces and features and matching with models from preoperative scans represents a faster and more accurate method for intraoperatively registering anatomy than current stylus point cloud approaches. [Means for solving the problem]

[0005] The present invention generally comprises novel sensory augmentation devices or apparatuses comprising at least one augmentation for a user's vision, hearing, or touch to assist in the performance of a medical procedure. Visual assistance can be provided in the form of a real-time visual overlay on the user's field of view in the form of augmented reality, or as a replacement for the visual scene in the form of virtual reality. Auditory assistance can be provided in the form of simple beeps or tones, or more complex sounds such as voices or instructions. Haptic assistance can be provided in the form of simple alerting haptic feedback or more complex haptic generation intended to guide the user. In preferred embodiments, visual (augmented or virtual) assistance is complemented by audio or haptic feedback, or both audio and haptic feedback.

[0006] The present invention provides a mixed reality surgical navigation system, comprising a head-mounted display device (e.g., a headset) worn by a user (e.g., a surgeon) during surgery, the head-mounted display device having a processor unit, a display generator, and a sensor suite having at least one tracking camera, and at least one visual marker trackable by the camera and fixedly attached to a surgical instrument, the processing unit using data received from the sensor suite to map a three-dimensional surface of a partially exposed surface of an anatomical object of interest, the processing unit establishing a reference frame of the anatomical object by matching the three-dimensional surface to a three-dimensional model of the anatomical object, the processing unit using data received from the sensor suite to track a six-degree-of-freedom pose (consisting of position and orientation) of the surgical instrument, and the processing unit communicating with a display to provide a mixed reality user interface including a stereoscopic virtual image of desired features of the surgical instrument and desired features of the anatomical object within the user's field of view.

[0007] The present invention further provides a method of using a mixed reality surgical navigation system for a medical procedure, the method comprising the steps of: (a) providing a mixed reality surgical navigation system comprising: (i) a head-mounted display device comprising a processor unit, a display, and a sensor suite having at least one tracking camera; and (ii) at least one visual marker trackable by the camera; (b) attaching the display device to a user's head; (c) providing a surgical instrument having a marker; (d) scanning an anatomical object of interest with the sensor suite to obtain three-dimensional surface data of desired features of the anatomical object; (e) transmitting the three-dimensional surface data to the processor unit to register a virtual three-dimensional model of the desired features of the anatomical object; (f) tracking the surgical instrument in a six-degree-of-freedom pose with the sensor suite to obtain data that is transmitted to the processor unit; and (g) displaying a mixed reality user interface including a stereoscopic virtual image of features of the surgical instrument and features of the anatomical object within the user's field of view.

[0008] The present invention further provides a mixed reality user interface for a surgical navigation system, the mixed reality user interface including a stereoscopic virtual image of desired features of a surgical instrument and desired features of an anatomical object of interest in a user's field of view provided by the mixed reality surgical navigation system, the mixed reality user interface including: (i) a head-mounted display device including a processor unit, a display, and a sensor suite having at least one tracking camera; and (ii) at least one visual marker trackable by the camera, wherein the mixed reality user interface is obtained by the following steps: (a) attaching the head-mounted display device to the user's head; (b) providing a surgical instrument having a marker; (c) scanning the desired anatomical object with the sensor suite to obtain three-dimensional surface data of a partially exposed surface of the anatomical object; (d) transmitting the three-dimensional surface data to the processor unit to register a virtual three-dimensional model of the features of the anatomical object; (e) tracking the surgical instrument in a six-degree-of-freedom pose with the sensor suite to obtain data to be transmitted to the processor unit; and (f) displaying the mixed reality user interface including the stereoscopic virtual image of the features of the surgical instrument and the features of the anatomical object in the user's field of view.

[0009] The present invention also provides a method for tracking a probe during surgery. For example, the method may include receiving two-dimensional images of a patient's internal anatomy using an ultrasound transducer, tracking the position and orientation of the ultrasound transducer, tracking the position and orientation of the patient, combining the two-dimensional images of the patient with the position and orientation of the ultrasound transducer relative to the patient, reconstructing the two-dimensional images in a common frame of reference using the position and orientation of the ultrasound transducer and the position and orientation of the patient to generate a three-dimensional image of the patient's internal anatomy, tracking the position and orientation of the probe, displaying the axis and position of the tip of the probe relative to the three-dimensional image of the patient's internal anatomy, and advancing the tip of the probe to a desired position based on its position relative to the patient's internal anatomy. The method may further include receiving two-dimensional images of a patient's external anatomy or outer surface using one or more stereo cameras, tracking cameras, or ultrasound transducers, and displaying the two-dimensional images of the external anatomy in the reconstructed three-dimensional image. The method may also be used to monitor the position, advancement, retraction, etc. of pins, needles, screws, injection devices, probes, etc. The method may be performed by any of the head-mounted display devices and / or mixed reality surgical systems described elsewhere herein.

[0010] One aspect of the present disclosure is directed to a self-contained head-mounted surgical navigation system. In some embodiments, the system includes a display generator for generating a visual display on a display device, a sensor suite having at least one tracking camera, and a processor unit configured to receive data from the sensor suite and calculate positions and orientations of at least two markers by: determining a position of a first of the at least two markers within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to guide the user to a position of a second of the at least two markers relative to the first marker; and determining a position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

[0011] Another aspect of the present disclosure is directed to a self-contained surgical navigation system. In some embodiments, the system includes a head-mounted display device worn by a user during surgery, the head-mounted display device including a display generator for generating a visual display on the display device and a sensor suite having at least one tracking camera. The system also includes a support module including a user-replaceable modular battery removably inserted into a housing of the support module and a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least one marker.

[0012] In any of the previous embodiments, the system further includes one or more of a face shield and a helmet, and the display device is adapted to be mounted to the face shield or the helmet.

[0013] In any of the preceding embodiments, the system further includes at least one marker affixed to the object of interest for tracking the object of interest. In some such embodiments, the at least one marker is outside the field of view of the at least one tracking camera, such that the processor unit is further configured to: track an angle of the user's head using one or more sensors in the sensor suite; calculate a relative position of the at least one marker based on a last known position of the at least one marker when the at least one marker was located within the field of view of the at least one tracking camera, the last known position being relative to the angle of the head; and display a virtual guide to the user on the display device to direct the user to the position of the at least one marker.

[0014] In any of the previous embodiments, the support module is electrically coupled to the head mounted display device to provide power and data to the head mounted display device.

[0015] In any of the previous embodiments, the support module is attached to the user's body at a location other than the user's head.

[0016] In any of the previous embodiments, the display device and the support module together comprise all of the sensing and computing power of the system without the need for external sensors, cameras, computers, or other electronic equipment.

[0017] In any of the previous embodiments, the system further includes at least two markers affixed to the object of interest for tracking the object of interest, a first marker within a field of view of at least one tracking camera and a second marker outside a field of view of the at least one tracking camera. In some such embodiments, the processor unit is further configured to: determine a position of the first marker within a field of view of the at least one tracking camera; display a virtual guide on the display device to a user to guide the user to a position of the second marker relative to the first marker; and determine a position of the second marker with the at least one tracking camera based on the guidance from the virtual guide.

[0018] In any of the previous embodiments, the system further includes acquiring initial positions of the first marker and the second marker, and, if the second marker is not within the field of view of the at least one tracking camera, estimating the position of the second marker relative to the first marker based on the acquired initial positions.

[0019] In any of the previous embodiments, the system further includes obtaining initial positions of the first marker and the second marker relative to known anatomical markers, calculating a distance between the known anatomical markers, and, if the second marker is not within the field of view of the at least one tracking camera, estimating a position of the second marker relative to the first marker based on the calculated distance.

[0020] In any of the previous embodiments, the system further includes tracking head movements of the user using one or more sensors in the sensor suite and calculating a position of the second marker based on the last known position of the second marker when it was within the field of view of the at least one tracking camera.

[0021] In any of the preceding embodiments, the system further includes at least two markers affixed to the object of interest for tracking the object of interest. In some such embodiments, one or both of the at least two markers are outside the field of view of the at least one tracking camera, and the processor unit is further configured to: display a virtual control between the at least two markers, display a user input control configured to align with the virtual control based on user input, adjust the position of the virtual control to align the user input control with the virtual control when the user turns their head, and track the at least two markers within the field of view of the at least one tracking camera when both of the at least two markers are within the field of view of the at least one tracking camera.

[0022] In any of the previous embodiments, the head mounted display device further includes infrared light.

[0023] In any of the previous embodiments, the system further includes a visible light and an infrared light filter coupled to the visible light, such that the infrared light filter prevents the visible light from emitting infrared light when coupled to the visible light.

[0024] In any of the previous embodiments, the system further includes a shroud disposed around the infrared light and including a plurality of side walls defining an opening through which light from the infrared light is emitted.

[0025] In any of the previous embodiments, the at least one tracking camera, the visible light, and the infrared light are positioned behind a face shield when the head-mounted display device is mounted on a helmet.

[0026] In any of the previous embodiments, the side walls contact the face shield when the head-mounted display device is attached to the helmet, preventing light emitted by the infrared light from being reflected back to the at least one tracking camera and passing only through the face shield.

[0027] In any of the previous embodiments, the system further includes the face shield and the helmet.

[0028] In any of the previous embodiments, the housing of the support module further includes a base including a circuit board arranged to conduct power from the battery to the processor unit and the head mounted display device.

[0029] In any of the previous embodiments, the housing of the support module further comprises a bracket configured to securely and removably restrain the battery and the processor unit when placed on the bracket.

[0030] Another aspect of the present disclosure is directed to a self-contained surgical navigation system configured for use with a helmet and face shield. In some embodiments, the system includes a head-mounted display device worn by a user during surgery, the head-mounted display device comprising: a display generator for generating a visual display on the display device; a sensor suite having at least one tracking camera, visible light, and infrared light; and a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least one marker.

[0031] In any of the previous embodiments, the system further includes a shroud disposed around the infrared light and including a plurality of side walls defining an opening through which light from the infrared light is emitted.

[0032] In any of the previous embodiments, the at least one tracking camera, the visible light, and the infrared light are positioned behind a face shield when the head-mounted display device is mounted on a helmet.

[0033] In any of the previous embodiments, the side walls contact the face shield when the head-mounted display device is attached to the helmet, preventing light emitted by the infrared light from being reflected back to the at least one tracking camera and passing only through the face shield.

[0034] In any of the previous embodiments, the system further includes an infrared light filter coupled to the visible light such that when the infrared light filter is coupled to the visible light, the visible light is prevented from emitting infrared light.

[0035] In any of the previous embodiments, the system further includes at least two markers affixed to the object of interest for tracking the object of interest, a first marker within a field of view of the at least one tracking camera and a second marker outside a field of view of the at least one tracking camera. In some such embodiments, the processor unit is further configured to: determine a position of the first marker within a field of view of the at least one tracking camera; display a virtual guide on the display device to a user to guide the user to a position of the second marker relative to the first marker; and determine a position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

[0036] In any of the previous embodiments, the system further includes a support module comprising a user-replaceable modular battery removably inserted into a housing of the support module and a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least one marker.

[0037] In any of the previous embodiments, the support module is electrically coupled to the head mounted display device to provide power and data to the head mounted display device.

[0038] In any of the previous embodiments, the support module is attached to the user's body at a location other than the user's head.

[0039] In any of the previous embodiments, the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electronic equipment.

[0040] In any of the previous embodiments, the shroud has a monolithic structure.

[0041] In any of the previous embodiments, a front surface coupled to the side walls is in contact with the face shield and has a radius of curvature that matches a radius of curvature of the face shield.

[0042] In any of the previous embodiments, a front surface coupled to the side walls is in contact with the face shield and has a radius of curvature that generally matches a radius of curvature of the face shield.

[0043] In any of the previous embodiments, one or more of the side walls are at an angle of 10 to 20 degrees relative to the central axis of the infrared light.

[0044] Another aspect of the present disclosure is directed to a self-contained surgical navigation system configured for use with a helmet and face shield. In some embodiments, the system includes a head-mounted display device worn by a user during surgery, the head-mounted display device comprising a display generator for generating a visual display on the display device, the display device mounted on one or more of a surgical helmet and a face shield, and further comprising a sensor suite having at least one tracking camera.

[0045] In any of the previous embodiments, the system further includes a support module including a user-replaceable modular battery removably inserted into a housing of the support module, and a processor unit.

[0046] In any of the previous embodiments, the support module is electrically coupled to the head mounted display device to provide power and data to the head mounted display device.

[0047] In any of the previous embodiments, the support module is attached to the user's body at a location other than the user's head.

[0048] In any of the previous embodiments, the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electronic equipment.

[0049] In any of the previous embodiments, the processor unit is configured to receive data from the sensor suite and calculate the position and orientation of the at least two markers by: determining a position of a first of the at least two markers within a field of view of the at least one tracking camera; displaying a virtual guide to a user on the display device to guide the user to a position of a second of the at least two markers relative to the first marker; and determining a position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

[0050] Another aspect of the present disclosure is directed to a head-mounted surgical navigation system for determining a joint center. Any of the head-mounted surgical systems described herein may be used to determine a joint center. The system may include a display generator for generating a visual display on a display device, a sensor suite having at least one tracking camera, at least one fiducial marker affixed to a bone for tracking the bone, the at least one fiducial marker being positioned so that the bone pivots at or relative to the joint, at least one fixed fiducial marker positioned to be substantially fixed relative to the joint, and a processor unit. The processor unit may be configured to: register points on the bone in a reference coordinate frame; create a bone coordinate frame based on the registered points; transform from the reference coordinate frame to the bone coordinate frame; acquire points of the at least one fixed marker in the reference frame using the at least one tracking camera, such that a position of at least a portion of the visual display moves synchronously with movement of the head-mounted surgical navigation system during acquisition; and determine a joint center in the bone coordinate frame.

[0051] In any of the preceding embodiments, determining includes calculating the location of the joint center in the bone coordinate system, processing substantially continuously through an optimal estimation filter to determine the joint center, determining includes batch processing after all points have been acquired to determine the joint center, or a combination thereof.

[0052] In any of the previous embodiments, the bone is one of a femur, tibia, humerus, radius, or vertebral body.

[0053] In any of the previous embodiments, the joint is one of a hip, knee, shoulder, elbow, ankle, or vertebral body.

[0054] In any of the previous embodiments, the stationary further includes being fixed in inertial space.

[0055] Another aspect of the present disclosure is directed to a head-mounted surgical navigation system for determining a hip joint center. Any of the head-mounted surgical navigation systems described herein may be used. The system may include a display generator for generating a visual display on a display device, a sensor suite having at least one tracking camera, at least one fiducial marker affixed to a femur for tracking the femur, the femur positioned to pivot at or relative to the hip joint, and at least one fixed fiducial marker arranged to be substantially fixed relative to the hip joint, and a processor unit. The processor unit is configured to register points on the femur in a reference coordinate frame, create a femoral coordinate frame based on the registered points, transform from the reference coordinate frame to the femoral coordinate frame, acquire points of the at least one fixed marker in the reference frame using the at least one tracking camera, and during acquisition, move a position of at least a portion of the visual display synchronously with movement of the head-mounted surgical navigation system to determine a hip joint center in the femoral coordinate frame.

[0056] In any of the previous embodiments, determining further includes calculating a location of a hip joint center in the thigh coordinate system, substantially continuously processing through an optimal estimation filter to determine the hip joint center, batch processing after all points have been acquired to determine the hip joint center, or a combination thereof.

[0057] In any of the previous embodiments, the stationary further includes being fixed in inertial space.

[0058] Another aspect of the present disclosure is directed to a method for registering condylar surfaces prior to setting a resection angle, as performed by any of the head-mounted surgical navigation systems described herein, the method being executed by a processor unit and including the steps of: displaying a target including one or more regions on a display of the head-mounted surgical navigation system; providing a movable icon on the display representing one or more angles received from a condylar guide in real time; receiving one or more user inputs for adjusting a position of the movable icon relative to one or more regions within the target; and outputting a visual marker on the display over any of the one or more regions of the target with which the movable icon interacts during adjustment of the position of the movable icon, such that the visually marked region indicates a captured valid depth reference point.

[0059] In any of the previous embodiments, the method further includes restricting movement of the movable icon so as not to record a previously captured valid depth reference point.

[0060] In any of the previous embodiments, the method further includes forming a database in which the captured valid depth reference points are stored.

[0061] In any of the previous embodiments, the target is a grid or a bullseye.

[0062] In any of the previous embodiments, each of the one or more regions is sequentially highlighted such that the method includes outputting instructions to the user on the display to move the condyle guide relative to the condyle until the movable icon at least partially overlaps the highlighted region.

[0063] In any of the previous embodiments, any one of the one or more regions is highlighted such that the method includes outputting instructions on the display to the user to move the condyle guide relative to the condyle until the movable icon at least partially overlaps the highlighted region.

[0064] In any of the previous embodiments, the method further includes deactivating the highlighted region and highlighting a second region of the one or more regions when the highlighted region and the movable icon at least partially overlap.

[0065] In any of the previous embodiments, the method further includes prompting the user to remove the condylar guide and attach a cutting guide.

[0066] In any of the previous embodiments, the method further includes calculating a resection depth based on a distance from a current resection plane defined by the cutting guide to one of the effective depth reference points corresponding to a depth reference plane.

[0067] In any of the previous embodiments, the method further includes providing a condylar guide comprising: a body having a first end and a second end; at least one planar surface extending from at least a lateral region of the first end, the planar surface configured to rest on one or more femoral condyles and establish a zero-depth plane for calculating resection depth; at least one tracker disposed on the at least one planar surface for tracking a pose of the condylar guide; and a connector extending from the second end of the body and configured to couple to a cutting guide.

[0068] In any of the previous embodiments, the condylar guide includes an elongated handle extending from the first end of the body.

[0069] In any of the previous embodiments, the body of the condylar guide further defines an aperture configured to receive a pin therethrough for insertion into bone.

[0070] In any of the previous embodiments, the diameter of the opening is sized to allow the condylar guide to tilt when a pin is inserted through the opening.

[0071] In any of the previous embodiments, the condylar guide further includes a release mechanism extending from the second end of the body in a direction opposite the connector. In any of the previous embodiments, the release mechanism is configured to couple the condylar guide to the bone prior to pinning the cutting guide to the bone.

[0072] In any of the previous embodiments, at least a portion of the second end of the body of the condylar guide defines a slot configured to receive a slider, and within the slot the connector and release mechanism are inserted on opposite sides of the slider.

[0073] In any of the previous embodiments, the at least one planar surface of the condylar guide is configured to simulate a plane that contacts a femoral condyle.

[0074] In any of the previous embodiments, the method further includes tracking the condylar guide using the at least one tracker positioned on the at least one planar surface to determine one or more valid depth reference points.

[0075] In any of the previous embodiments, the method further includes pinning the cutting guide only after determining the one or more valid depth reference points using the condylar guide coupled to the cutting guide.

[0076] In any of the previous embodiments, the connector of the condylar guide is removable.

[0077] Another aspect of the present disclosure is directed to a method for registering condylar surfaces prior to setting a resection angle, the method being performed by any of the head-mounted surgical navigation systems described herein, the method being executed by a processor unit and including the steps of: displaying a target including one or more regions on a display of the head-mounted surgical navigation system; receiving and displaying, in real time, one or more angles received from a condylar guide on the display; receiving one or more user inputs to adjust the condylar guide relative to the one or more regions in the target; and outputting, on the display, a visual marker over any of the one or more regions of the target, the visually marked region indicating a captured and valid depth reference point.

[0078] In any of the previous embodiments, the method further includes limiting the recording of previously captured valid depth reference points.

[0079] In any of the previous embodiments, the method further includes forming a database in which the captured valid depth reference points are stored.

[0080] In any of the previous embodiments, the target is a grid or a bullseye.

[0081] In any of the previous embodiments, each of the one or more regions is sequentially highlighted such that the method includes outputting instructions on the display to a user to move the condyle guide relative to the condyle until the angle of the condyle guide at least partially overlaps the highlighted region.

[0082] In any of the previous embodiments, the method further includes a step in which any one of the one or more regions is highlighted such that the method includes outputting instructions to the user on the display to move the condyle guide relative to the condyle until the angle of the condyle guide at least partially overlaps the highlighted region.

[0083] In any of the previous embodiments, the method further includes deactivating the highlighted region and highlighting a second region of the one or more regions when the highlighted region and the angle of the condylar guide at least partially overlap.

[0084] In any of the previous embodiments, the method further includes prompting a user to remove the condylar guide and attach a cutting guide.

[0085] In any of the previous embodiments, the method further includes calculating a resection depth based on a distance from a current resection plane defined by the cutting guide to one of the effective depth reference points corresponding to a depth reference plane.

[0086] In any of the previous embodiments, the method further includes providing a condylar guide, the condylar guide comprising: a body having a first end and a second end; at least one planar surface extending from at least a portion of a lateral region of the first end, the planar surface configured to rest on one or more femoral condyles and construct a zero-depth plane for calculating resection depth; at least one tracker disposed on the at least one planar surface for tracking a pose of the condylar guide; and a connector extending from the second end of the body and configured to couple to a cutting guide.

[0087] In any of the previous embodiments, the condylar guide includes an elongated handle extending from the first end of the body.

[0088] In any of the previous embodiments, the body of the condylar guide further defines an aperture configured to receive a pin therethrough for insertion into bone.

[0089] In any of the previous embodiments, the diameter of the opening defined by the body is sized to allow the condylar guide to tilt when a pin is inserted through the opening.

[0090] In any of the previous embodiments, the condylar guide includes a release mechanism extending from the second end of the body in a direction opposite the connector, the release mechanism configured to couple the condylar guide to the bone before pinning the cutting guide to the bone.

[0091] In any of the previous embodiments, at least a portion of the second end of the body defines a slot configured to receive a slider, and within the slot the connector and the release mechanism are inserted on opposite sides of the slider.

[0092] In any of the previous embodiments, the at least one planar surface is configured to simulate a plane that contacts a femoral condyle.

[0093] In any of the previous embodiments, the method further includes tracking the condylar guide using the at least one tracker positioned on the at least one planar surface to determine one or more valid depth reference points.

[0094] In any of the previous embodiments, the method further includes pinning the cutting guide to determine the one or more valid depth reference points only after using the condylar guide coupled to the cutting guide.

[0095] Some embodiments of the present invention are illustrated by way of example and not limitation in the accompanying drawings in which like references may indicate similar elements and in which: [Brief explanation of the drawings]

[0096] [Figure 1] 1 is a diagrammatic depiction of an expansion system in accordance with the principles of the present invention; [Figure 2A] FIG. 2 is a perspective front view of a diagrammatic representation of a display device of the system of FIG. 1. [Figure 2B] 2B is a perspective rear view of the display device of FIG. 2A. [Figure 3] 2 is a diagrammatic depiction of another embodiment of a display device of the system of FIG. 1. [Figure 4] FIG. 2 is a schematic diagram showing the electrical hardware configuration of the system of FIG. [Figure 5] FIG. 2 is a diagrammatic depiction of the markers and cameras of the system of FIG. 1. [Figure 6] 10 is a diagrammatic representation of a mixed reality user interface image ("MXUI") provided by the system of FIG. 1 during positioning of an acetabular shell in a hip replacement surgery showing a virtual pelvis. [Figure 7] 10 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during positioning of the acetabular shell in a hip replacement procedure, showing a virtual pelvis and a virtual acetabular impactor. [Figure 8]2 is a flow chart illustrating the operational process of the system of FIG. 1 during a medical procedure. [Figure 9] 2 is a flowchart illustrating a method of using the system of FIG. 1 to perform a hip replacement procedure in accordance with the principles of the present invention. [Figure 10] 2 is a flow chart illustrating a method of using the system of FIG. 1 to perform a general medical procedure in accordance with the principles of the present invention. [Figure 11] FIG. 1 is a perspective view diagrammatically depicting a hip joint impactor assembly including an acetabular shell and optical markers. [Figure 12] FIG. 12 is an exploded view of the hip joint impactor assembly shown in FIG. [Figure 13A] FIG. 2 is a perspective view of a diagrammatic representation of an anatomical marker assembly optionally included in the system of FIG. 1. [Figure 13B] FIG. 13B is a perspective view of the clamping assembly of the anatomical marker shown in FIG. 13A. [Figure 14] FIG. 13B is an exploded view of the anatomical marker assembly shown in FIG. 13A. [Figure 15] FIG. 2 is a diagrammatic perspective view of a calibration assembly optionally included in the system of FIG. 1. [Figure 16] FIG. 16 is an exploded front view of the calibration assembly shown in FIG. 15. [Figure 17] FIG. 17 is an exploded rear view of the calibration assembly shown in FIG. 16. [Figure 18] 2 is a diagrammatic representation showing the MXUI provided by the system of FIG. 1 during various calibration steps. [Figure 19] FIG. 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during the pelvic registration step of a hip replacement surgery. [Figure 20] 10 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during pin insertion into the pelvis of a hip replacement. [Figure 21]FIG. 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during the pelvic registration step of a hip replacement surgery. [Figure 22] 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during the femoral registration step of a hip replacement surgery. [Figure 23] 2 is a diagrammatic depiction of the MXUI provided by the system of FIG. 1 during femoral neck resection in a hip replacement procedure. [Figure 24] 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during positioning of the acetabular shell in a hip replacement procedure. [Figure 25] 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during positioning of the acetabular shell in a hip replacement procedure. [Figure 26] 2 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during femur repositioning in a hip replacement procedure. [Figure 27] FIG. 2 is a diagrammatic depiction of the MXUI provided by the system of FIG. 1 using a C-arm during a hip replacement procedure. [Figure 28] 2 is a flowchart illustrating how the system of FIG. 1 can be used in conjunction with a C-arm in a surgical procedure in accordance with the principles of the present invention. [Figure 29] FIG. 2 is a front view of a diagrammatic representation of an equipment identification and tracking label optionally included in the system of FIG. 1. [Figure 30] 2 is a flowchart of a method for registering, sharing, and / or tracking medical devices using the system of FIG. 1 in accordance with the principles of the present invention. [Figure 31] 10 is a diagrammatic representation showing the MXUI provided by the system of FIG. 1 during registration of the spine with an ultrasound transducer in a spinal fusion procedure. [Figure 32]10 is a diagrammatic representation showing the MXUI provided by the system of FIG. 1 during registration of the spine with a stylus in an open spinal fusion procedure. [Figure 33] FIG. 33 is a close-up front view of the surgical exposure of FIG. 32. [Figure 34] 10 is a diagrammatic representation showing the MXUI provided by the system of FIG. 1 during pedicle drilling in a spinal fusion procedure. [Figure 35] FIG. 35 is a close-up view of the virtual drill and target portion of FIG. 34. [Figure 36A] FIG. 2 is a perspective front view of a diagrammatic representation of a user wearing an AR headset of the system of FIG. 1. [Figure 36B] FIG. 2 is a perspective rear view of a diagrammatic representation of a user wearing an AR headset of the system of FIG. 1 with a protective face shield. [Figure 37A] FIG. 2 is a perspective front view of a diagrammatic representation of a user wearing an AR headset of the system of FIG. 1 with a surgical helmet. [Figure 37B] FIG. 37B is a perspective rear view of the item shown in FIG. 37A. [Figure 38A] FIG. 2 is a perspective front view of a diagrammatic depiction of various components of the system of FIG. 1. [Figure 38B] FIG. 37B is a perspective rear view of the surgical helmet shown in FIG. 37A. [Figure 39] FIG. 36B is a perspective front view of the AR headset shown in FIG. 36A. [Figure 40] FIG. 37B is an exploded perspective view of the surgical helmet shown in FIG. 37A. [Figure 41A] FIG. 41 is a perspective bottom view of the electromechanical coupling plate shown in FIG. 40. [Figure 41B] FIG. 41 is a perspective top view of the electromechanical coupling plate shown in FIG. 40. [Figure 42] FIG. 37B is a perspective front view of the components of the system shown in FIG. 37A used in knee replacement surgery. [Figure 43]FIG. 2 is a diagrammatic representation showing the MXUI provided by the system of FIG. 1 during registration of the distal femur in a knee replacement procedure. [Figure 44] FIG. 2 is a diagrammatic depiction of the MXUI provided by the system of FIG. 1 during resection plane planning in knee replacement surgery. [Figure 45] 10 is a diagrammatic representation of the MXUI provided by the system of FIG. 1 during placement of pins for positioning a cutting block in a knee replacement procedure. [Figure 46] 10 is a diagrammatic depiction of the MXUI provided by the system of FIG. 1 during tibial resection in a knee replacement procedure. [Figure 47] FIG. 2 is a perspective front view of a diagrammatic representation of a knee balancing device optionally included in the system of FIG. 1 for use during knee replacement surgery. [Figure 48] FIG. 2 is a diagrammatic depiction of the MXUI provided by the system of FIG. 1 during balance assessment in knee replacement surgery. [Figure 49] FIG. 48 is a perspective front view of the knee balancing device shown in FIG. 47. [Figure 50A] FIG. 1 is a diagrammatic representation of the exposed surfaces on the hip bone and proximal femur in a reference position. [Figure 50B] FIG. 1 is a diagrammatic representation of the exposed surfaces on the acetabulum and proximal femur in a displaced position. [Figure 51] FIG. 1 is a diagrammatic representation of the hip joint and leg showing the reference axes and planes for calculating femoral version. [Figure 52] FIG. 1 is a diagrammatic depiction of a hip joint with implanted components. [Figure 53] FIG. 10 is a diagrammatic representation of the hip joint impactor and shell showing the surfaces mapped onto the impactor. [Figure 54] 2 is a flowchart illustrating how the system of FIG. 1 can be used to analyze hip joint kinematics in accordance with the principles of the present invention. [Figure 55]1 is a flowchart illustrating an exemplary method for navigating a knee replacement procedure. [Figure 56] FIG. 1 is a diagrammatic representation of a knee with a unicondylar implant. [Figure 57] FIG. 1 is a diagrammatic representation of a tibia with a unicondylar implant. [Figure 58A] FIG. 1 is a diagrammatic depiction of a knee showing exemplary areas for surface mapping in a reference position. [Figure 58B] FIG. 1 is a diagrammatic representation of a knee showing exemplary areas for surface mapping in displaced positions. [Figure 58C] FIG. 1 is a diagrammatic depiction of a knee showing exemplary regions for surface mapping. [Figure 59] 1 is a flowchart illustrating an exemplary method for navigating a hip replacement surgery. [Figure 60] FIG. 1 is a diagrammatic representation of an eyepiece with a bracket. [Figure 61] FIG. 61 is an exploded perspective view of the eyepiece depicted in FIG. 60. [Figure 62] FIG. [Figure 63A] FIG. 6 is a schematic diagram showing the electrical hardware configuration of the support module circuit board 6212. [Figure 63B] 1 is a schematic diagram of a circuit board according to one embodiment of a support module. [Figure 64] FIG. 1 is a diagrammatic depiction of an MXUI showing elements to assist the user in positioning the camera FOV to encompass a desired marker. [Figure 65] 1 is a flowchart illustrating an exemplary method for optimizing surgical parameters. [Figure 66] FIG. 1 illustrates components of a system for measuring resection depth in knee surgery. [Figure 67] 10 is a flowchart illustrating an exemplary method for measuring resection depth on a femur. [Figure 68A] FIG. 1 is a diagrammatic representation of an adjustable cutting block. [Figure 68B] FIG. 68B is an exploded view of the cutting block depicted in FIG. 68A. [Figure 69] FIG. 61 is an exploded view of the eyepiece and bracket depicted in FIG. 60. [Figure 70] FIG. 61 is a side view of the eyepiece and bracket depicted in FIG. 60. [Figure 71] FIG. 61 shows the eyepiece and bracket depicted in FIG. 60 attached to a surgical helmet. [Figure 72A] FIG. 61 is a top view of the eyepiece depicted in FIG. 60 attached to a surgical face shield. [Figure 72B] FIG. 61 is a side view of the eyepiece depicted in FIG. 60 attached to a surgical face shield. [Figure 73A] FIG. 72C is an enlarged view of the shroud of FIGS. 72A and 72B. [Figure 73B] FIG. 72C is an enlarged front view of the shroud of FIGS. 72A and 72B. [Figure 73C] FIG. 72C is an enlarged top view of the shroud of FIGS. 72A-72B. [Figure 74] 10A-10C illustrate an embodiment of a method for guiding a user to adjust a condylar guide. [Figure 75] 10A-10C illustrate another embodiment of a method for guiding a user to adjust a condylar guide. [Figure 76] 10A-10C illustrate another embodiment of a method for guiding a user to adjust a condylar guide. [Figure 77] 10A-10C illustrate an embodiment of a method for storing and evaluating reference depths in a database during condylar surface mapping. [Figure 78] 10A-10C illustrate an embodiment of a method for retrieving reference depths from a database during condylar surface mapping. [Figure 79A] 1A-1C illustrate an embodiment of a condylar guide. [Figure 79B] FIG. 79B shows an enlarged view of the opening defined by the body of the condylar guide of FIG. 79A. [Figure 80]1 illustrates one embodiment of a cutting guide. [Figure 81] 10A-10C illustrate an embodiment of a method for determining joint centers for a surgical procedure. [Figure 82] FIG. 1 illustrates an embodiment of a method for determining the orientation of a marker in inertial space. [Figure 83] FIG. 1 illustrates an embodiment of a method for calibrating an IMU to a head-mounted display vision system. [Figure 84] FIG. 1 illustrates an embodiment of a gaze control method. [Figure 85] FIG. 1 illustrates one embodiment of a speech recognition and / or gaze control method. [Figure 86A] FIG. 1 is a perspective view of one embodiment of a fixture for calibrating a head-mounted navigation system. [Figure 86B] FIG. 86B is a front view of the fixture of FIG. 86A. DETAILED DESCRIPTION OF THE INVENTION

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used herein, the singular forms "a," "an," and "the" are intended to include the plural as well as the singular, unless the context clearly indicates otherwise. It will be further understood that as used herein, the terms "comprise" and / or "comprising" specify the presence of stated features, steps, operations, elements, and / or components, and / or groups thereof.

[0098] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms as defined in commonly used dictionaries should be interpreted to have a meaning consistent with the meaning in the context of the relevant art and this disclosure, and it will be further understood that they should not be interpreted in an idealized or overly formal manner unless expressly defined as such herein.

[0099] In describing the present invention, it will be understood that several techniques and steps have been disclosed. Each of these has its own advantages, and each can be used in combination with one or more, or even all, of the other disclosed techniques. Thus, for the sake of clarity, this specification refrains from unnecessarily repeating every possible combination of the individual steps. Nevertheless, the specification and claims should be read with the understanding that such combinations are fully within the scope of the present invention and the claims.

[0100] Novel sensory augmentation devices, apparatus, and methods for providing data to assist in medical procedures are discussed herein. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without the specific details.

[0101] Additionally, those skilled in the art will appreciate that any of the embodiments described herein may be combined with any other embodiment. For example, any combination of face shields, helmets, display devices, etc. is contemplated herein. Furthermore, any method executable by any processor unit may be implemented with any combination of face shields, helmets, display devices, etc. described herein or generally available in the art.

[0102] I. Sensory Augmentation System 1, 2A-B, and 3, a sensory augmentation system 10 of the present invention is provided for use in a medical procedure. The system 10 includes one or more visual markers (100, 108, 110), a processing unit 102, a sensor suite 210 having one or more tracking camera(s) 206, and a display device 104 having a display generator 204 that generates a visual display on the display device 104 for viewing by the user 106. The display device 104 is attached to the user 106 so that the display device 104 can augment the user's visual input. In one embodiment, the display device 104 is attached to the user's 106's head. Alternatively, the display device 104 is positioned separately from the user 106 while still augmenting the visual scene. In one embodiment, each of the markers (100, 108, and 110) is visually distinguishable and distinct from one another so that they can be individually tracked by the camera(s) 206.

[0103] 2A-2B, another exemplary embodiment of display device 104 includes a visor housing 200 having optics 202 that allows the video display of display generator 204 to be focused onto the eyes of user 106. Sensor suite 210 is attached to or made part of display device 104. Visor housing 200 includes a mounting mechanism 208 that allows it to be attached to the head or face of user 106 such that the alignment of display device 104 with respect to the visual path of user 106 is consistently repeatable.

[0104] 3, another exemplary embodiment of display device 104 includes a transparent face shield 300 that allows projection from a display generator 302 onto the shield 300, overlaying data and images within the visual pathway of the eyes of user 106. A sensor suite 306 is attached to or is part of the display device, shown here as face shield 300. Face shield 300 further includes an attachment mechanism 304. Sensor suite 306 and attachment mechanism 304 provide the same functionality as sensor suite 210 and attachment mechanism 208 described above.

[0105] Referring to FIG. 4 , which illustrates the electronic hardware configuration of system 10, the sensor suite (210, 306) includes one or more tracking cameras 402, 404, 406 (same as 206), as well as an optional inertial measurement unit (“IMU”) 408, a radio 410 for communicating with other sensors or control units, a microphone 416 for voice activation of different display modes, including, but not limited to, removal of all display items for bright field, one or more speakers 418 for audible alerts and other purposes, and haptic feedback 420 in the form of a shaker motor, piezoelectric buzzer, or other embodiment. The IMU 408 provides additional orientation and positioning data for non-visually based objects. The IMU 408 may be used for, but is not limited to, camera tracking and generation of simultaneous localization and mapping (“SLAM”) data from the IMU 408 data, and may be used to determine non-marker-specific room features that aid in localization and generation of a surface map of an object of interest. Additionally, the sensor suite(s) (400, 210, and 306) include external data 414, such as relayed by wire, wireless, or stored memory. The external data 414 may optionally be in the form of fluoroscopic images, computerized axial tomography ("CAT or CT") scans, positron emission tomography ("PET") scans, and / or magnetic resonance imaging ("MRI") data, etc. Such data may be combined with other data collected by the sensor suite (400, 210, and 306) to create an augmented image.

[0106] During operation of system 10, display generator 412 (also designated 204 and 302) and processing unit 401 (also designated 102) are in electronic communication with the components described above for sensor suite (210, 306). Processing unit 401 is a central processing unit ("CPU") that controls display management and algorithm execution. Referring to FIG. 4, system 10 may optionally include one or more remote sensor suites 422. These remote sensor suites 422 are physically located at a location separate from display device 104. Each of these remote sensor suites 422 includes some or all of the components described above for sensor suite (210, 306), such as a camera 425, an IMU 427, a radio 429, and a cable 431 (e.g., for sharing data with system 400). Also, optionally, a separate remote processing unit 423 may be included. The remote sensor suite 422 contributes data to the external data 414, which may be further processed by the processing unit 401 as desired. In another embodiment, the system 10 uses the remote suite(s) 422 to track not only markers located in the field of view, but also any marker(s) attached to the display unit 104 worn by the user 106, in order to localize objects in the field of view relative to the user 106.

[0107] In one exemplary embodiment, system 10 uses the sensor suite(s) (422, 210, 306) to create a three-dimensional point cloud of data representing objects in the workspace, which can be used to create or match previously modeled objects for use in subsequent tracking, visualization, or later playback.

[0108] Additionally, system 10 may optionally overlay images and masks using art-disclosed means, including but not limited to, retractors or soft tissue surrounding exposures not targeted for treatment, to blur objects in the field of view and help highlight areas, items, or regions of interest. In one embodiment, external images may be projected with the overlay in an augmented reality ("AR") mode. In another embodiment, external images may be ignored and computer-generated graphics alone may be used to display data to user 106 in a virtual reality ("VR") mode. VR mode is supported when display device 104, or portions thereof, are made opaque to block external visual data, or when some other method is used to emphasize to user 106 that they should focus on the image and not the external image.

[0109] Other alternative embodiments of the display device 104 may include, but are not limited to, a holographic or pseudo-holographic display projected into the field of view of the user 106. Additionally, the display device may optionally provide a technology-disclosed means of eye tracking to enable determination of an optimal display image relative to the field of view of the user 106.

[0110] System 10 can optionally use algorithms to identify items within the field of view to identify what constitutes objects of interest versus objects that are not important to the task at hand, which may include, but is not limited to, identifying bony landmarks on the hip acetabulum for use in comparison and merging with pre-operative scans, despite soft tissue and instruments visible in the same field of view.

[0111] Referring to FIG. 5 , one or more cameras 500, 506 and one or more visual markers 502, 504 of the sensor suite (400, 422, 210, and 306) are used to visually track distinct objects (e.g., surgical instruments, desired locations within an anatomical object, etc.) and determine their altitude, location, orientation, and / or position relative to the user 106. In one embodiment, each of the one or more markers is visually distinct and different from one another. Standalone object recognition and machine vision techniques may be used for marker recognition. Alternatively, the present invention also provides assisted tracking using the IMU 408 on one or more objects of interest, including, but not limited to, the markers 502, 504. Note that one or more cameras 500, 506 can be located remotely from the user 106 to provide additional data for tracking and localization.

[0112] An optimal filtering algorithm is optionally used to combine data from all available sources to provide the most accurate position and orientation data for items within the field of view. This filtering scheme will be able to accommodate events including, but not limited to, occlusion of the camera(s) field of view(s), blood, tissue, or other organic temporary occlusion of the desired region of interest, head movement or other camera movement that moves the camera(s) field of view(s) away from the region of interest, data dropouts, and battery / power depletion or other equipment loss.

[0113] 36A-B, 37A-B, 38A-B, and 39-41A-B, another exemplary embodiment of display device 104 is a self-contained AR headset 3600. Previously available systems suffered from several technical challenges or limitations. For example, previously available systems (1) required external sensors, cameras, computers, and / or power sources for full operation of the display device worn by the user, (2) had limited useful life during a procedure due to power source constraints (e.g., the power source was not easily or quickly replaceable during a procedure without experiencing data loss), and / or (3) the self-contained system was not adaptable to various helmets, face shields, or hoods. The self-contained AR headset described herein overcomes these technical challenges with a technical solution. As described in more detail elsewhere herein, the self-contained AR headset of the present disclosure (1) includes all sensors, cameras, computers, and / or power sources necessary to completely perform a surgical procedure (i.e., no external electrical equipment is required), (2) includes a user-replaceable power supply or battery, or a modular battery (i.e., not integrated into the support module, but easily removable and separable from the support module) so that the battery can be easily replaced during the surgical procedure without instruments, operating latches, or data loss, allowing the surgery to proceed without delay, and (3) is easily adaptable to a variety of surgical helmets, hoods, and face shields. Various embodiments of such self-contained AR headsets are now described in more detail.

[0114] The AR headset 3600 is used in various sterile surgical procedures (e.g., spinal fusion, hip and knee arthroplasty, etc.). The AR headset 3600 is clamped to the head of the surgeon 3602 (i.e., user 106) by turning a thumbwheel 3606 and adjusting a head strap 3604. A transparent protective face shield 3608 is optionally attached to the device 3600 by attachment to Velcro strips 3610. Alternatively, attachment may be via adhesive, magnetic, hook, or other art-disclosed attachment means. A coupling mechanism 3612 is present to both mechanically and electrically attach a surgical helmet 3700 to the AR headset 3600. The surgical helmet 3700 is optionally connected to a surgical hood (not shown) that covers the entire body of the surgeon 3602. Full body coverage is useful for certain surgical procedures, such as hip arthroplasty and knee arthroplasty. When the surgical helmet 3700 is attached to the surgical hood, a fan draws air through the surgical hood into an air intake 3702 and circulates it under the surgical hood and helmet, cooling the surgeon 3602 and preventing fogging of the optics. A chin piece 3704 positions the helmet 3700 (and attached surgical hood, if applicable) away from the face of the surgeon 3602. The position of the surgical helmet 3700 relative to the AR headset 3600 is designed to allow an unobstructed view of the surgical site for the surgeon 3602 and all cameras and sensors. The surgical helmet 3700 includes the necessary features to attach and interface with the surgical hood. A flexible cord 3706 connects the AR headset 3600 to a support module 3708, which can be worn on the surgeon's 3602 belt or elsewhere on the surgeon's body other than the surgeon's head. For example, the support module may be worn on the hip, waist, back, shoulder (e.g., using a strap), chest, thigh, wrist, biceps, etc. A replaceable battery 3800 is inserted into the support module 3708.

[0115] Referring to FIG. 39 , the AR headset 3600 includes a display section 3900 having a pair of see-through optical displays 3902 for visual augmentation and one or more tracking cameras 3904 for tracking and stereoscopic functions, including 2D and 3D digital zoom capabilities. The display section 3900 includes a depth sensor 3906 and a structured light projector 3908. The depth sensor 3906 and projector 3908 are preferably located in the center of the display section 3900. A surgical headlight 3909 may optionally be attached to the display section 3900 and electrically connected to the AR headset 3600 so that its brightness can be controlled by the AR headset 3600's software, including by voice command. This feature may be implemented to dim or turn off the surgical headlight when in mixed reality mode, for example, to allow better visualization of virtual content against bright backgrounds. It may also be adjusted to optimize optical tracking, which may sometimes be impaired by high-contrast target lighting or low ambient lighting. In another exemplary embodiment, the operating room lights may be wirelessly controlled by the software of the AR headset 3600 for the same reasons.

[0116] 39-40 , the rear section 3910 of the AR headset 3600 may optionally include heat sinks and other circuit components, such as a microprocessor and an internal battery. The arched bridge section 3912 and head strap 3604 of the AR headset 3600 mechanically connect the rear section 3910 and the display section 3900. A portion of the bridge section 3912 is flexible to accommodate size adjustment. The bridge section 3912 may include wiring or a flexible circuit board to provide an electrical connection between the display section 3900 and the rear section 3910. The bridge section 3912 includes a coupling mechanism 3612, which is a ferromagnetic plate with a plurality of positioning holes 3914 that define openings 3918 providing access to two electrical contacts 3916 for powering the fan of the surgical helmet 3700. In another embodiment, the coupling mechanism 3612 may be other art-disclosed means, such as Velcro®, a latch, or a screw-type fastener. The coupling mechanism 3612 may optionally include anti-vibration mounts to minimize the transmission of mechanical noise from the fan of the surgical helmet 3700 to the AR headset 3600, which may be detrimental to tracking performance. The fan 4004 may be software controlled, allowing it to be slowed or stopped to minimize the generation of mechanical noise. It may also be controlled by the surgeon 3602 using voice commands. A flexible cord 3706 connects the rear section 3910 to a support module 3708, shown in FIG. 38A.

[0117] 40, the surgical helmet 3700 includes a hollow shell 4002 into which air is drawn by a fan 4004 and exhausted through various vents within the shell to provide cool air to the surgeon. A brim vent 4006 provides airflow over the visor of the surgical hood, and a rear vent 4008 provides cool air to the rear, including the rear section 3910 of the AR headset 3600.

[0118] 41A-B, coupling plate 3802 includes a plurality of bosses 4102 for aligning with holes 3914 in AR headset 3600. Coupling plate 3802 also includes spring-loaded electrical contacts 4104 that connect with electrical contacts 3916 in AR headset 3600 to provide power to fan 4004. Coupling plate 3802 further includes magnets 4106 that provide a mechanical holding force between coupling plate 3802 and coupling mechanism 3612.

[0119] Referring to FIG. 60 , another exemplary embodiment of a display device is an eyepiece 6002 that includes a modular bracket 6004 configured to fit onto a headband or other support structure, such as a surgical helmet 3700. Multiple brackets 6004 can be interchanged to attach the eyepiece 6002 to different types of headgear. A focused spotlight or visible light 6006 is integrated to provide illumination to the treatment site and is mounted on the bracket so that it can pivot up and down relative to the eyepiece, allowing both the eyepiece display and the spotlight or visible light to be adjusted to the correct angle for each user, independent of each other. In this embodiment, a handle 6008 is integrated to allow the user to easily adjust the position of the display device, even when worn under a surgical hood.

[0120] To focus the display, it must be positioned at the correct distance and angle relative to the user's eyes. Due to variability in anatomy among users, it is beneficial to provide a means to adjust the position and angle of the eyepiece 6002 for each user. Referring to FIG. 69 , several additional features of the eyepiece 6002 and bracket 6004 that allow for this adjustment are shown. The bracket 6004 is attached to the eyepiece 6002 using one or more attachment mechanisms 6902, such as screws. The bracket 6004 includes a lower bracket 6912 and an upper bracket 6910, which are connected by a locking knob 6904. The upper bracket 6910 further includes a clamp 6908 configured to be securely coupled to a support structure, such as a headband or surgical helmet. In this embodiment, the clamp 6908 is configured to attach the bracket 6904 to a Stryker Flyte surgical helmet. The lower bracket 6912 is securely coupled to the eyepiece 6002. The upper bracket 6910 includes a slot 6906 that interfaces with the locking knob 6904, allowing the lower bracket 6912 and eyepiece 6002 to slide forward and backward when the locking knob 6904 is loosened. The lower bracket 6912 can also pivot about the locking knob 6904 to adjust the angle of the eyepiece 6002. When worn under a surgical hood (not shown), the eyepiece 6002 is located behind a semi-rigid, transparent face shield and can be difficult to reach and manipulate. In this embodiment, a handle 6008 is integrated into the lower bracket 6912, allowing the user to adjust the position and angle of the eyepiece 6002 when worn under the hood.

[0121] Referring to FIG. 71 , the eyepiece 6002 and bracket 6004 are shown attached to a Flyte surgical helmet. The helmet includes a headband 7102 and a duct 7104 connected by a brace 7106. The bracket 6910 and clamp 6908 completely surround the brace 7106 and fit securely against its sides, top, and bottom, preventing angular movement between the bracket components (6908, 6910) and the brace 7106. In this embodiment, the clamp 6908 contacts both the duct 7104 and the headband 7102, preventing the bracket from moving forward or backward relative to the helmet. The bracket 6910 and clamp 6908 are held securely together by two screws.

[0122] Referring to FIG. 61 , components of one embodiment of the eyepiece 6002 include a modular transparent visor 6102 and housing components 6114, 6116, and 6118 for protecting the optical display 3902. The visor 6102 is removable and replaceable without tools for easy replacement in the event of damage or wear. Spring tabs 6120 engage with the bottom housing 6114 to hold the visor 6102 in place. To attach the visor, the user presses the visor into place against the bottom housing. The visor 6102 can be removed from the bottom housing 6114 by lifting the tabs 6120 and peeling the visor off. Multiple optional visor 6102 sizes and shapes allow for an optimal fit for each user, taking into account prescription eyewear use, anatomical variations, and preferences. In one embodiment, the visor 6102 is configured to minimally obstruct the outward view, allowing the user 106 to look below the visor 6102 when not actively viewing information in the optical display 3902. This may be additionally enabled by mounting the eyepiece 6002 high in the user's 106 line of sight. Still referring to FIG. 61 , this embodiment of the eyepiece 6002 includes a stereo camera module 6106, such as an Intel Realsense D435. In one embodiment, the stereo camera module 6106 utilizes an infrared camera, with the camera's viewing axis 7002 angled 20-30 degrees below the display's neutral viewing angle 7004, as shown in FIG. 70 as angle α. In this embodiment, the camera module 6106 is positioned in front of the other internal electrical components, allowing cooling air to pass around the camera module via vents in the lower housing part 6114 and the upper housing part 6118. Positioning the camera module 6106 in front of the display module also brings the camera module closer to the face shield 3608 (shown in FIG. 36B), reducing the effect of light reflections from the face shield 3608.The eyepiece 6002 further includes an infrared light 6108 to provide illumination for the stereo camera module 6106, allowing control over scene illumination independent of ambient room or procedure lighting. In one embodiment, the infrared light 6108 uses one or more dome LED components, such as Lumileds L1I0-0850090000000. One embodiment includes a shroud 6104 with multiple sidewalls 7320 that define an opening 7316 through which light from the infrared light 6108 emits and then shines through the face shield. In some embodiments, the multiple sidewalls 7320 resemble a single sidewall, such that the shroud 6104 includes a conical or continuous sidewall. The shroud 6104 is configured to fit closely to the face shield 3608 to minimize reflection of light from the infrared light 6108 back into the camera module 6106. The shroud 6104 may be formed from or include a front surface 7204 coupled to a border 7310 and may be constructed of a modular structure such that the shroud 6104 is easily replaceable or removable. The shroud 6104 may comprise a monolithic structure. Alternatively, the border 7310 and the front surface 7204 may be bonded, glued, or otherwise secured to form the shroud 6104. The shroud 6104 may be further configured to avoid extending into the field of view of the camera module 6106 based on, for example, one or more of the height of the shroud, the shape of the shroud (e.g., conical, elliptical, circular, etc.), or how the shroud is positioned or positioned in the FOV of one or both tracking cameras. In one embodiment, the shroud 6104 can be removed and replaced without tools, allowing the user 106 to select from multiple shrouds 6104 to optimize contact with the face shield 3608, accounting for variations in the visual acuity and position of the eyepiece 6002 relative to the anatomy of different users. In one embodiment, the spotlight or visible light 6006 includes an infrared light filter to prevent infrared light from the spotlight or visible light from reaching the camera module 6106.Infrared light illuminating the treatment site and reflecting back to the camera module 6106 can also be limited by applying an infrared light filter to the spotlight 6006 to ensure its output is limited to only visible wavelengths. A circuit board 6110 coordinates communication between the camera module 6106 and optical display 3902 and a computer located on the support module 3708.

[0123] Referring to FIGS. 72A and 72B, which show the eyepiece 6002 in its installed position relative to the face shield 3608 (shown transparent for clarity), several features of the shroud are illustrated. FIG. 72A shows a top view of the system, and FIG. 72B shows a side view of the same system. Because the infrared light 6108 and stereo camera module 6106, shown in FIG. 61 as components of the eyepiece 6002, are both behind the face shield 3608, the infrared light 6108 can reflect off the face shield 3608 back onto the camera module 3608 and disrupt marker tracking. This issue is mitigated by the inclusion of a shroud 6104 around the infrared light 6108 that extends to the face shield 3608. In some embodiments, the opening 7316 contacts the face shield 3608, while in other embodiments, the front surface 7204, which is coupled to and / or surrounds the perimeter 7324 of the side walls 7320 of the shroud 6104, contacts or is in close proximity (e.g., 0-5 mm, 0-1 mm, 0-2 mm, 0-3 mm, 0-4 mm, 0-6 mm, etc.) to the face shield 3608, or is adjacent to the face shield 3608, such that light emitted by the infrared light escapes only through the face shield and does not interfere with the camera module. The contact or proximity between any one or more portions of the shroud 6104 and the face shield 3608 prevents infrared light from escaping except through the opening 7316 defined by the side walls 7320 of the shroud 6104, and thus through the face shield 3608. Any reflection of infrared light 6108 from the face shield 3608 is contained within the shroud 6104 and prevented from reaching the camera module 6106. The side walls 7320 of the shroud 6104 may be constructed from, integrated with, coated with, or include a material that has low reflectivity of infrared light at wavelengths discernible to the camera module 6106, such as nylon PA12 or a Cerakote ceramic coating.While the face shield 3608 remains in a fixed position relative to the user's head, the eyepiece 6002 may be adjusted forward or backward to accommodate differences in visual acuity and anatomy, thereby decreasing or increasing the distance from the shroud 6104 to the face shield 3608. To minimize the gap between the shroud and the face shield, various lengths L are available, as shown in FIG. 6104 Multiple shrouds 6104 may be provided, allowing the user to select the longest shroud that will fit behind the face shield for a given position of the eyepiece 6002. The shroud 6104 is held in place by one or more flexible spring tabs 7202 that mate with features on the eyepiece housing. The shroud 6104 snaps into place and can be removed without tools by lifting and releasing the spring tabs. To fit the curved surface of the face shield 3608 with minimal clearance, the shroud 6104 has a front surface 7204 with a radius of curvature that is approximately the same as the radius of curvature of the face shield, as shown in FIG. 72A . In other words, the radius of curvature of the front surface 7204 of the shroud 6014 matches or nearly matches the radius of curvature of the face shield. In other embodiments (without a front surface 7204), the opening 7316 has a radius of curvature that is approximately the same as the radius of curvature of the face shield. In other words, the radius of curvature of the opening 7316 in the shroud 6104 matches or nearly matches the radius of curvature of the face shield, which may be about 0 (flat), about 0 cm to about 4 cm, about 0 cm to about 8 cm, about 0 cm to about 10 cm, etc.

[0124] 73A-73C show perspective, front, and side views, respectively, of the shroud 6104. As shown in FIGS. 73A-73C, the shroud 6104 includes a plurality of side walls that define one or more openings. For example, a plurality of side walls 7320 define an opening 7316 that contains or encloses the infrared light 6108. Additionally or alternatively, a second plurality of side walls 7322 may define a second opening 7314 that contains a second infrared light, camera module, light projector, or other component 7330. In embodiments including openings 7314, 7316, the first and second openings 7314, 7316 are coupled to the module component via a front surface 7204 that is coupled to the interface 7310. The front surface 7204 interfaces with the face shield. In other embodiments, the shroud 6104 does not include the front surface 7204, such that the first and second plurality of side walls 7322, 7320 define the openings 7314, 7316, respectively. Additionally, one or more of the plurality of side walls 7320 may be angled at an angle α measured from the central axis of the infrared light 6108 or the central axis of the cone of light emitted by the infrared light 6108 (e.g., the cone may be substantially or about 90 degrees). 6104 The angle α 6104 may be approximately or substantially 0 to 50 degrees, 0 to 40 degrees, 0 to 30 degrees, 0 to 20 degrees, 0 to 10 degrees, 0 to 5 degrees, 5 to 10 degrees, 10 to 20 degrees, 5 to 20 degrees, 5 to 25 degrees, etc. In one embodiment, the angle α 6104 is substantially or about 12 degrees to about 16 degrees. 6104 is substantially or about 10 degrees to about 18 degrees. In some embodiments, each of the plurality of side walls is angled at the same or substantially the same angle. In other embodiments, opposing side walls have the same or similar angles. In yet other embodiments, each of the plurality of side walls is angled at a different angle than the other side walls.

[0125] Referring to FIG. 62 , which shows an exploded view of one embodiment of the support module 3708, all electronic components are housed or mounted in a housing including a base 6202 configured to receive a circuit board 6212, a coupler 6204 configured to couple the housing to clothing, a strap, a belt, or the like, and a bracket 6206 configured to securely and removably restrain the battery 3800 and processor unit 6210. The battery 3800 may be received in the housing in a fixed orientation; in other embodiments, the battery 3800 is configured to mate with the housing in multiple orientations. The replaceable battery 3800 provides power to the computer module or processor unit 6210 and the AR eyepiece 6002 or head-mounted display device. The bracket 6206 is configured to allow an assistant to replace the battery 6800 without using tools or operating a mechanical latch. The circuit board 6212 is configured to conduct power from the battery 3800 to the computer module or processor unit 6210 and the AR eyepiece 6002. In one embodiment, power and data flow between the support module 3708 and the AR eyepiece 6002 or head-mounted display device via a USB connection. In one embodiment, the computer module or processor unit 6210 is a cell phone with a single USB connector. In one embodiment, the computer module or processor unit 6210 receives power from a battery 3800 via a wireless charger 6208, allowing the USB connector of the computer module or processor unit 6210 to operate as a full-time power source and reducing the chance of it operating as a power "sink."

[0126] Referring to FIG. 63A , which shows an electrical schematic of the support module circuit board 6212, a battery connector 6302 receives power from the replaceable battery 3800, and a DC / DC step-down circuit 6304 steps down the voltage to the nominal system voltage. A DC / DC LDO regulator 6310 ensures that the voltage is at the required level and passes power to the CPU / radio 6308. Power flows to the wireless charger 6312 via a load switch 6314, as directed by the CPU / radio 6308. Power flows to both the phone USB connector 6306 and the headset USB connector 6318 via an NP FET switch 6316. The CPU / radio 6308 monitors the charge level of the battery 3800 and reports the level to the computer module 6210 using wireless transmissions.

[0127] Referring to FIG. 63B, which shows an electrical diagram of the support module circuit board 6212, the USB connector 6320 serves as a power and communications source for the headset when the headset is plugged into the USB connector 6320. Power provided to the headset from the phone 6322 is supplemented by a 12V battery 6324. In this case, the load switch 6326 from the headset to the phone may be disabled by the CPU 6328. In another embodiment, the CPU 6328 detects the absence of the 12V battery 6324 and enables the load switch 6326 from the headset 6320 to the phone 6322. In this embodiment, an external USB charger may be attached to the USB connector 6320 and used to recharge the battery of the phone 6322 as if the devices were directly connected to each other.

[0128] In one exemplary embodiment, the AR headset 3600 is optionally used as a system for reporting device complaints or design feature requests. The user interface may have a menu option or voice command to initiate a report as it occurs. This will activate audio and video camera recording, allowing the user 106 to capture and narrate the complaint in 3D while the problem is occurring. The user 106 ends the complaint by voice or by selecting an option. The complaint recording is compressed and transmitted to the company via the Internet, wirelessly providing superior data to complaint handling staff, allowing them to directly "replay" the situation for better diagnosis. Artificial intelligence can be used to analyze and aggregate the complaint material, establish patterns, and perform statistical analysis. The same sequence can be used to connect to live technical support during the procedure, except the data stream is transmitted in real time.

[0129] II. Preoperative care The present invention can be used in pre-operative tasks and surgical procedures. For example, the following describes an alternative general surgical procedure, including possible pre-operative activities: First, a scan of the patient's area of ​​interest, such as a CT or MRI, is obtained. If possible, the patient is preferably positioned in a manner that approximates their intraoperative positioning. Second, the scan data is segmented and converted into three-dimensional models of the items of interest, including, but not limited to, teeth and bone structures, veins and arteries of interest, nerves, glands, tumors and masses, implants, and skin surfaces. The models are separated so that they can later be independently displayed, labeled, or manipulated. These are referred to as pre-operative models. Third, pre-operative planning is performed using the models (optionally using VR to visualize and manipulate the models) to identify items including, but not limited to, the anatomical frame of reference, the target of the resection plane, the volume to be resected, the plane and level of resection, the size and optimal location of the implant to be used, the path and trajectory for accessing the target tissue, and the trajectory and depth of the guidewire, drill, pin, screw, or instrument. Fourth, the models and pre-operative planning data are uploaded to the memory of the display device 104 before or during surgery. This upload process will most conveniently be performed wirelessly via a radio.

[0130] Fifth, the patient is prepared and positioned for surgery. During surgery, the surgical site is ideally draped in a manner that maximizes visualization of the skin surface for subsequent registration purposes. This can be achieved through the generous use of Ioban. Like Ioban, it would be beneficial to use a film that fluoresces or reflects differently when aimed at with a wide illumination, point, or projection pattern by a specific LED or visible light emitter. This film could also have optical features, markers, or patterns, allowing it to be easily recognized by the optical camera in the headpiece.

[0131] Sixth, after the patient is prepared and positioned for surgery, the system 10 (e.g., via the AR headset 3600) scans the current skin envelope to establish a current contour and create a pre-operative 3D model that the user 106 can view on the display device 104. A preferred method is to project a grid or checkerboard pattern of infrared light (“IR”) bands, which allows the skin envelope to be determined from the calculated warp / tilt / scale of the known image. Another method is to move a stylus-shaped object with attached markers back and forth along the exposed skin, allowing for stylus position and orientation tracking and subsequent generation of the skin envelope. Optionally, the skin model is displayed to the user 106, who can then outline the general area of ​​the scanned exposed skin. The optimal position and orientation of the pre-operative skin model is calculated to match the current skin surface. The appropriate pre-operative model is displayed in 3D to the user 106 via the display device 104. Optionally, user 106 may then insert optical markers into the patient's bones for precise tracking. The placement of these markers may be informed by his visualization of the pre-operative model. The position and orientation of the pre-operative model may be further refined by alternative probing or imaging, including but not limited to ultrasound.

[0132] Seventh, during surgery, a user 106 using the system 10 with the display device 104 can view pre-operative planning information, track instruments and implants, and provide various types of intra-operative measurements, including, but not limited to, drill or screw depth relative to the anatomy, instrument angle, bone cut angle, etc.

[0133] Referring to FIG. 8 , an exemplary embodiment of the operational flow during a procedure using the system 10 is presented. In this embodiment, the CPU 401 starts (800) and initializes (802) one or more cameras 402, 404, 406. When within the field of view of the camera(s) 402, 404, 406, a first marker 100 is located and identified (804), followed by subsequent markers 108, 110 (806). The trajectories of these markers 100, 108, 110 provide their position and orientation relative to each other, as well as the position of the main camera (808). Alternative sensor data (810) from sensors such as IMUs and cameras from the remote sensor suite 422 can optionally be incorporated into the data collection. Additionally, external assistance data (812) regarding the patient, target, instrument, or other parts of the environment can optionally be incorporated for use in the algorithms. The algorithms used in the present invention are tailored to the specific procedure and collected data. The algorithm outputs (814) the desired assist data for use on a display device (816).

[0134] III. Hip Replacement Procedures In one exemplary embodiment of the invention, and referring to Figure 6, system 10 is used in a hip replacement surgery, where a first marker 600 is attached to a pelvis 604 via fixture 602 and a second marker 606 is attached to an impactor 608. A user 106 can view the mixed reality user interface image ("MXUI") shown in Figure 6 via display device 104. The MXUI provides a stereoscopic virtual image of pelvis 604 and impactor 608 in the user's field of view during the hip replacement surgery.

[0135] The combination of markers (600, 606) on these physical objects, combined with pre-processing and specific algorithms, allows for the calculation of measurements of interest to the user 106, including a real-time version of the impactor 608 and tilt angle relative to the pelvis 604 for precise placement of the acetabular shell 612. Additionally, measurements of physical parameters from pre-operative to post-operative states can be presented, including but not limited to, changes in overall leg length. Presentation of data can be in the form of a readable form 610 or an image, including but not limited to, a 3D representation of the instrument or other guidance form.

[0136] 7 depicts another view of the MXUI previously shown in FIG. 6, in which a virtual target 700 and a virtual instrument 702 are presented to the user 106 for easy use in achieving the desired version and tilt. In this embodiment, a further combination of virtual reality is used to optimize the natural-feeling experience for the user by having a virtual target 700 with the real instrument 702 fully visible, or a virtual instrument (not shown) with the virtual target fully visible. Other combinations of real and virtual imagery may optionally be provided. Presentation of data may be in the form of a readable form 704 or images, including, but not limited to, a 3D representation of the instrument or other forms of guidance.

[0137] Referring to FIG. 9 , the present invention further provides a method of performing a hip replacement procedure (900) using system 10, in which the hip bone is reamed from the acetabulum and a replacement cup is inserted for use in the patient's leg. In this embodiment, a first marker (e.g., 100, 108, or 110, etc.) is placed on a fixture of known dimensions relative to the marker, and the fixture is placed on the patient's hip bone (902). A second, separate marker (e.g., 100, 108, or 110, etc.) is placed on a pointing device of known dimensions relative to the first marker (904). The position and orientation of the bony landmarks or other anatomical markers relative to the hip fixture are registered (906) using the optical markers and the position / orientation difference between the hip joint and the pointer. These points are used to determine (908) a local coordinate system. A pointer is used to determine the position and orientation of the femur before it is dislocated and before the acetabulum of the hip bone is reamed to make room for the replacement shell (910). A third, separate marker is attached to the impactor with the replacement shell (912), providing known dimensions for the impactor. The impactor with the shell is tracked relative to the hip marker (914) according to the algorithm previously described. The relative position and orientation between the hip marker and the impactor is used to guide surgical placement of the shell into the acetabulum at the desired position and angle per the patient's medical requirements (916) via an AR or VR display. Leg length change can also be calculated at this point in the procedure (918) using the marker position and orientation of the replaced femur. In another embodiment, preoperative CT data is used to augment this procedure and determine component positioning. In another embodiment, the display output is used with an AR or VR method to determine femoral head resection. In another embodiment, the data is used to place screws in the acetabulum.

[0138] A coordinate reference frame of the operating table or support on which the patient lies is desirable in some embodiments. Alignment of the operating table with respect to the ground, specifically gravity, can be achieved as follows: An IMU (from each of the sensor suites, such as those located within the AR headset 3600) provides the pitch and roll orientation of the display device 104 with respect to gravity at any given moment. Alternatively, a SLAM or similar environment tracking algorithm would provide the pitch and roll orientation of the display device 104 with respect to gravity, assuming that most walls and associated features are constructed parallel to the gravity vector. Separate from the relationship of the display device 104 with respect to gravity, the orientation of the operating table may be determined by registering three independent points on the operating table using a stylus. Selecting these three points in the coordinate frame of the display device 104 can then determine the roll and pitch angles of the operating table with respect to gravity. Alternatively, the operating table may be identified and recognized using a machine vision algorithm to determine its orientation with respect to gravity. The alignment of the patient's spine relative to the display device 104, and therefore the alignment in pitch and roll of any other target coordinate system as defined by the hip markers, is now known. To provide a yaw reference, a stylus can be used in combination with the hip markers to define where the patient's head is, thereby providing the orientation of the spine relative to the patient. Alternatively, this can be determined automatically using image recognition of the patient's head. Finally, the roll, pitch, and yaw of the operating table and / or patient's spine are now fully defined in the display device 104 and all associated coordinate systems.

[0139] 11-12, the system 10 may optionally include a hip joint impactor assembly 1100 for use in a hip arthroplasty procedure. The assembly includes an acetabular shell 1102 and an optical marker 1104 (similar to 100, 108, 110, 502, 504, 600, 606, 804, 806, 904, 912 described above) assembled to an acetabular impactor 1106. FIG. 12 is an exploded view showing how the optical marker 1104 is reproducibly attached to the impactor 1106 by inserting an indexed post 1200 into an indexed hole 1202. The acetabular shell 1102 is reproducibly assembled to the impactor 1106 by threading it onto the impactor's threaded distal end 1204 and seating it against a shoulder 1206. The marker 1104 includes a first fiducial 1108, a second fiducial 1110, and a third fiducial 1112, each with adjacent black and white regions that form a straight line where their boundaries intersect. Algorithms within the AR headset 3600 process images from the stereoscopic camera (3904) to calculate the intersection points of each fiducial (1108, 1110, 1112), thereby determining the 6-DOF pose of the marker 1104. A "pose" is defined herein as a combination of an object's position and orientation. The fiducials (1108, 1110, 1112) may be created by printing on self-adhesive stickers, laser etching black areas into the surface of a white plastic material, or alternative methods. The shell includes fixation holes 1114 through which the shell 1102 is fixed to the acetabular bone, optionally using screws.

[0140] In another exemplary embodiment, referring to FIGS. 13A-B and 14 , the system 10 optionally includes an anatomical marker assembly 1300 with a clamp assembly 1302 and an optical marker 1304. The clamp assembly 1302 includes a base 1400 defining a first teardrop-shaped cavity 1402 and a second teardrop-shaped cavity 1404. A fixation pin (not shown) secured to the bone is inserted through the teardrop-shaped cavity (1402, 1404) and clamped between the clamp jaws 1406 and the body 1400, thereby securing the clamp assembly 1302 to the pin and, therefore, to the bone. A clamp screw 1408 engages the threads of the jaws and is used to tighten the assembly 1302 to the pin. A hex socket 1410 allows a hex driver to be used to tighten the assembly 1302. A first retaining pin 1412 and a second retaining pin 1414 prevent disassembly of the clamp assembly 1302. The marker body 1416 has a first positioning post 1418, a second positioning post 1420, and a third positioning post 1422, which provide positioning for the base 1400 by engaging the two positioning posts with a positioning hole 1424 and a positioning slot 1426 in the base. This design provides two possible rotational positions for the marker 1304, allowing the marker 1304 to be oriented with respect to a camera (e.g., 3904) in the display device 104 (e.g., AR headset 3600) for optimal tracking. The marker body 1416 encapsulates a magnet (not shown) that provides sufficient holding force for the base 1400.

[0141] 15-17, the system 10 may optionally include a calibration assembly 1500 including a plate 1502 and a marker 1504 having a tongue and groove assembly mechanism for joining the plate 1502 and the marker 1504 together. The tongue and groove assembly mechanism is particularly useful for precisely assembling metal parts to plastic parts that have a different thermal expansion rate than the metal parts. The plate 1502 has multiple holes 1506 with multiple thread types to accept various impactor types. The marker 1504 has a dimple 1508 into which the tip of a stylus can be inserted for registration. The marker 1504 has multiple fiducials 1510.

[0142] FIG. 18 illustrates an exemplary embodiment of an MXUI shown to a user 106 via a display device 104 (e.g., an AR headset 3600) showing the calibration assembly 1500 being used for various calibration steps. First, the hip impactor assembly 1100 can be screwed into the appropriate holes in the plate 1502 so that the shoulder 1206 is squarely aligned with the surface of the plate 1502 without any play. The camera 3904 of the AR headset 3600 can then capture images that are processed by an algorithm to determine the relationship between the impactor's shoulder, where the acetabular shell seats, and the markers 1104 on the hip impactor assembly 1100. A stylus 1800 is shown that includes multiple fiducials 1802 for tracking. The tip 1804 of the stylus 1800 is inserted into a dimple 1508 in the plate 1502, allowing the coordinates of the tip 1804 relative to the markers on the stylus 1800 to be determined. A virtual guide point 1806 is shown projected into the field of view of the user 106 at a specific location relative to the marker 1504. The user 106 places the tip 1804 of the real stylus 1800 where the virtual guide point 1806 will be located according to the depth perception of the user 106, thereby connecting the real field of view with the virtual field of view represented by the virtual guide point. An algorithm then applies correction factors to account for variables such as the intraocular distance of the user 106. This is useful when the user's depth perception is relied upon in a mixed reality situation for the precise location of an appliance or implant.

[0143] FIG. 19 depicts an exemplary embodiment of an MXUI shown to a user 106 via a display device 104 of a patient 1900 at the beginning of a hip replacement surgery. A femoral fiducial 1902 having multiple fiducials 1904 for tracking is affixed to the skin of the patient's 1900 thigh with an adhesive tape such as Ioban. Alternatively, the femoral fiducial 1902 may be fixed directly to the femoral bone by use of a pin and clamp assembly as depicted in FIG. 13B . The user 106 uses the tip 1804 of a stylus 1800 to register an anterior marker on the pelvis and determine the position of the pelvis in the reference frame of the femoral fiducial 1902 to establish a temporary pelvic reference frame. In another embodiment, this registration may be within a body reference frame defined by a SLAM scan of the patient's visible surface. In another embodiment, the anterior pelvic markers can be registered by generating a surface map with SLAM and having the user 106 identify each point by positioning a virtual point 1910 on each marker in turn with head movement. In another embodiment, a single fiducial 1906 can be placed at the registration location. A virtual circle 1908 can be used to define a mask whose position is controlled by the user's 106 gaze. The machine vision algorithm need only locate the single fiducial 1906 within the virtual circle 1908. The registration step can be triggered by a voice command from the user 106, such as "register points." The user 106 can also register points representing the distal femur, such as the center of the patella or the medial and lateral epicondyles. As each point is registered, a virtual marker, such as a small sphere, is positioned to remain at the tip during and after registration, providing visual confirmation to the user 106 and enabling them to verify the quality of the registration.

[0144] FIG. 20 shows an exemplary embodiment of an MXUI shown to a user 106 via a display device 104 of a virtual pelvis 2000 and a virtual femur 2002 during a hip replacement procedure. If patient-specific models have been uploaded to the display device 104, these virtual models will be displayed along with other virtual features of interest, such as neurovascular structures. If not, the virtual pelvis and virtual femur may be gender-specific models scaled to best match the spacing of the registered markers. A first virtual trajectory 2004 and a second virtual trajectory 2006 for each of two fixation pins are displayed. In other embodiments, these may be tubular or conical. A drill 2008 is displayed, including multiple fiducials 2010 defining markers on multiple surfaces, allowing its pose to be tracked from various viewpoints. Insertion of each pin can be guided by aligning a real pin 2012 with the virtual trajectory 2004 if the drill is not tracked, or by aligning a virtual pin (not shown) with the virtual trajectory if the drill is tracked. As the drill is tracked, the angle of the drill relative to the pelvic reference frame is displayed numerically for additional reinforcement. Virtual text 2014 is located on the surface 2016 of the actual drill and moves with the drill to give the user an intuitive understanding of the object to which the angle represented by the virtual text is associated.

[0145] FIG. 21 depicts an exemplary embodiment of an MXUI shown to a user 106 via a display device 104 during a hip replacement procedure, with anatomical markers 1300 attached to the patient's pelvis by clamping to pins 2106 inserted into the iliac crest. At this point, the frame of reference for tracking the pelvis is shifted from the previous frame of reference to that of the anatomical markers 1300. If desired, the pelvis may be re-registered for increased accuracy. Next, the user 106 incises the femur using the virtual pelvis 2102, virtual femur 2104, and virtual neurovascular structures (not shown) as guides for the location of the incision, and dissects the muscles and capsule to expose the hip joint and femoral neck. At this point, the user 106 places the leg in a reference position with approximately neutral abduction, flexion, and rotation relative to the pelvis.

[0146] FIG. 22 illustrates an exemplary embodiment of an MXUI shown to the user 106 via the display device 104 during femoral registration for hip replacement surgery. The tip of the stylus 1800 is placed over a reference point 2200 on the proximal femur. At this time, the baseline orientation of the femur relative to the pelvis, defined by the relationship between the markers 1902 and 1300, is determined and recorded. Additionally, the coordinates of the reference point 2200 in the pelvic reference frame are recorded. The reference point 2200 may be reinforced by marking it with a surgical pen, drilling a small hole in the bone, or inserting a small tack. To improve the accuracy of the registration, a magnified stereo image 2202 centered on the tip of the stylus is displayed, as shown in FIG. 22. A baseline image, or an image of the area around the stylus point, may be recorded during registration to assist the user 106 in locating the reference point later in the procedure. These may be stereoscopic images. Next, the user 106 uses the tip 1804 of the stylus 1800 to register a point at the desired location of the femoral neck resection, typically the most superior / lateral point on the femoral neck. An optimal resection plane is calculated that passes through this point at the appropriate abduction and version angles.

[0147] FIG. 23 depicts an exemplary embodiment of an MXUI shown to the user 106 via the display device 104 during resection of the femoral neck for hip replacement surgery using a virtual resection guide 2300. A sagittal saw 2302 is shown, having a number of fiducials 2304 that define markers that allow the pose of the sagittal saw 2302 to be tracked. Resection of the femoral neck can be guided by aligning a real saw blade 2306 with the virtual resection guide 2300 if the drill is not tracked, or by aligning a virtual saw blade (not shown) with the virtual resection guide 2300 if the saw 2302 is tracked. Similar to the tracked drill shown in FIG. 20 , if the saw 2302 is tracked, the angle of the saw 2302 can be displayed numerically. These angles can be displayed relative to the pelvic or femoral reference frame.

[0148] 24 illustrates one exemplary embodiment of an MXUI shown to the user 106 via the display device 104 during positioning of the acetabular shell for a hip replacement, showing a virtual target 2400 and a virtual shell 2402 for the acetabular impactor assembly 1100. Placement of the acetabular impactor assembly 1100 is guided by manipulating it to align with the virtual target 2400. The posterior / lateral quadrants of the shell portion of the virtual target may be displayed in a different color or otherwise visually distinguished from the remainder of the shell 2402 to provide the user 106 with a target for safe placement of screws into the acetabulum. The numerical angle of the acetabular impactor and insertion depth for a reamed or unreamed acetabulum are displayed numerically as virtual text 2404. A magnified stereoscopic image (not shown) similar to 2202 centered on the tip of the impactor may be displayed to show how the virtual shell interfaces with the acetabulum of the virtual pelvis 2102.

[0149] 25 illustrates an exemplary embodiment of an MXUI shown to user 106 via display device 104 during positioning of the acetabular shell for a hip replacement, showing a virtual axis 2500 of the acetabular impactor and a virtual target 2400. Placement of the acetabular impactor is guided by manipulating virtual axis 2500 to align with virtual target 2400.

[0150] 26 illustrates an exemplary embodiment of an MXUI shown to user 106 via display device 104 during femur repositioning and registration for hip replacement surgery. A virtual femur target 2600 is shown, representing the preoperative positioning of the femur relative to the pelvis during baseline femur registration. The high apex of this virtual femur target is located near a reference point on the proximal femur. A virtual femur frame 2602 is shown, representing the current orientation of the femur. As the femur is moved, the virtual femur frame 2602 rotates about the high apex of the virtual femur target 2600. Repositioning the femur to the baseline orientation is achieved by manipulating the femur to align the virtual femur frame 2602 with the virtual femur target 2600 in abduction, flexion, and rotation. With the femur repositioned in the baseline orientation, the user then uses the tip 1804 of the stylus 1800 to re-register reference points on the proximal femur to determine changes in leg length and lateral offset from the baseline measurements. The previously recorded baseline image 2604 may be displayed during baseline femur registration to assist in accurately re-registering the same reference points.

[0151] IV. Tracking and Related Methods In some applications, it may be advantageous to use a camera with a relatively small field of view to maximize tracking accuracy, effectively reducing the size of available pixels. As a result, it becomes more difficult for a user to position the camera so that all necessary markers are within its field of view, particularly because it may not be clear to the user which markers are or are not within its field of view, or which direction the camera(s) should be pointed to capture all necessary markers. FIG. 64 depicts one example embodiment of an MXUI with functionality designed to assist the user in positioning the field of view of the camera(s) to include all necessary markers. In this embodiment, two markers, 6402 and 6408, are required to be tracked by the camera(s) to register points or calculate navigation outputs. One marker, 6402, is positioned within the field of view, 6404, of the camera(s). The second marker, 6408, is outside the field of view, 6404, of the camera(s). A virtual guide 6410 (e.g., a head-fixed object as described elsewhere herein) is displayed to the user 106 on the display device 104, indicating the likely direction in which the missing marker 6408 can be found. The virtual guide 6410 may be a symbol, such as an arrow, or directional text. In one embodiment, the expected location of marker 6408 is based on the relative positions of markers 6402 and 6408, either previously recorded when both markers were visible or estimated by the system based on typical marker placements. In many applications, markers are reasonably expected to move only small distances once set up for a particular procedure. For example, two markers attached to the pelvis and thigh during hip replacement surgery will remain in roughly the same relative positions throughout the procedure. In this case, once the system detects the two markers simultaneously (e.g., they are inertially fixed objects as described elsewhere herein) and measures their relative positions, it can indicate to the user the orientation of the missing marker if either marker is within the camera field of view 6404. Similarly, knowledge of typical anatomy informs the system about likely locations of markers.For example, markers placed by a user on the iliac crest of the pelvis and the anterior aspect of a hip replacement patient's thigh will always be approximately the same distance apart and approximately oriented in the same direction. In a simple example, assuming that the second marker 6408 is positioned approximately along the positive x-axis of the first marker 6402 allows the system to generate a useful virtual guide 6410 that instructs the user to shift the camera field of view 6404 along that axis. In another embodiment, for example, when no marker is in the camera field of view, inertial sensors in the sensor suite are used to track the user's head movement (e.g., head angle), and the relative position of the marker 6408 is calculated based on the last known position (e.g., from the current head position and / or angle) when the marker 6408 entered the camera field of view 6404. In another embodiment, the system calculates the 3D position and orientation values ​​of the marker 6408 when it is in the field of view 6404 and is successfully tracked. The system may track the user's position within the room using visual inertial odometry (VIO), SLAM, or other similar methods. The system also tracks the orientation of the user's head and, therefore, the field of view 6404. If a marker leaves the field of view 6404, the last known position of the marker 6408 can be propagated based on the user's position and the orientation of the display field of view 6404 to generate an estimated position (but still outside the field of view). The estimated position and the currently measured display field of view 6404 are used to present the user with an indicator showing which direction they should face if they wish to have the marker 6408 within the field of view 6404. This approach can be used for multiple indicators representing multiple trackers and is not limited to the case of a single marker. In one embodiment, a virtual control 6406 (e.g., an inertial fixed object as described elsewhere herein) is shown to the user via a display device mounted on the user's head. The user must activate the virtual control 6406 by moving their head to align a fixed reticle or cursor or user input control 6412 (e.g., a head-fixed object as described elsewhere herein) with the virtual control 6406 (e.g., to register a point).In this embodiment, the virtual control 6406 is positioned by the system relative to the marker 6402 so that it is centered between the two required markers 6402 and 6408, and the position of the virtual control 6406 is adjusted as the user turns their head to align the user input control 6412 with the virtual control 6406 until they are aligned. As the user turns their head to align the user input control 6412 with the virtual control 6406, the camera field of view 6404 is moved or adjusted to encompass both markers 6402 and 6408, thereby enabling tracking of at least two markers within the camera's field of view.

[0152] In another embodiment of any of the systems and devices described elsewhere herein, the system presents an information screen or displays content that is locked to a position and / or orientation in inertial space (i.e., inertially fixed). In other words, as the user's head moves or rotates, the content remains in a predetermined position in inertial space, which can result in the content moving out of the user's field of view and no longer being visible through the head-mounted display. To reduce workflow changes and facilitate ease of use by the surgeon, several methods are presented that allow for automatic repositioning of displayed content for the user.

[0153] For example, the system may recenter the displayed content in the yaw direction when the user tilts their head in a predetermined manner. For example, tilting the head (and headset) below a pitch angle of approximately -10 degrees (+ / - approximately 5 degrees) will trigger a recentering of the displayed content. For example, tilting the head (and headset) approximately 3 degrees (+ / - approximately 5 degrees) left or right, such that the user is touching their ear to their shoulder, will trigger a recentering of the displayed content. For example, a gesture combining tilting the head approximately 10 degrees (+ / - approximately 5 degrees) up and approximately 3 degrees (+ / - approximately 5 degrees) left or right will trigger a recentering of the displayed content. This head tilt is not limited to pitch or angle alone. Any head gesture that can be resolved by the inertial measurement system in the headset can be used to trigger this recentering action.

[0154] Further, for example, if a tracking marker is identified by the tracking system and is within the FOV of the tracking system, the system can recenter the display in the yaw direction. In some embodiments, the yaw position can be aligned with the marker or offset from the marker.

[0155] 82, further provided herein is a method for determining markers in inertial space using a head-mounted display and navigation system. For example, some procedures (e.g., reporting the position of an acetabular cup adjusted to correspond to its orientation in inertial space) require measuring the relationship of the marker to inertial space (i.e., relating the marker coordinate frame to the inertial frame). Determining the orientation of a fiducial marker attached to the patient's hip with respect to gravity is measured using an inertial measurement unit (IMU) that is not located on the patient or the marker, but instead is located elsewhere in the head-mounted display and navigation system described elsewhere herein. To achieve this, the following method may be performed by the system: receiving inertial data (e.g., acceleration data, rate data, etc.) from the IMU, head-mounted display, and navigation system in block S8200; determining the position of the gravity vector in the head-mounted display IMU reference frame using an attitude estimator in block S8210; obtaining the orientation of the reference marker in three-dimensional space relative to the camera reference frame using one or more tracking cameras in block S8220; generating a static transformation matrix from the camera reference frame to the IMU reference frame, which may optionally include an intermediate frame transformation, in block S8230; and transforming unit vectors in the Z direction of the inertial space measured by the IMU and formed by the attitude estimation from the IMU reference frame to the marker reference frame in block S8240.

[0156] As used herein, a "pose estimator" combines accelerometer and velocity sensor data using a Kalman filter, complementary filter, or other technique to generate a 3D orientation for the headset, which can be in any output format (e.g., Euler angles, quaternions, or similar).

[0157] In some embodiments, generating the static transformation matrix is ​​based on the mechanical structure of the head-mounted display, the IMU, and a calibration procedure for the camera.

[0158] In some embodiments, the step of transforming the unit vectors includes:

[0159]

number

[0160] 50-52, system 10 may optionally include means for tracking anatomical structures without external fiducials fixed to the anatomy. FIGS. 50A-B depict an exemplary embodiment in which a femur 5002 is dislocated, enabling system 10 to create a reference three-dimensional surface map 5014 of the exposed surface of the lesser trochanter 5010 using sensor suite 210. The surface of the lesser trochanter remains unchanged throughout the procedure and can be used by system 10 to track the femur without additional fiducials. The boundaries of reference three-dimensional surface map 5014 may optionally be indicated by the user by tracing a curve using a cursor or pointing device, which may operate by tracking the user's gaze. System 10 may store reference three-dimensional map 5014 as a point cloud, a mathematical surface, or by other means. The system 10 may create a reference frame 5016 relative to the sensor suite 210 and record the initial pose of the surface map 5014 in the reference frame 5016. The user 106 may register additional reference points or structures on the same bone or rigid body, such as the femoral head 5006, the femoral neck 5008, and the acetabulum 5012. The system may create additional three-dimensional surface maps 5020, 5022, 5024 for the femoral head, femoral neck, and acetabulum, respectively, whose poses the system 10 records relative to the reference frame 5016. The system 10 continuously rescans the lesser trochanter 5010 using the sensor suite 210 to generate a displaced three-dimensional surface map 5018 of the body. The displaced three-dimensional surface map 5018 is then compared to a reference three-dimensional surface map 5014 created for the same surface, and the system 10 determines the geometric rotations and translations required for a best fit between the displaced surface map 5018 and the reference surface map 5014. The system 10 then applies the same rotations and translations to all stored reference points and structures on the rigid body of the femur 5002, and calculates the current pose of all such points and structures relative to the frame of reference of the sensor suite 210. The system 10 may calculate the diameter of the femoral head 5006 or acetabulum 5012 and display it to the user 106 as a guide for selecting an acetabular reamer size.The system 10 may calculate the center of the femoral head 5006 relative to the reference plane map 5014. The system 10 may also calculate the location of the center of the acetabulum 5012 relative to the pelvis 5004. The user 106 then inserts a broach or reamer 5102 with an attached fiducial 5104 into the femoral canal to identify the femoral axis 5106. The system 10 calculates the femoral neck axis 5118 between the femoral head 5006 and the femoral axis 5106. With the knee 5110 flexed approximately 90°, the camera 206 scans the shank 5112 to identify its approximate central axis 5114, which is used together with the femoral axis 5106 to define the reference plane 5116 from which the version angle of the inherent femoral neck axis 5118 is calculated. During the course of treatment, the native femoral head 5006 and acetabulum 5012 are replaced with a femoral implant 5202 and an acetabular implant 5204, respectively. The system 10 may detect the centers of the implanted acetabular shell 5204 and femoral head 5208, allowing the system 10 to calculate and display the change in distance from the femoral axis 5106 to the femoral head 5208 (femoral offset) or the change in location of the center of the acetabulum 5208 between each structure's respective native and implanted states. After replacement of the femoral head 5006 but before replacement of the acetabulum 5012, the system 10 may calculate and display the femoral version based on a new calculation of the femoral neck axis 5206 using the replaced femoral head 5208. The system 10 may calculate and display the additional abduction required at the acetabular implant 5204 to achieve the combined abduction goal of the femoral implant 5202 and the acetabular implant 5204. The system 10 may calculate and display the change in distance between the femur 5002 and the pelvis 5004 that occurs as a result of the procedure.

[0161] 53 shows an exemplary embodiment of a hip joint impactor 5300 that is tracked not by means of supplemental fiducials, but via a three-dimensional map of a portion 5302 of its exposed surface. The system 10 may register the acetabular shell 5304 to this surface by simultaneously scanning the shell 5304 and the impactor surface using the camera 206.

[0162] 59 depicts a flowchart showing how the system 10 and its sensor suite 210 can be used for navigation in hip arthroplasty. The sensor suite 210 can scan the lesser trochanter 5010 (5902). From this scan, a reference three-dimensional surface map 5014 can be stored (5904). The system 10 can then establish (5906) a reference frame 5016 of the femur 5002 relative to the sensor suite 210. The exposed lesser trochanter 5010 is then repeatedly scanned, and the system 10 generates (5908) a displaced three-dimensional surface map 5018 for each scan. With each successive scan, the system can compare the displaced surface map 5018 to the reference surface map 5014 for the same region on the lesser trochanter 5010. Based on this comparison, the system 10 can track (5910) the pose of the femur 5002 relative to the sensor suite 210 by determining the translation and rotation required to best fit the displacement surface map 5018 to the reference surface map 5014.

[0163] FIG. 54 shows a flowchart illustrating how the system 10 and its sensor suite 210 can be used to analyze hip joint kinematics. The sensor suite 210 can scan 5400 the exposed surfaces of a patient's anatomy, including the native femoral head 5006 and acetabulum 5012. From these surfaces, three-dimensional maps 5020, 5024 of each structure can be stored 5402. The system 10 can then rotate the surfaces to the expected orientation for a standing patient and translate them together in the direction of body weight 5404. The system 10 can then calculate the contact point or patch between the two surfaces, which may be a more appropriate center of rotation than the center of the approximately spherical surfaces 5406. Following replacement of the native anatomy with the femoral implant 5202 and acetabular implant 5204, the system 10 can similarly identify the contact points for the implants 5408. Using the implant geometry, the system 10 can perturb the hip joint angle to calculate the angular range of motion allowed in each direction before impingement between the implants or between the implant and bone (5410). The location of the first impingement that limits the range of motion can be highlighted on the display device 104 (5412). For example, the femoral neck 5008 may abut the exposed rim of the acetabulum 5012 or the acetabular implant 5204. If at least one of the impingement surfaces is on the native bone, the user 106 may choose to trim the bone to increase the range of motion. If at least one of the impingement surfaces is on the implant, the user 106 may choose to adjust the position or angle of the implant.

[0164] Referring to FIG. 83, in some embodiments, an end-to-end inertial-to-marker calibration is required via the inertial and vision systems to provide the necessary accuracy when, for example, a combination of inertial data and optical tracking is required during surgical use. For example, determining the hip joint center in knee surgery is one situation where inertial measurements and optical tracking can be used to provide data to the surgeon. Further, for example, determining the hip joint tilt angle in inertial space during placement of a hip acetabular cup is another example where inertial measurements and optical tracking can be used to provide data to the surgeon. As shown in FIG. 83, the method for combining inertial data and optical tracking includes, in block S8300, a step of determining a camera-to-eyepiece rotation matrix from a mechanical design (

number

number

number

[0165] As used herein, "mechanical design" means a drawing that shows how the camera and / or eyepiece are constructed so that the angle between the camera and eyepiece / headset housing can be determined.

[0166] In some embodiments, the calibrating includes using a precision fixture. Further, in some embodiments, the calibrating includes positioning a fiducial marker having a known coordinate system on the fixture, positioning an eyepiece on the fixture to place the fiducial marker within a field of view (FOV), and calculating a rotation matrix from the fiducial marker to the camera.

number

[0167] 83 further includes calibrating the IMU using a precision fixture. For example, the method may include positioning the head-mounted system in a fixture that allows accurate positioning in all three orthogonal positions in both directions (six positions total), positioning the fixture on a horizontal plane with the system inside so that the eyepiece Z axis of the head-mounted system is aligned with the local gravity vector within a tolerance (i.e., the "in front of you" position), acquiring accelerometer data (A1) from all three axes of the IMU output, acquiring velocity sensor data (G1) from all three axes of the IMU output, and repeating for all remaining five positions (i.e., acquiring A2, A3, A4, A5, A6 along with G2, G3, G4, G5, G6 corresponding to "under eye line," "behind eye line," "above eye line," "under right ear," and "under left ear") and calculating the bias and scale factors of the IMU using one or more of the following equations, or alternatively or additionally, using least squares or other techniques:

[0168] Accelerometer bias X=(A2.x+A4.x) / 2 Accelerometer scale factor X=(A2.x-A4.x) / 2 Accelerometer bias Y=(A5.y+A6.y) / 2 Accelerometer scale factor Y=(A5.y-A6.y) / 2 Accelerometer bias Z=(A1.z+A3.z) / 2 Accelerometer scale factor Z=(A1.z-A3.z) / 2

[0169] The method may further include averaging the rate sensor data to arrive at a rate sensor bias value for each of the rate sensors using the following formula: Rate Bias X=(G1.x+G2.x+G3.x+G4.x+G5.x+G6.x) / 6 Rate bias Y=(G1.y+G2.y+G3.y+G4.y+G5.y+G6.y) / 6 Rate bias Z=(G1.z+G2.z+G3.z+G4.z+G5.z+G6.z) / 6

[0170] The method further includes the IMU-to-inertial rotation (

number

[0171] Calibration of these items allows for the reduction of errors when transforming data from an inertial to a reference marker frame, or vice versa.

number

number

number

number

[0172] As used in Figure 83, "tolerance" refers to a threshold level of degrees of pitch and / or roll from the absolute gravity vector. In some embodiments, the tolerance may be about 1 degree, about 0.5 degrees to about 3 degrees, about 1 degree to about 2 degrees, about 0.75 degrees to about 5 degrees, about 2 degrees to about 4 degrees, etc.

[0173] 86A-86B, one embodiment of a fixture 8600 for calibrating a head-mounted display and navigation system 8610 is shown. The fixture 8600 functions to hold the head-mounted display and navigation system 8610 so that the system can be calibrated. The fixture includes a plurality of side walls 8620, each orthogonal to an adjacent side wall. In one embodiment, the fixture 8600 includes six side walls 8620a, 8620b, 8620c, 8620d, 8620e, and 8620f, although other side wall numbers are contemplated herein, such as 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. The fixture 8600 is fixedly connected to the system 8610 such that when the system is positioned therein, the fixture 8600 can be moved to rest on each of the side walls 8620 during the calibration method. One or more side walls 8620 of fixture 8600, e.g., side walls 8620e, 8620b, and / or 8620f, may define a notch 8630 such that a cable 8650 may be connected to system 8610 but does not prevent fixture 8600 from being placed horizontally on a surface when placed on any one of its side walls 8620. Any one or more of the side walls 8620 may further define openings 8660, 8670 through which system 8610 is visible and / or accessible (e.g., to interact with components, user input elements, etc.) when placed in fixture 8600.

[0174] Use of a system linked with a VC arm system FIG. 27 illustrates an exemplary embodiment of an MXUI shown to user 106 via display device 104 during C-arm imaging of a patient. A C-arm imaging system 2700 is shown, including an X-ray source 2702, an imaging unit 2704, and a display unit 2706. A trackable label 2708 is affixed to the C-arm 2700. A virtual hip alignment guide 2710 and a virtual pelvis alignment guide 2712 are shown, which are perpendicular to the anterior surface of the pelvis and centered over the hip joint and pubic symphysis, respectively. Placement of the C-arm 2700 is guided by adjusting the surface of the imaging unit 2704 to align with the appropriate virtual alignment guide. If the C-arm 2700 is trackable, a virtual C-arm alignment guide 2714 may be displayed. In this case, placement of the C-arm 2700 is guided by adjusting the virtual C-arm alignment guide 2714 to align with the appropriate virtual alignment guide 2710 or 2712. The positional and angular misalignment relative to the target may also be displayed numerically as virtual text 2718 .

[0175] FIG. 28 shows a flowchart illustrating how the system 10 and its display device 104 (e.g., AR headset 3600) can be used in conjunction with a C-arm 2700 in a surgical procedure. A camera 3904 (e.g., a high-definition camera, etc.) integrated into the AR headset 3600 can be used to capture an image that is displayed on a C-arm monitor (2800). The image can be adjusted to “align” it to match what would be seen if the camera 3904 were perfectly centered and perpendicular to the image on the monitor (2802). Knowledge of the position of the imager and source relative to the anatomy being imaged can be used to correct the image for magnification and parallax distortion due to divergence of the x-ray beam from the source (2804). The corrected image can then be displayed on the AR headset 3600 (2806). This can then be used to enable the user 106 to take measurements related to the procedure, such as acetabular cup placement or leg length (2808). Other images may be simultaneously displayed, overlaid, mirrored, or otherwise manipulated to allow user 106 to make a comparison, at least as shown in block 2810 of FIG. 28, for example.

[0176] In another embodiment, image capture can be achieved via wireless communication between the C-arm 2700 and the AR headset 3600, e.g., by transferring a DICOM file. Alternatively, algorithms incorporating machine vision can be employed to automatically perform measurements such as acetabular shell tilt and version. Edge detection can be used to trace the shell's contour. The parameters of a best-fit ellipse for the contour can be determined, which can be used to calculate the shell's anteversion from the ratio of the lengths of the short and long axes of the best-fit ellipse. Tilt can be calculated, for example, by placing a tangent to the most inferior aspect of the pubic arch and calculating the angle between the long axis of the shell ellipse and the tangent. Similarly, the relative leg length and lateral offset of the femur can be determined, and changes or discrepancies in femoral abduction can be corrected by identifying the center of rotation from the femoral head or the center of the spherical cross-section of the shell and performing a virtual rotation around this point to match the abduction angle. This type of calculation can be performed almost instantaneously, saving time or the need to take additional radiographic images. Additionally, in another embodiment, the algorithm may correct for the effect of pelvic position errors on the apparent tilt and anteversion of the shell by performing a virtual rotation to match the width and aspect ratio of the radiolucent region representing the blunt hole.

[0177] In yet another embodiment, C-arm imaging can be used to register the position of anatomical structures, such as the pelvis. To do this, the anatomical markers 1300 incorporate radiopaque features of a known pattern and known geometry. The C-arm image is captured, scaled, and displayed on the AR headset 3600 based on the known marker features. A virtual model of the body, generated from a prior CT scan, is displayed to the user 106. The user 106 can manipulate the virtual model and position it so that its contours match the C-arm image. This manipulation is preferably performed by tracking the position and movement of the user's 106's hand using SLAM. Alternatively, the user 106 can manipulate a physical object incorporating markers that move with the virtual model. Once the virtual model is properly registered with the C-arm image, the relationship between the patient's anatomy and the anatomical markers 1300 can be calculated. These steps and operations can also be performed computationally by software using edge detection and matching it to a projection of the model's profile generated from the CT.

[0178] Due to the limited size of available C-arms, it may be difficult or impossible for a user to position the C-arm to image the entire anatomy of interest. For example, a user may want to capture a 14-inch wide image of the pelvis, but only have available a C-arm capable of capturing a 10-inch diameter field of view. This problem is further exacerbated by distortion near the edges of the C-arm image, effectively reducing the usable image size. Algorithms exist for stitching multiple images together based on identifying and aligning common features contained in each image, but these techniques rely on overlap between images to create common features for registration. For example, a user with a 10-inch C-arm would need to acquire at least four (and likely more than four) overlapping images to create an image showing two anatomical features 36 inches apart in their correct anatomical alignment. In another embodiment of the present invention, the system can be used to digitally stitch multiple images from the C-arm 2700 to create an image of a larger portion of the patient 2716 without overlap between the images. For each image captured by the C-arm 2700, the AR headset 3600 uses a tracker, such as label 2708, to measure the corresponding position of the C-arm 2700 relative to the patient 2716. The system then displays the collected images on the display 2706 or the AR headset 3600, with each image in the correct position and alignment relative to a common frame of reference, allowing the user 106 to view and take measurements on a virtual image that includes a portion of the patient 2716 that is larger than could fit in a single image, such as imaging the full pelvis with the C-arm 2700 where the image size is smaller than the extent of the full pelvis, or viewing a single image of the hip and a single image of the ankle in anatomical alignment, etc. This functionality is useful for assessing the alignment and / or length of limbs, spine, etc., while minimizing radiation from the imaging system.

[0179] VI. Spinal Procedures FIG. 31 illustrates an exemplary embodiment of an MXUI presented to the user 106 via the display device 104 during ultrasound spinal registration. Anatomical markers 1300 are fixed to vertebrae adjacent to the surgical site. An ultrasound transducer 3104 is provided, including multiple fiducials 3106 that define the markers. In one embodiment, the ultrasound transducer 3104 is battery-powered, cordless, and capable of wireless communication with the AR headset 3600. Software contains the necessary geometric and other information to position and scale the 2D ultrasound image relative to the location of the markers 1300. The ultrasound transducer 3104 is moved over the surface of the patient 3100 to scan the region of interest. The software combines the 2D image data with six-degree-of-freedom pose information of the ultrasound transducer 3104 relative to the anatomical markers 1300 to generate a virtual model 3108 representing the surface of the vertebrae in the region of interest. The ultrasound transducer 3104 may be rotated relative to the anatomy of interest to obtain a more complete 3D image. The posterior contours of the spinous processes and the left and right mammillary processes can be matched to the same features in the CT-generated 3D model of the vertebrae, allowing for registration and subsequent positioning of the virtual model in the mixed reality view. Alternatively, appropriate features visible in the ultrasound scan can be utilized, or the virtual model's position can be relative to the patient's surface as determined by SLAM. The latter is suitable for procedures where the patient's anatomy of interest remains stationary during the procedure and attaching markers would be unnecessarily invasive or burdensome. Ultrasound can also be used in this manner to generate models of anatomy of interest, such as (but not limited to) bony structures, nerves, and blood vessels. Registration of any anatomy can be achieved. For example, a reference frame for the pelvis can be established with ultrasound, and the proximal apices of the left and right ASIS and the pubic bone can be located. The same method can be used to track the position of instruments and implants percutaneously.

[0180] FIG. 32 depicts an exemplary embodiment of an MXUI shown to a user 106 via a display device 104 during spinal registration using a stylus 1800. Anatomical markers 1300 are fixed to vertebrae adjacent to the surgical site. A virtual model 3200 of the patient's vertebrae, generated from preoperative imaging, is displayed. The virtual model includes a first marker 3202, a second marker 3204, and a third marker 3206. FIG. 33 shows a close-up view of the exposed anatomy shown in FIG. 32. The patient's soft tissue has been sufficiently dissected to expose a first bony prominence 3300, a second bony prominence 3302, and a third bony prominence 3304, which contain the three markers. The user 106 registers the three markers by placing the stylus tip 1804 at a point on the actual vertebra that best matches the location of the markers shown on the virtual model. The software then repositions the virtual model 3200 within the user's field of view to best align these points. The user 106 visually checks the quality of the registration by comparing the virtual model to the actual exposed areas of the vertebrae. If necessary, the user 106 may make adjustments by repositioning the virtual model using the tip 1804 of the stylus 1800. In another embodiment, the markers are arcs traced on the posterior-most aspect of each process. In another embodiment, the contours of the exposed processes are established in SLAM, and the software performs a best fit on the position of the virtual model to match these contours.

[0181] FIG. 34 depicts one exemplary embodiment of an MXUI shown to user 106 via display device 104 during a spinal fusion procedure. A virtual target 3400 for the drill bit and a virtual drill bit 3402 are shown. A virtual vertebra 3404 is shown rendered transparent to the virtual target 3400 and virtual drill bit 3402. The drill bit angle number and the penetration depth or distance from the tip of the drill bit to the maximum safe insertion depth are displayed numerically as virtual text 3406. FIG. 35 depicts a close-up view of the virtual target 3400 and virtual drill bit 3402 shown in FIG. 34. The virtual target 3400 is shown in the form of a rod 3500 having a proximal crosshair 3502 and a distal crosshair 3504. To maintain the actual drill bit on a safe target trajectory, the user must maintain a position where the virtual drill bit 3402 passes through both crosshair rings of the virtual target 3400. The ideal trajectory is achieved when the virtual drill bit 3402 passes through the center of both crosshairs. If the real drill bit moves outside of the safe target trajectory, the virtual target 3400 changes color to alert the user and an audible warning is sounded. The distal crosshairs 3504 are positioned at the planned starting point on the bone surface. The axial lengths of the virtual target 3400 and virtual drill bit 3402 are scaled so that their proximal ends coincide when the drill reaches its maximum planned depth. The scaling for the virtual drill bit 3402 displacement motion is 1:1 when far from the virtual target 3400 but is magnified to a higher factor when closer, allowing for greater precision.

[0182] While this is described in the context of drilling holes with a drill bit, this mixed reality can be used for multiple steps, including tapping the pedicle or driving the pedicle screw, or using the trackable radius to find the pedicle screw tract. When changing drills, taps, or screws, the user places the tip against the marker recess as a means of quickly realigning the axial position. AR guidance can also enable more minimally invasive introduction of implants, such as positioning an interbody cage during a PLIF, XLIF, or TLIF procedure.

[0183] In another embodiment, a surgical drill may be equipped to wirelessly communicate with a headset to provide two-way communication. This may facilitate various safety and usability features, including automatically stopping the drill or preventing operation if the drill is not within a safe target trajectory or has reached a maximum safe depth, and / or providing a convenient user interface for specifying appropriate torque setting parameters for torque limiting applications, such as the maximum insertion torque for a given size pedicle screw or the seating torque of a pedicle screw set screw. The actual values ​​used may be recorded in the patient record for documentation or research purposes, such as the torque curve during drilling, the final seating torque of the pedicle screw or set screw, the implantation position of the pedicle screw, or the specific implant used.

[0184] In another embodiment, the AR headset 3600 may be wirelessly connected to a neuromonitoring / neurolocation system to provide real-time alerts and measurements within the field of view of the user 106 (e.g., a spine surgeon), particularly during minimally invasive procedures such as XLIF. Additionally, when used in conjunction with pre-operative imaging in which the patient's actual nerves are imaged and reconstructed into a 3D model, if the system detects that a particular nerve has been stimulated or that a stimulation probe is approaching, a hologram representing that nerve structure can be highlighted to the user 106 to facilitate avoiding contact with or damage to the nerve structure.

[0185] VII. Knee replacement surgery In another exemplary embodiment of the present invention, and referring to FIGURE 42, the system 10 is used in knee replacement surgery. A knee replacement patient's pelvis 4202, femur 4204, and tibia 4206 are shown in FIGURE 42, and a surgeon 4208 (i.e., user 106) is shown wearing an AR headset 3600. A femoral landmark 4210 and a tibial landmark 4212 are pinned to the femur and tibia, respectively. The femur is moved through a range of motion to determine a center of rotation as a proxy for the hip joint center in the reference frame of the femoral landmark 4210.

[0186] In some embodiments, any of the head-mounted display devices described herein give the user the ability to move so that obstacles to the navigation system can be avoided. This allows for a greater and / or different range of motion of the femur for knee replacement surgery, as opposed to a stationary navigation system that uses a fixed camera, for example, in an operating room. To determine the joint center (e.g., hip joint center), a center-of-rotation least-squares fit (or similar) can be performed, which involves one or more trackers fixed in inertial space and one or more trackers affixed to the bone (e.g., the femur). In one example, as shown in Figure 81, a method for determining a joint center using a head-mounted display and navigation device optionally (shown in dashed lines) includes the steps of attaching one or more fiducial trackers to fixed markers fixed relative to the bones and joints at block S8100, registering points on the bones to a reference coordinate frame at block S8110, creating a bone coordinate frame (e.g., a femur coordinate frame) based on the registered points at block S8120, transforming from the reference coordinate frame to the bone coordinate frame at block S8130, acquiring points of the stationary trackers in the reference frame using a head-fixed head-mounted display and navigation system at block S8140, and determining the joint center in the bone coordinate frame at block S8150. Either a head-mounted display system or a navigation system may be used herein for the method of Figure 81.

[0187] In some embodiments, to limit the number of points or to limit the number of overlapping points, a new point is acquired only if it is separated from the previous point by some nominal distance or other measure. For example, the navigation system outputs a three-dimensional position for each point, such that the navigation system is configured to determine the distance of a point from any other point.

[0188] In some embodiments of the method, the head-mounted display and navigation system may be moved during acquisition to allow tracking of the reference tracker over a larger range of motion, as the user can adjust the system's field of view by moving their head.

[0189] In some embodiments, the points may be transformed to a femoral coordinate system, used to calculate the location of the hip joint center in the femoral coordinate system, processed substantially continuously through a real-time optimal estimation filter to determine the hip joint center, and / or processed as a batch process after final acquisition of all points to determine the hip joint center. For example, processing substantially continuously through a real-time estimation filter may provide feedback to a user, such as a surgeon, that they are approaching a valid solution. In some embodiments, batch processing occurs after a large number of points are collected and attempted, and if processing is insufficient, the user is prompted to try again and the process is repeated.

[0190] The knee is then flexed through a range of motion to determine a baseline, preoperative flexion axis of the knee. The surgeon 4208 then makes an incision to expose the knee joint. The stylus 1800 is used to register the center of the distal femur based on markers, such as the distal-most point of the trochlear groove. The proximal center of the tibia is defined by registering the ACL footprint with the tip of the stylus. For certain minimally invasive procedures, bone landmarks can be registered with the arthroscope by inserting the stylus into the joint capsule through one port and visualizing it with an arthroscope 4214 inserted through a second port. Additionally, an arthroscopic image 4216 from the arthroscope may be wirelessly communicated to the AR headset 3600 and displayed as part of the MRUI. In another embodiment, a trackable arthroscope may incorporate a stylus tip, allowing marker registration to be performed through a single port. The stylus 1800 can then be used to register the medial and lateral malleoli and interpolate these points to determine the center of the ankle in the reference frame of the tibial landmark 4212. At this point, a femoral reference frame is established with a first axis originating at the center of the distal femur and extending toward the center of the hip joint, a second axis defined by the knee flexion axis, and a third axis defined as a normal to the first and second axes. A tibial reference frame is defined with a first axis originating at the center of the proximal tibia and extending toward the center of the ankle, a second axis defined by the knee flexion axis, and a third axis normal to the first and second axes. These reference frames may be presented to the MRUI as virtual images.

[0191] FIG. 43 shows one exemplary embodiment of an MXUI shown to a surgeon 4208 via an AR headset 3600 during knee replacement surgery when the knee is exposed. A local map of the femoral condyles 4302 and tibial plateau 4304 can be generated by scanning with a depth sensor 3906 in the AR headset 3600 or by using a stereoscopic camera 3904 and SLAM. The knee is flexed through its range of motion, and the surgeon 4208 adjusts their viewpoint to visualize the condyles as closely as possible. A circle 4306 in the center of the field of view is used by the surgeon 4208 to “draw” the condyles during the registration process and as a mask for the mapping algorithm. This circle may coincide with the projection field of a structured light projector used to increase the speed and accuracy of mapping. As the surface is mapped, a virtual 3D mesh 4308 of the mapped area can be projected onto the articular surface to guide the surgeon 4208 and provide visual confirmation of the quality of the surface registration. An algorithm is then used to determine the lowest point on the articular surfaces of the distal femur and proximal tibia to determine the depth of the distal femur and proximal tibia resection. From the local map, the ideal implant size can be determined.

[0192] In another exemplary embodiment, the system 10 may use local maps of the femur 4204 and tibia 4206 to track the pose of the bones (4204, 4206) instead of affixing fiducial markers to each bone (4204, 4206). In one embodiment, the user 106 may select regions of the bones (4204, 4206) that remain visible when the knee is flexed and extended. With reference to FIGS. 58A-C, the user 106 may select to map the anterior-medial surface of the tibia 5808 or the anterior-medial surface of the distal femur 5806 to create reference three-dimensional surface maps 5802 and 5804, respectively. These regions are visible through a typical skin incision. Conventional retraction tools and techniques may be used to maintain visibility. The system 10 may store the reference three-dimensional maps 5802 and 5804 as point clouds, mathematical surfaces, or by other means. The system 10 may create a tibia reference frame 5812 and a femur reference frame 5810 relative to the sensor suite 210 and record the initial pose of the surface maps 5802 and 5804 relative to the reference frames 5812 and 5810, respectively. The user 106 may register additional reference points or structures on the same bone or rigid body to the poses that the system 10 records relative to the reference frame 5812 or 5810. The system 10 may successively rescan the same section of the anatomy using the sensor suite 210 to create displaced three-dimensional surface maps 5816 and 5814 for the tibia and femur, respectively. Each displaced surface map 5816, 5814 is then compared to the corresponding reference surface map 5802, 5804 created for the same surface, and the system 10 determines the geometric rotation and translation required to align the displaced surface map and the reference surface map to a best fit. The system 10 then applies the same rotation and translation to all stored reference points and structures on the rigid body of the femur 4204 or tibia 4206 and calculates the current pose of all such points and structures relative to the reference frame of the sensor suite 210.

[0193] 55 is a flowchart illustrating an exemplary method for using the system to navigate a knee replacement procedure. The user 106 first exposes the knee to visualize the bony anatomy 5500. The sensor suite 210 then scans 5502 the anteromedial aspect of the distal femur 5806 and the anteromedial aspect of the proximal tibia 5808. From these surfaces, reference three-dimensional surface maps 5802, 5804 are stored 5504. The system can optionally scan and map larger areas of the femoral condyles 5818, trochlear region 5820, tibial plateau 5822, posterior condyles 5824, or epicondyles 5826. From these augmented surface maps 5828, 5830, 5832, 5834, 5836, respectively, and optionally using external anatomical data, the system 10 identifies 5506 the center of the distal femur 4204 and the center of the proximal tibia 4206. The femur is moved 5508 through a range of motion while scanning the distal femur 5806 to determine the femur's center of rotation about the hip joint as a proxy for the hip joint center relative to the mapped distal femur anatomy 5804. The user 106 then positions the knee in 90° flexion by positioning the lower leg 5112 approximately perpendicular to the femur 4204. With the knee flexed, the system 10 uses its sensor suite 210 to scan the distal femur 5806 and lower leg 5112 to identify their approximate central axes 5114. Alternatively, the system 10 uses its sensor suite 210 to scan the distal femur 5806 and the proximal tibia 5808 as the knee flexes through a 90-degree range of motion to identify the knee's mean flexion axis. The system 10 then establishes (5510) a reference frame 5810 of the femur 4204 relative to the sensor suite 210, with an origin at the center of the distal femur, a first axis extending toward the center of the hip joint, a second axis parallel to the axis of the lower leg 5114, and a third axis defined as normal to the first and second axes. Alternatively, the system establishes a reference frame 5810 of the femur 4204 relative to the sensor suite 210, with an origin at the center of the distal femur, a first axis extending toward the center of the hip joint, a second axis parallel to the knee's flexion axis, and a third axis defined as normal to the first and second axes.The position of the posterior condyles relative to the tibia is recorded, and an axis is constructed between the posterior condyles. The system 10 generates a cross-sectional surface map of the dorsal surface of the foot for purposes of tracking its pose. In another embodiment, the foot may be tracked via markers affixed to the skin or an overlying drape, bandage, or boot. The foot is moved through a range of motion to determine its center of rotation as a proxy for the ankle center relative to the mapped proximal tibia anatomy (5512). A mechanical axis of the tibia is then constructed between the proximal tibia and the ankle center, establishing (5514) a reference frame 5812 of the tibia 4206 relative to the sensor suite 210, with the origin at the center of the proximal tibia, a first axis extending toward the hip joint center, a second axis parallel to the axis of the lower leg 5114, and a third axis defined as normal to the first and second axes. Alternatively, the system establishes a reference frame 5812 of the tibia 4206 relative to the sensor suite 210, with the origin at the center of the proximal tibia, a first axis extending toward the center of the ankle, a second axis parallel to the knee flexion axis, and a third axis defined as normal to the first and second axes. Next, by repeatedly scanning the exposed distal femur 5806 and proximal tibia 5808, the system 10 generates (5516) displacement surface maps 5814 and 5816 for each scan. With each successive scan, the system can compare the displacement surface maps 5814 and 5816 with the original surface maps 5804 and 5802 for corresponding regions on the distal femur 5806 and proximal tibia 5808, respectively. Based on this comparison, the system 10 can track (5518) the pose of the femur 4204 and tibia 4206 relative to the sensor suite 210 by determining the translation and rotation required to align the displaced surface maps 5814 and 5816 with the reference surface maps 5804 and 5802. The system 10 then calculates and displays (5520) the angle and depth of the resection of the distal femur and proximal tibia by simultaneously tracking the respective mapped anatomical surfaces and the cutting instrument or guide. The system 10 may then display (5522) a virtual guide to assist the user 106 in aligning the cutting instrument or guide to the user-defined target angle or depth.The system 10 may suggest an implant size to the user 106 based on the external implant data (5524). Following placement of the implant or trial implant, the system 10 may track the femur and tibia through a range of flexion and measure the relative rotation of the femur and tibia around one or more axes representing, for example, axial rotation or aneurysmal / valvular rotation (5526).

[0194] Optionally, system 10 may use the mapped topography to automatically determine the center of each of the distal femurs 5804 (e.g., by identifying the most distal point on the trochlea or the center of a line passing through the widest part of the condyles) or the center of the proximal tibia 5802 (e.g., by calculating the centroid of the plateau). Optionally, the identification of center points may be supplemented by external data, such as a library of local anatomical maps from which centers have been identified, allowing system 10 to calculate center points when parts of the anatomy are obscured, preventing mapping of the entire surface.

[0195] FIG. 56 depicts a knee with an implanted unicondylar component. One section of each of the femur 5602 and tibia 5604 has been resected. A femoral implant 5606 and a tibial implant 5608 have been placed. In one exemplary embodiment, the system 10 tracks and records the relative motion of the native femur 5602 and tibia 5604. The camera 206 may then be used to scan and map the surfaces of the implants (5606, 5608), and the system 10 may calculate a path of the implant surfaces that follows the recorded tibial-femoral motion. The system 10 may also map the remaining exposed bone 5610 and detect impingement between the implants (5606, 5608) and the bone 5610. A volume representing the overlap between the interferences may be calculated and overlaid as a virtual model on the display device 104. The system 10 may also highlight the impingement site on the display device 104. For example, the femoral implant 5606 may abut a prominence on the tibia adjacent the sagittal resection surface 5610, or the prominence may abut the femur adjacent the femoral implant 5606. If at least one contact surface is bone, the user 106 may choose to trim the bone to change the contact point. If at least one contact surface is on the implant, the user 106 may choose to adjust the position of the implant to reduce impingement.

[0196] Referring to FIG. 57, the system 10, having recorded the native tibiofemoral kinematics, may project the trajectories of the inter-implant contact points 5702 and predefined safety zones 5704 onto the surface of the implant and display them to the user 106.

[0197] 44, a virtual tibial implant 4402 and a virtual femoral implant 4404 may be displayed in an MXUI shown to the surgeon 4208 via the AR headset 3600. The surgeon 4208 can resize and adjust the position of these virtual models until satisfied. In another embodiment, the virtual tibial implant may be displayed during preparation of the tibia for broaching to provide a guide for the rotational alignment of the tibial component.

[0198] 45 , a virtual guide 4502 for pin placement for a tibial cutting block is displayed on the MXUI shown to the surgeon 4208 via the AR headset 3600. A virtual guide 4504 for pin placement for a distal femoral cutting block is displayed. A virtual guide 4506 for pin placement for a four-in-one cutting block is displayed. The actual pin placement is guided by aligning with the virtual guides 4502, 4504, or 4506. The femur 4508 and tibia 4510 can then be resected by placing cutting blocks over these pins.

[0199] 46 illustrates another embodiment of the MXUI shown in FIG. 45, in which a virtual guide 4602 is used to display the ideal resection plane, and the surgeon 4208 may resect the bone directly by aligning a real saw blade with the virtual guide 4602. Alternatively, in the case of a tracked saw 4604, the surgeon 4208 may resect the bone by aligning a virtual saw blade 4606 with the virtual guide 4602. Virtual text 4608 indicating the varus / valgus angle, flexion angle, and depth of each resection may be displayed numerically, if relevant.

[0200] 47 and 49 depict a knee balancing device 4700 that may be optionally included in the system 10, having a base element 4702, a spring 4902, a condylar element 4904, and a condylar plate 4906. The base element 4702 includes a handle 4908, a target 4714, and a tibial plate 4910. The condylar element 4904 includes a handle 4912 and a cylindrical bearing hole 4914. The condylar plate 4906 includes a cylindrical bearing shaft 4916, a target 4716, and two paddles 4706 and 4707. The condylar plate 4906 pivots about the cylindrical bearing 4916, allowing for medial / lateral tilt of the condylar plate 4906 relative to the base plate 4910. In another embodiment, the bearing 4916 may be ball-shaped, allowing for medial / lateral and flexion / extension tilt of the condylar plate 4906. In another embodiment, the condylar plate 4906 may be contoured to match the topography of the bearing surface of the tibial implant. In another embodiment, the design may include two completely separate condylar elements, each with a firmly integrated distraction paddle and marker.

[0201] 47 , the tibial plate 4910 seats on the resected tibia 4704, with the distraction paddles 4706 and 4707 maintaining contact with the medial femoral condyle 4708 and the lateral femoral condyle 4712, respectively. The distraction paddles 4706 and 4707 are biased by a spring 4902 and pivot about an anterior-posterior axis to provide a substantially equal and constant distraction force between each femoral condyle (4708, 4712) and the tibia 4704. The base element 4702 and traction paddles (4706, 4704) contain optical markers (4714, 4716) that allow software to measure the degree of traction on each femoral condyle (4708, 4712).

[0202] As the knee flexes through its range of motion, the position of each target is tracked, as are the poses of the tibia and femur. This data is used to generate a plot of medial and lateral laxity as a function of flexion angle. This information is used to calculate ideal locations for distal femoral cutting block positioning pins to balance the knee throughout its range of motion, and to guide osteophyte removal and soft tissue release to balance the knee throughout its range of motion. This plot may be displayed in an MXUI as shown in FIG. 48, where a first three-dimensional arc 4802 represents medial laxity throughout the knee's range of motion and a second three-dimensional arc 4804 represents lateral laxity. Additionally, a virtual text 4806 may display the actual current knee flexion angle.

[0203] 66A and 66B illustrate one embodiment of the system 10 used to measure resection depth in knee surgery. A distal femur 6602 includes condyles 6604 and 6606 and a mechanical axis 6614. Markers 6608 and 6610 are rigidly fixed to the femur 6602 and the condylar guide 6612, respectively. A marker 6620 is rigidly fixed to the cutting guide 6616.

[0204] A challenge in measuring resection depth is that the femoral condyles, used as depth references, are irregularly shaped, such that their most prominent point changes with the angle of the resection plane. A common solution is to register and map many points on the condylar surface, which is time-consuming but allows a computer to calculate the depth at a particular angle by calculating the distance to the most prominent point along a vertical path. FIG. 67 is a flowchart illustrating a method for registering distal femoral anatomy and measuring depth in knee surgery without mapping the condylar surface using system 10. A user 106 positions a condylar guide 6612 on the condyles 6604, 6606 (block 6700). Following guidance from system 10, the user 106 adjusts the angle of the condylar guide 6612 to the target resection angle while maintaining contact between the condylar guide 6612 and at least one of the condyles 6604, 6606 (block 6702). The system 10 uses the sensor suite 210 to track the markers 6608 and 6610 to measure the pose of the condylar guide 6612 relative to the femur 6602 and records a depth reference plane 6618 that coincides with the surface of the condylar guide 6612 that is in contact with one or more of the condyles 6604, 6606 (block 6704). The system 10 then constructs and records a depth reference point 6622 at the intersection of the mechanical axis 6614 and the depth reference plane 6618 (block 6706). Optionally, the system 10 may instruct the user 106 to adjust the condylar guide 6612 in multiple orientations while keeping the condylar guide 6612 in contact with at least one of the condyles 6604 or 6606 to record additional depth reference planes 6618 and depth reference points 6622 (block 6708).

[0205] In some embodiments, there are additional or alternative ways to guide the user in different directions for depth reference point capture. In one embodiment, as shown in FIG. 74 , a target 7400 (e.g., a bull's-eye) including, for example, one or more regions 7410 is presented with a movable icon 7412 representing one or more angles from the condylar guide. While a circular target is shown, one skilled in the art will understand that any style or shape (e.g., square, rectangular, two-dimensional, three-dimensional, etc.) is encompassed herein. In this embodiment, the system outputs instructions to the user to move the guide relative to the condyle and virtually paint, mark, or highlight 7414 the target 7400 using the movable icon 7412 to capture a valid depth reference point. At angles where a valid depth reference point is recorded, the system displays or outputs an indicator or marker on the target (indicated by the painted region 7414) to notify the user that a valid depth reference point has been obtained for the indicated region and that there is no need to move the movable icon again in that region. In other embodiments, the system restricts the user to virtually painting previously captured valid depth reference points. During the movement of the guide on the condyle, a database is formed of all or a subset of the acquired valid depth reference points for use in the cutting step.

[0206] In another embodiment, as shown in FIG. 75 , a grid 7500 with one or more regions 7510 is presented with a movable icon 7512 representing the angle from the condylar guide. In this embodiment, the system displays output instructions to the user to move the guide relative to the condyle and virtually draw 7514 a target 7500 using the movable icon 7512. At angles where a valid depth reference point is recorded, the system outputs or displays an indicator or marker on the target 7500 (shown by a painted region 7514) to inform the user that valid depth reference point data has been obtained for the indicated region and that there is no need to move the movable icon again in that region. During movement of the guide over the condyle, a database is formed of all or a subset of the valid depth reference points obtained for use in the cutting step.

[0207] In yet another embodiment, as shown in FIG. 76 , a target 7600 is displayed including one or more regions 7602, 7604, 7606, 7608. For example, one or more regions may be located around, adjacent to, or proximate to the outer periphery 7610 of the target 7600. One of these outer regions, e.g., region 7606, is highlighted (e.g., changes color, is activated, lights up, flashes or blinks, emits an audible beep, vibrates, etc.), while the other region, e.g., region 7608, is deactivated (e.g., grayed out). There is a movable icon 7612 that represents the angle from the condylar guide. In this embodiment, the system outputs or displays instructions to the user to move the guide relative to the condyle until the movable icon at least partially overlaps or at least partially overlaps the highlighted region 7606. Once this is achieved, the currently highlighted region 7606 is deactivated, and the next, subsequent, or adjacent region, e.g., region 7604, is highlighted. This is repeated for all regions 7604, 7606, 7608, after which the system outputs or displays instructions to the user to move the movable icon to the central region 7602 of the target 7600. During this operation, depth reference points are acquired in the background. During the movement of the guide over the condyle, a database is built from all the points acquired for use in the cutting step.

[0208] In yet another embodiment, the system displays or presents a moving target to the user and outputs or displays instructions to the user to move the guide relative to the condyle until a movable icon at least partially overlaps or is overlaid on the target highlighted by the system. The speed and pattern of the moving target can be varied by software to acquire data in areas that prioritize accuracy of the device. During the movement of the guide relative to the condyle, a database is built of all points acquired for use in the cutting step.

[0209] Next, the user 106 removes the condylar guide 6612 from the femur and attaches a cutting guide 6616 configured to allow the user 106 to adjust its angle and depth on the femur 6602 (block 6710). As the cutting guide 6616 is adjusted, the system 10 measures the position of the cutting guide 6616 relative to the femur 6602 using the tracking markers 6620 and 6608, respectively. The instantaneous resection depth is calculated as the normal distance from the current resection plane defined by the cutting guide 6616 to a depth reference point 6622 corresponding to a depth reference plane 6618 that is most approximately parallel to the angle of the cutting guide 6616 (block 6712). In another embodiment, the depth reference point 6622 corresponding to the depth reference plane 6618 may be determined by either interpolation or extrapolation of reference planes and / or other depth reference points acquired at different orientations if the current orientation of the cutting guide is not the one recorded during condylar navigation. The user 106 adjusts the cutting guide 6616 to the desired resection angle and depth according to feedback from the system 10 (block 6714). Depth measurement accuracy decreases as the angle from the depth reference plane increases due to the irregular shape of the condyles and uncertainty in identifying the most prominent point on the condylar surface. To minimize depth errors due to misalignment, the system 10 does not display depth measurements if the cutting guide 6616 is more than a predetermined angle limit (e.g., 1 degree) away from the most nearly parallel depth reference plane 6618. Once the cutting guide 6616 is at the desired angle and depth, the user 106 resects the femur by sawing through the slot or against the face of the cutting guide 6616 (block 6716). The angle limit may be selected based on the desired resolution. For example, a 1-degree angle limit may result in an error of approximately 1 mm or substantially 1 mm.

[0210] In some embodiments, the system needs to store reference depth points (i.e., three-dimensional points) at multiple points in the background without requiring the user to manually select many points. To solve this technical problem, one or more reference depth points are stored, located, and associated in a database. The reference depth points are the minimum amount of information required in the database to solve the technical problem. One exemplary, non-limiting embodiment of the database includes a matrix as a grid (e.g., from about -3 to about +3 on both axes, with intervals of about 0.5) overlaid with markers representing measurements of the reference depth points, e.g., VV angles, FE angles, or 2.4 and 1.7, respectively. The database is configured to determine or indicate that the marker or known value in the database closest to the measured value is 2.5, 1.5, and to express that distance (calculated as sqrt((2.5-2.4)^2+(1.5-1.7)^2 or root sum of squares (RSS) distance) as the figure of merit (FOM) for that reference depth point. In other words, the RSS distance is the FOM for that reference depth point. When the reference depth point and associated FOM are stored in the database, a flag or other indicator marks the database index with the known value or closest index (in this example, 2.5, 1.5). If a new lower FOM is calculated for a newly measured reference depth point, the new lower FOM and the newly measured depth point will overwrite the previous entry in the database. If a new, equal, or higher FOM is calculated for the newly measured reference depth point, the new equal or higher FOM is not recorded in the database.

[0211] One exemplary embodiment is shown in FIG. 77. The method of FIG. 77, performed by any of the systems described herein, includes creating a database containing N×M entries in block S7700. Before a condylar guide is used, the system creates an empty database or two-dimensional matrix. The database consists of N×M entries, where N is the number of varus / valgus (VV) angles to be acquired and M is the number of flexion / extension (FE) angles to be acquired. One way to determine N and M is to subtract the minimum desired target angle from the maximum desired target angle while maintaining the maximum point of the array and divide by the resolution required for accuracy. In the example of varus / valgus, the minimum may be approximately -3 degrees, the maximum approximately +3 degrees, and the resolution approximately 0.5 degrees. Further, for a VV minimum of approximately -3.5 degrees and a maximum of approximately +2 degrees, with a resolution of approximately 0.25 degrees, the N value would be 24, including the maximum and minimum endpoints. In this case, an N value of 14 would be obtained, including the maximum and minimum endpoints.

[0212] The method further includes, in block S7710, initializing the database with the target VV angle, the target FE angle, an arbitrary value of the measured VV angle (e.g., it may be any value without a specific unit), an arbitrary value (e.g., it may be any value without a specific unit) of the measured FE angle, an artificially large figure of merit (FOM), a reference point depth vector of (0,0,0), and a reference depth valid flag indicating that the reference depth has been entered and that the reference depth valid flag is already set to false (i.e., false means that no data is stored at this point in the 2D array or matrix). Block S7710 may be based on the user adjusting the condylar guide to the target condylar angle using any of the methods described elsewhere herein. As shown in FIGS. 74-75, the virtually painted areas 7414, 7514 indirectly indicate the valid flags stored in the system.

[0213] Referring to block S7720, the method includes outputting one or both of the current VV angle or the current FE angle of the guide positioned at the target condylar angle.

[0214] In some embodiments, the method includes, at block S7730, determining the closest position to the target VV angle and / or target FE angle in the database. For example, the system may calculate a figure of merit (FOM) representing the root-sum-square (RSS) distance of the current VV angle and FE angle to the target angle calculated at block S7700. In some embodiments, this calculation may be performed by a binary search or linear method. The FOM is equal to SQRT((VV_current-VV_i)^2+(FE_current-FE_j)^2), where i and j are varied to calculate the FOM for all points in the database. The smallest FOM represents the closest point in the database or 2D array or matrix to which the reference depth point should be stored. The i and j indices in the database are stored for subsequent steps.

[0215] The method further includes, at block S7740, if the reference depth has not previously been recorded in the database, calculating the FOM, the current VV angle, the current FE angle, and the depth reference point to be stored in the database at these indices, and setting a reference depth valid flag to true, which indicates that the data for this point in the 2D array or matrix has been successfully stored.

[0216] Alternatively, in block S7750, if the reference depth was previously recorded in the database, one or both of the FOM of the current VV angle or the FOM of the current FE angle is compared with the FOM in the database, and if the current FOM is smaller than the previously recorded reference depth, the reference depth in the database is overwritten. If the current FOM is larger than what is stored, the reference depth is not overwritten.

[0217] Once the cutting guide is attached and a reference depth point is needed for depth calculations, the reference depth point is retrieved using a method such as that shown in Figure 78. As shown in Figure 78, the method includes, in block S7800, determining one or both of the current VV angle or the current FE angle of the cutting guide positioned at one or both of the desired depth and desired angle. For example, this may be based on a user adjustment of the cutting guide relative to the femur.

[0218] The method of FIG. 78 further includes, in block S7810, determining the closest position to one or both of the target VV angles or target FE angles in the database, as described above in FIG. 77. Block S7810 may include calculating an FOM, which represents the root-sum-square (RSS) distance of the current VV and FE angles to the target angles in the database. In some embodiments, this may be performed by a binary search or linear method. The FOM is equal to SQRT((VV_current-VV_i)^2+(FE_current-FE_j)^2), where i and j are varied to calculate the FOM for all points in the database. The minimum FOM represents the closest point in the database array to which the reference depth point should be stored. The indices of i and j in the database are stored for subsequent steps.

[0219] In some embodiments, the method further includes, at block S7820, outputting an indicator indicating that a reference point is unavailable if a valid reference depth point is not recorded in the database or if a valid reference depth point cannot be interpolated (e.g., using a local reference point near the location in the database).

[0220] Alternatively, as shown in block S7830, when a valid reference depth point is recorded in the database or interpolated, an indicator that a valid reference point is available is output. The indicator in blocks S7820 and S7830 may be a visual indicator (e.g., a display on the display of the head-mounted display, a flashing signal, a lighted indicator, a text indicator, a pop-up, etc.), an audible indicator (e.g., a beep, a specific sound, a specific tone, etc.), or a tactile indicator (e.g., haptics or feedback on the head-mounted display, support module, helmet, etc.).

[0221] 79A-79B illustrate one embodiment of a condylar guide 6612. The condylar guide 6612 includes a body 7914 having a first end 7918 and a second end 7916. An elongated handle 7902 extends from the first end 7918 of the body 7914. The user 106 holds the handle 7902 to control the position of the condylar guide 6612. The handle 7902 is of a suitable length to allow the user 106 to make fine angular adjustments and to resist external forces applied to the condylar guide 6612, for example, from pinning the cutting guide 6616 to the femur 6602. The condylar guide 6612 further includes at least one planar surface 7904 (but in some embodiments, two or more) extending from at least a lateral region 7920 of the first end 7918. The planar surface 7904 is configured to rest on one or more femoral condyles 6604, 6606 and establish a zero-depth plane for calculating resection depth. In some embodiments, the planar surface is configured to simulate a plane tangent to the femoral condyles. The condylar guide 7914 includes at least one tracker disposed on the planar surface 7904 to track the pose of the condylar guide 6614. The tracker markings 7906 are fabricated directly on the condylar guide 6612 to enable the system 10 to track the pose of the condylar guide 6612. In another embodiment, the tracker markings 7906 are fabricated on a separate component rigidly attached to the condylar guide 6612. The trackers disposed on the planar surface or on a separate component are used to determine one or more valid depth reference points.

[0222] The condylar guide 6612 further includes a connector 7908 extending from a second end 7916 of the body 7914 and configured to couple to a cutting guide 6616, as shown in Figure 80. In some embodiments, the connector is removable.

[0223] In some embodiments, the body 7914 further defines an aperture 7912 configured to receive a pin 7922 therethrough for insertion into a bone, such as a femur. As shown in FIG. 79B , the aperture 7912 is configured, or the diameters D2, D3 of the aperture 7912 are sized to allow the condylar guide 6612 to tilt when the pin 7922 is inserted through the aperture 7912. The aperture 7912 is oversized to allow the condylar guide 6612 to tilt with the pin 7922 in place. The amount of oversizing can be varied to allow for more or less angular tilt of the guide 6612 about the pin axis. For example, the diameter D1 of the aperture 7912 in the first position can be substantially equal to the diameter of the pin 7922. The diameters D2, D3 of the opening 7912 at the second and third positions may be larger than the diameter of the pin 7922 to allow for angular tilt of the guide 6612 around the pin axis 7924 of approximately + / - 15 degrees, approximately + / - 10 degrees, approximately + / - 5 degrees, approximately + / - 2 degrees, etc., respectively.

[0224] In some embodiments, the condylar guide 6612 includes a release mechanism 7910 extending from the second end 7916 of the body 7914 in a direction opposite the connector 7908. The release mechanism 7910 is configured to couple the condylar guide 6612 to the bone before pinning the cutting guide 6616 to the bone and to remove the condylar guide 6612 after the cutting guide 6612 is pinned to the femur 6602.

[0225] In some embodiments, at least a portion of the second end 7916 of the body 7914 of the condylar guide 6612 defines a slot 7926 configured to receive a slider 7922, with the connector 7908 and release mechanism 7910 inserted on opposite sides of the slider 7922. The slider 7922 allows the user to move the guide 6612 to the correct angle, then slide the cutting block posteriorly (backward) toward the femur to make contact prior to pinning. Contact with the femur supports the cutting guide 6616 during pinning and minimizes the tendency for it to be displaced from the desired angle.

[0226] In some embodiments, pinning of the cutting guide 6616 occurs only after determining one or more effective depth reference points using the condylar guide 6612 coupled to the cutting guide 6616.

[0227] 68A and 68B illustrate one embodiment of a cutting guide 6616 configured to be adjustable after attachment to bone. A fixed base 6806 is rigidly attached to the bone. A movable cutting head 6802 includes a cutting slot 6804. Two valgus adjustment screws 6808 can be turned to adjust the angle of the cutting head 6802 in the frontal plane, while a flexion adjustment screw 6810 can be turned to adjust the angle of the cutting head 6802 in the vertical plane. The valgus adjustment screws 6808 actuate left and right adjustment posts 6814 and 6816, respectively, via interlocking threads. Axial movement of either of these adjustment posts 6814 or 6816 in turn rotates the valgus block 6812 and cutting head 6802 about one of the valgus pins 6818. In one embodiment, the valgus pin 6818 is spaced approximately the same as the femoral condyles 6604 and 6606, allowing the cutting head 6802 to rotate about an axis aligned with one of the condyles so that the distance from the cutting slot 6804 to that condyle remains constant as the user 106 adjusts the angle of the cutting head 6802. This addresses a common problem with existing cutting guides where adjusting the angle of the guide in the coronal plane also changes the depth of resection measured from one or both condyles. With further reference to FIG. 68B , the cutting head 6802 is configured to pivot in the sagittal plane relative to the valgus block 6812 about the flexion pin 6820 when the flexion adjustment screw 6810 is turned, actuating the cutting head 6802 via interlocking threads 6822 incorporated therein.

[0228] FIG. 80 is a diagram of a cutting guide 6616 rigidly attached to a condylar guide 6612. The proximal surface 7904 is shown in contact with the femur 6602. The cutting guide 6616 and condylar guide 6612 are configured such that the distance 8002 between the proximal surface 7904 and the cutting slot 6804 corresponds to a resection depth corresponding to the intended femoral implant, e.g., approximately 9 mm. While 9 mm is typical, a resection depth of approximately 7 mm to approximately 12 mm may also be used. Because the depth is mechanically fixed, the user 106 need only adjust the angle of the assembled cutting guide 6616 and condylar guide 6612 while placing the proximal surface 7904 on the femur 6602. Once the target angle is achieved, the user 106 inserts one or more pins 8004 through the cutting guide 6616 and into the femur 6602 while holding the handle 7902 to prevent the angle of the condylar guide 6612 from changing during pinning. The condylar guide 6612 is then removed. The angle and depth of the cutting guide 6616 relative to the femur 6602 can still be measured and reported by the system 10 by the tracking marker 6620, which is still rigidly attached to the cutting guide 6616. If the position of the cutting guide 6616 is still acceptable, the user 106 then removes the marker and 6620 and resects the femur 6602 through the slot 6804. If the cutting guide 6616 moves during pinning, its angle and / or depth can be adjusted prior to resection.

[0229] VIII. Other Medical Procedures Referring to FIG. 10 , the present invention also provides a method for performing other surgical procedures using the system 10 (specific examples are provided below). The method includes data collection (1000), including but not limited to tracking and recognition of visual markers and IMUs. This data is used to determine (1002) the relative and / or absolute orientation and position of multiple items in a working view. External data (1004) is introduced into an algorithm. The algorithm is used to process (1006) the data for a specific use case and determine (1008) the required output. This data is used in an augmented reality (AR) or virtual reality (VR) output display (1010) to assist medical personnel.

[0230] For example, these methods may be used in total hip replacement surgery. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used to collect data (1000) and determine (1002) the position and orientation of the hip joint and surgical instruments. Algorithms (1006) are used to determine solutions including, but not limited to, component positioning, femoral head resection, acetabular positioning, screw placement, leg length determination, and proper bone positioning within the acetabulum for revision setup.

[0231] These methods may also be used in total knee arthroplasty. Anatomical and instrument markers (e.g., 100, 108, 110, etc.) are used to collect data (1000) and determine (1002) the position and orientation of the knee, tibia, and surgical instruments. Algorithms (1006) are used to determine solutions, including, but not limited to, the location, angle, and inclination of the tibial osteotomy, guide placement and refinement, avoidance of intramedullary guides, and / or refinement of the femoral osteotomy.

[0232] These methods may be used for corrective osteotomy for nonunion of distal radius fractures. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan data for determining malunion and surgical instrument position and orientation (1002). Algorithms (1006) are used to determine solutions, including but not limited to osteotomy location, osteotomy angle, and outcome assessment.

[0233] These methods may be used for revision osteotomies for malunions of the humerus, including the humerus, distal humerus, radius, and ulna, which have fractures that may be complex and require angular and rotational correction. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan data for determining the malunion and surgical instrument position and orientation (1002). Algorithms (1006) are used to determine solutions, including, but not limited to, osteotomy location, angle of cut, degree of correction, and outcome assessment.

[0234] These methods may be used for distal femoral and proximal tibial osteotomies to correct early osteoarthritis and malalignment. Anatomical and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan data or long leg x-ray images to determine (1002) osteotomy location and scale and surgical instrument position and orientation. Algorithms (1006) are used to determine solutions, including but not limited to, osteotomy location, angle of cut, degree of correction, and outcome assessment.

[0235] These methods may be used for periacetabular osteotomy for acetabular dysplasia. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan data to determine (1002) the osteotomy location and surgical instrument position and orientation. Algorithms (1006) are used to determine solutions, including but not limited to, osteotomy location, angle, degree of correction, and outcome assessment.

[0236] These methods may be used for pediatric orthopedic osteotomies similar to the previous embodiments. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan data to determine (1002) the osteotomy location and surgical instrument position and orientation. Algorithms (1006) are used to determine solutions, including, but not limited to, osteotomy location, cut angle, degree of correction, and outcome assessment.

[0237] These methods may be used for elbow ligament reconstruction procedures, including, but not limited to, radial collateral ligament (RCL) reconstruction and UCL reconstruction (Tommy John). Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the location and orientation of isometric points for ligament reconstruction and surgical instruments. Algorithms (1006) are used to determine solutions, including, but not limited to, accurate positioning of tunnel placement and evaluation of results.

[0238] These methods may be used for knee ligament reconstruction, including, but not limited to, MCL, LCL, ACL, PCL, and posterolateral corner reconstruction. Anatomical markers and markers for instruments (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the location and orientation of isometric points for ligament reconstruction and surgical instruments. Algorithms (1006) are used to determine solutions, including, but not limited to, precise positioning of tunnel placement, tunnel depth, tunnel angle, graft placement, and outcome evaluation.

[0239] These methods may be used in ankle ligament reconstruction procedures, including but not limited to, reconstruction procedures to correct instability. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used in data collection (1000), which may be combined with preoperative CT scan or MRI data to determine the location and orientation of isometric points (1002) for ligament reconstruction and surgical instruments. Algorithms (1006) are used to determine solutions, including but not limited to, precise positioning of tunnel placement, tunnel depth, tunnel angle, and outcome evaluation.

[0240] These methods may be used in acromioclavicular (AC) joint reconstructive surgical procedures, including, but not limited to, placement of tunnels in the clavicle. Anatomical markers and markers for instruments (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the location and orientation of isometric points for ligament reconstruction and surgical instruments. Algorithms (1006) are used to determine solutions, including, but not limited to, precise positioning of tunnel placement, tunnel depth, tunnel angle, and outcome evaluation.

[0241] These methods may be used in anatomical and reverse total shoulder replacement (TSA and RSA) surgical procedures, including revision TSA / RSA. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data, to determine (1002) the position and orientation of the humeral head, associated markers, and surgical instruments. Algorithms (1006) are used to determine solutions, including, but not limited to, humeral head cut and articular bone placement, precise positioning of base plates and screws, reaming angles and guide placement for articular bone correction, and evaluation of results.

[0242] These methods may be used in total ankle arthroplasty surgery. Markers (e.g., 100, 108, 110, etc.) are used to collect data (1000) for anatomical markers and instruments, which may be combined with preoperative CT scan or MRI data to determine (1002) the position and orientation of the tibia, fibula, talus, navicular, and other relevant markers and surgical instruments. Algorithms (1006) are used to determine solutions, including but not limited to, accurate positioning of the tibial head resection, determining the anatomical axis, and evaluating the results.

[0243] These methods may be used for percutaneous screw placement for pelvic fractures, tibial plateau, acetabulum, and pelvis, but are not limited to these areas. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the position and orientation of anatomical and other relevant markers and surgical instruments, including screws. Algorithms (1006) are used to determine solutions, including, but not limited to, precise positioning of the bone to receive the screw, surrounding anatomy and soft tissue features to avoid, screw positioning, insertion angle (e.g., for injections), insertion depth (e.g., for injections), and outcome evaluation.

[0244] These methods may be used for in-office injections into areas including, but not limited to, the ankle, knee, hip, shoulder, and spine. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the location and orientation of associated markers and surgical instruments. Algorithms (1006) are used to determine solutions including, but not limited to, precise positioning of injection location, angle, and depth to maximize efficacy and minimize interaction with internal organs and anatomy.

[0245] These methods may be used for the placement of pedicle screws for spinal fusion procedures, including, but not limited to, the lumbar and thoracic spine. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), which may be combined with preoperative CT scan or MRI data to determine (1002) the position and orientation of anatomical and other relevant markers and surgical instruments, including screws. Algorithms (1006) are used to determine solutions, including, but not limited to, precise positioning of the bone to receive the screw, cortical opening, cranio-caudal angulation or the like, midline-lateral tilt, screw insertion trajectory, insertion depth, and outcome evaluation.

[0246] These methods may be used for visualization of alternate spectrum images, including, but not limited to, infrared, ultraviolet, ankle, knee, hip, shoulder, and spine. Markers (e.g., 100, 108, 110, etc.) are used for data collection (1000), including, but not limited to, dual color camera(s) with alternate spectral sensitivity and / or injected dyes for determining the location and orientation of associated markers and surgical instruments (1002) and highlighting patient features to determine the location, position, and type of anatomical features more readily visible in alternate spectra, including nerves, tumors, soft tissue, and arteries. Algorithms (1006) are used to determine solutions, including, but not limited to, precise location of nerves, tumors, soft tissue of interest, arteries, and other features of interest that can be enhanced with this technique.

[0247] These methods can be used for tumor diagnosis, staging, and curative surgical procedures. Anatomical markers and instrument markers (e.g., 100, 108, 110, etc.) are used in data collection (1000), which may be combined with preoperative CT scan or MRI data to determine tumor location and surgical instrument position and orientation (1002). Alternatively, tumor localization relative to the anatomical markers can be performed during diagnostic surgery. Algorithms (1006) are used to determine solutions, including but not limited to, tumor site location and size extent, removal guidance, and outcome evaluation.

[0248] These methods may be used to project visible or invisible, camera-visible light points onto objects within a region of interest, including, but not limited to, bone landmarks, nerves, tumors, and other organic and inorganic objects. Markers (e.g., 100, 108, 110, etc.) are used to augment or replace external datasets for anatomical data and may be used in place of physical pointers or instruments as previously described. Light points may be displayed, such as from a user's head display. Light points may also be displayed as patterns or other light arrangements. These light(s) highlight features on the patient for determining the position and orientation of associated markers and surgical instruments (1002), as well as for augmenting datasets including, but not limited to, fluoroscopy, CT scan, and MRI data. An algorithm (1006) is used to determine the solution described above, but with alternative or additional selection options.

[0249] These methods can be used for minimally invasive implant positioning and percutaneous locking screw insertion. A marker (e.g., 100, 108, 110, etc.) is attached to the proximal end of an intramedullary nail. Another marker (e.g., 100, 108, 110, etc.) is attached to a cross-head screw insertion tool. A virtual model of the nail is displayed, including the target trajectory of the locking cross-head screw. The surgeon can insert the cross-head screw by aligning the virtual cross-head screw with the target trajectory. In another embodiment, the same method can be applied to an external fixation plate. In this case, a virtual locking plate with multiple locking screw trajectories, one for each hole, would be displayed.

[0250] These methods can be used for visualization of ultrasound image data. In one application, the system can assist in needle guidance during a medical procedure, such as injecting an anesthetic. Ultrasound images can assist in visualization of the needle, but not before it enters the ultrasound field of view within the tissue; by that time, its trajectory is already established and cannot be adjusted without causing pain to the patient. The system of the present invention can assist in tracking the needle both before and after insertion. Referring to FIGS. 10 and 31 , a fiducial 3106 is attached to an ultrasound transducer 3104. As a user 106 uses the ultrasound transducer to acquire 2D images of the patient's internal anatomy, the system 10 simultaneously tracks the position and orientation of the ultrasound transducer 3104 and receives the 2D ultrasound images 1004. The system 10 can optionally and / or additionally track the patient 1900. The system 10 then combines the 2D image of the patient with the position and orientation of the ultrasound transducer 3104 relative to the patient, reconstructs a 2D image in a common frame of reference using the acquired ultrasound transducer and patient position and orientation data, and displays the reconstructed or 3D image to the user 106 in the AR headset 3600. The system 10 can also use image analysis algorithms 1006 to generate and display a surface or solid model 1008 created from anatomical structures identified in the image data. The system 10 can optionally display a virtual instrument superimposed on the 3D imaging data based on the tracked positions of one or more physical instruments, such as a needle. Because the accuracy of the 3D reconstruction is subject to errors such as magnification discrepancies due to the speed of sound in various tissues, the relative position of the virtual instrument may be imperfect. However, once the needle is in the ultrasound field of view, its positional accuracy is improved because it is directly visualized in the image. At this point, 3D reconstruction of the needle is useful for localizing the needle tip, which is difficult to distinguish from random cross-sections in standard 2D images. Knowing the location of the needle tip, as well as the needle shaft, assists the user in inserting the needle to the desired depth without damaging adjacent tissue.The system 10 keeps track of the position and orientation of the probe (e.g., needle, syringe, pin, screw, etc.) and displays the axis (e.g., along the axial length of the probe) and / or the position of the probe's tip relative to a 3D image of the patient's internal anatomy. The axis may be, for example, a virtual axis of the probe or a graphical representation of the probe. The tip of the probe is then advanced to a desired position based on its position relative to the patient's internal anatomy. Optionally, as shown in FIG. 31 , the patient's external surface is mapped using a stereo camera and displayed in relation to the 3D image of the patient's internal anatomy and / or ultrasound transducer 3104.

[0251] IX. Traceable Instruments and Equipment Database The present invention optionally includes building an electronic database of instruments and devices to enable the AR headset 3600 to identify what instruments are present in a surgical or operating room area. Referring to FIG. 29 , to facilitate building such a database, a serialized tracking label 2900 is optionally included in the system. The serialized tracking label 2900 includes a machine-readable serial number code 2902, a human-readable serial number 2904, and a set of optical features that facilitate six-degree-of-freedom optical pose tracking, such as a plurality of fiducials 2906. In one embodiment, the machine-readable number code 2902 pattern can be imaged by the camera(s) 3904 of the AR headset 3600 and used alone to determine the pose and position of the medical instrument using machine vision algorithms. In another embodiment, the serial number image 2904 can be imaged by the camera(s) 3904 and used alone to determine the pose and position of the medical instrument using machine vision algorithms. In yet another embodiment, the entire physical model of the tracking label 2900 may be imaged by the camera(s) 3904 and used alone to determine the pose and position of the medical instrument using machine vision algorithms. In another embodiment, the tracking label 2900 may comprise or include a wireless RFID tag for non-optical identification of the equipment in the kit, which can then be automatically verified using optical recognition.

[0252] Referring to FIG. 30 , a flowchart is provided illustrating a system for registering equipment element types and physical parameters and storing and sharing this data for use in surgery using an augmented reality headset. In this exemplary embodiment, serialized, trackable labels are pre-printed on a durable, self-adhesive material. The labels are affixed (3002) to an element of equipment (3000), which may be, but is not limited to, a C-arm, impactor, pointer, or any other equipment used in the procedure, in a location that will be most advantageously viewed during surgery or in preparation for surgery (i.e., back-table surgery). The labels are then registered (3004) by viewing with camera(s) 3904, identifying the label and initiating a database record associated with its serial number. Shapes of interest associated with the equipment element may also be registered (3006) with and stored in association with the trackable sticker. For example, in the case of a C-arm, a registration stylus may be used to register three points around the perimeter of the imager face and a point representing the origin of the X-ray beam source. This provides the coordinate frame, orientation (pose) data, and position data of the X-ray beam source relative to the AR headset 3600 coordinate frame for use by the AR headset 3600 algorithms. In one alternative embodiment, the camera 3904 is a stereo camera and is used to scan and recognize the C-arm geometry by recognizing key features such as the cylindrical or rectangular surface of the imager. Additional relevant specifications (3008) about the equipment elements may be entered into the record, including, but not limited to, the equipment type and model, calibration date, electronic interface parameters, and wireless connection password. An image of the device is captured (3010) with the camera(s) 3904. An image of the device's equipment label (3012) is captured. All of these items are added to a completed record (3014) that is now local to the AR headset 3600. The record is then time-stamped and shared with a central database (3016).This may be located on a local server within the hospital system or any remote server, including any cloud-based storage over the Internet. Database upload may occur via Wi-Fi common network protocols or other art-disclosed means. The above operations may be performed by company personnel, hospital-employed technicians, or other trained individuals. Administrative privileges may be required to import records to prevent poorly registered devices from entering the database.

[0253] When an item of equipment is used in surgery, the camera(s) 3904 are utilized to recognize the label as a trackable item of equipment and read the serial number (3018). The AR headset 3600 can then connect to a database (3020) and download the equipment record (3022). In this manner, the equipment can be used in a six-degree-of-freedom trackable manner during surgery (3024). If applicable to equipment with a data label, the record (3026) may also be updated with data specific to the equipment itself, such as uploading images captured by the equipment during surgery or capturing a log of the equipment's activity during surgery. Log entries describing the equipment's use in surgery can be added to the database and to patient records indicating the equipment's use. The database thus generated can be mined for various reasons, such as searching for faulty equipment usage.

[0254] The system can also be used to recognize surgical instruments and implants encountered during surgery. A database of scaled CAD models of instruments and devices is kept in memory. During surgery, SLAM or similar machine vision algorithms can capture the topography of items in the scene and compare it to the database of instruments and devices. If there is a match, the system can take appropriate action, such as tracking the position and orientation of the instrument relative to the patient and other instruments used in the surgery, or entering a mode related to the use of that instrument. For example, in a hip replacement surgery, if an acetabular impactor is detected, a mode for cup placement navigation will be entered.

[0255] The system may also use knowledge of the current software workflow step to provide applicable instructions to OR staff, such as a surgical technician. The instructions may be displayed on a remote monitor or a second AR headset 3600 networked with the surgeon's system. For example, the system may display information about the next step in the workflow and instruct the surgical technician or assistant on which instruments to prepare, optionally including image, video, or audio instructions for locating, identifying, or assembling the required instruments. The system's camera may be used to identify a specific instrument or set of instruments and instruct the assistant on the required instruments via the AR headset display. The surgeon or other expert user can optionally input custom instructions to be displayed to the assistant or staff for each step in the surgical workflow.

[0256] The system may also be used to optimize implant selection and / or placement based on outcome data or common practice. FIG. 65 is a flowchart illustrating an exemplary method for using the system to support surgical decision-making. The system first scans and maps the native anatomy using the sensor suite 210 (block 6502). Optionally, the anatomical data may be augmented or replaced with preoperative imaging, such as CT or MRI. The system then compares the anatomy to a database, identifying cases with similar anatomy, and outputs the implant type, alignment, and component positioning (block 6504). Alternatively, or additionally, the system outputs the implant type based on one or more shape-matching algorithms that match one or more characteristics of the anatomy with a best fit in a database of known implants. The one or more characteristics of the anatomy used in the one or more shape-matching algorithms may be based on intraoperative imaging scans, as opposed to preoperative imaging scans. The user 106 proceeds to navigate and complete the surgery once the system 10 records the actual alignment and positioning data (block 6506). The system 10 proceeds to record the implant type and size selected by the user 10, either by automatic scanning by the sensor suite 210 or by manual entry (block 6508). The surgical data, if available, is uploaded to a database containing the surgical results (block 6510). The updated database is used to inform the next case in block 6504. Suggestions may be based on the desired surgical outcome, if available in the database, or on common practices by the same user 106 or other users in similar situations. Other data may be collected intraoperatively, including data regarding procedure time and instrument use. The sensor suite 2010 may use machine vision algorithms to automatically identify instruments intraoperatively and record which instruments were used in each procedure and when they were used.Hospitals may use this information to efficiently package instrument sets to include the most commonly used instruments, or to train or instruct staff on when particular instruments are needed in a procedure. While the present invention has been shown and described herein with reference to preferred embodiments and specific examples thereof, those skilled in the art will readily recognize that other embodiments and examples may perform the same function and / or achieve similar results. All such equivalent embodiments and examples are within the spirit and scope of the present invention and are intended to be contemplated thereby and covered by the following claims.

[0257] X. Use of Audio and / or Gaze in the Systems and Methods Hereof As shown in FIG. 84 , in some embodiments, any one or more of the devices or systems described herein may implement gaze control. As used herein, “gaze” refers to a process in which an augmented reality virtual object is positioned and fixed in inertial space (i.e., an inertial fixed object 8400) but can also be interacted with by a user. The user can rotate their head or move around a room, and one or more inertial fixed objects 8400 remain fixed in the local environment or inertial space. In some embodiments, a reticle 8410 is displayed in the center or central area of ​​the eyepiece display. When the user moves this reticle 8410 so that it visually lines up with one of the virtual objects 8420, the head-mounted display and navigation system are configured to activate the virtual object 8420. As an example, the system presents a virtual object, such as a button with a light bulb. The virtual object is fixed in inertial space in the local environment or the environment surrounding the user. The system may prompt the user to orient their field of view so that the light bulb object is aligned with the reticle in the center of the eyepiece. An interaction between the reticle and the virtual object is detected, and the virtual object is activated, e.g., a headlamp is turned on (or off) based on this control input from the user (alignment of the reticle with the virtual object).

[0258] This gaze control can be used for many interactions, including but not limited to user input selections (e.g., button selections, on or off controls, slider controls, etc.), alphanumeric input (e.g., through selections on a virtual keypad), etc.

[0259] However, it may be more beneficial to have a head-fixed display that always displays content regardless of head position. This is called a "head-fixed" object, i.e., a virtual object that moves in sync with the movement of the head-mounted display. The reticle described in Figure 84 is a head-fixed object, while the virtual objects 8400, 8420 shown in Figure 84 are inertially fixed objects.

[0260] Head-fixed versus inertial-fixed objects can be managed for simultaneous use in a surgical procedure. Specifically, an inertial screen is used to control the system using information data and gaze control. However, when tracking a target, a "head-fixed" video screen showing the navigation camera's tracking scene is displayed in addition to the inertial screen. The inertial screen can be positioned so that the controls are very close to the surgical site. When the surgeon looks at the surgical site, a head-fixed screen displaying the tracking content is displayed. Because gaze control is inertially located in the same field of view, the surgeon can control the system with minimal head movement.

[0261] Additionally, as shown in FIG. 85, in some embodiments, any one or more of the devices or systems described herein may implement voice recognition. As used herein, "voice recognition" is the process of taking audible sound and processing it to recognize speech. This may include words like "go" or "stop." Simultaneous use of gaze control and voice recognition reduces the surgeon's reliance on gaze control (and, consequently, undesirable head movements at certain times during surgery). Activating both voice recognition and gaze control reduces the risk that exclusive use of one or the other would result in poor quality interaction for the surgeon (i.e., voice recognition being unable to determine his commands or gaze control being cumbersome for long procedures).

[0262] For example, in one embodiment, the system may prompt a user, such as a surgeon, to use a reticle or virtual control 8410 to select a user input element or virtual object 8420, such as a "forward" button. The system is configured to accept gaze-control-based input from the user and display a screen, window, or other indicator 8430 prompting the user to say "go," say "stop," or gaze at a button to start tracking, etc. In some embodiments, the virtual object 8420, such as a button, may also include a label 8440, such as text that says "go." The user is prompted by the system to either gaze at the virtual object 8420 or say the word "go" aloud to activate tracking. Once tracking begins, the virtual object label 8440 changes to "stop" or to another indicator, such as red. The system may then prompt the user to gaze at the virtual object 8420 to deactivate tracking or to say the word "stop."

[0263] Unless otherwise specified, the dimensions and geometries of the various structures depicted herein are not intended to limit the invention, and other dimensions or geometries are possible. Multiple structural components may be provided by a single, integrated structure. Alternatively, a single, integrated structure may be divided into separate components. Furthermore, while features of the invention may be described in the context of only one of the illustrated embodiments, such features may be combined with one or more other features of other embodiments for any given application. It will also be understood from the above that the fabrication of the unique structures herein and their operation also constitute methods in accordance with the invention.

[0264] The systems and methods of the preferred embodiment and variations thereof may be embodied and / or implemented, at least in part, as a machine configured to accept a computer-readable medium storing computer-readable instructions. The instructions are preferably executed by one or more portions of the processor(s) and / or computer-executable components integrated with the computing device within the system and supporting modules. The computer-readable medium may be stored on any suitable computer-readable medium, such as RAM, ROM, flash memory, EEPROM, optical device (e.g., CD or DVD), hard drive, floppy drive, or any suitable device. The computer-executable component is preferably a general-purpose or special-purpose processor, but may alternatively or additionally execute instructions on any suitable dedicated hardware or hardware / firmware combination.

[0265] As used in this specification and claims, the singular forms "a," "an," and "the" include both singular and plural references unless the context clearly dictates otherwise. For example, the term "sensor" can include, and is intended to include, a plurality of sensors. At times, the claims and disclosure may include terms such as "plurality," "one or more," or "at least one," but the absence of such terms is not intended, nor should it be interpreted, to mean that a plurality is not intended.

[0266] When used before a numerical designation or range (e.g., defining a length or pressure), the term "about" or "approximately" indicates approximation, which may vary by (+) or (-) 5%, 1%, or 0.1%. All numerical ranges recited herein are inclusive of the recited beginning and ending numbers. The term "substantially" refers to the majority (i.e., greater than 50%) or essentially all of a device, material, or composition.

[0267] As used herein, the terms "comprising" or "comprises" are intended to mean that the devices, systems, and methods include the recited elements and may further include any other elements. "Consisting essentially of" is intended to mean that the devices, systems, and methods include the recited elements and exclude other elements that are essential to the combination for the described purpose. Thus, a system or method consisting essentially of the elements defined herein will not exclude other materials, features, or steps that do not materially affect the basic and novel characteristic(s) of the claimed disclosure. "Consisting of" is intended to mean that the devices, systems, and methods include the recited elements and exclude more than insignificant or inconsequential elements or steps. Embodiments defined by each of these transitional terms are within the scope of this disclosure.

[0268] The examples and illustrations contained herein are illustrative, not limiting, of specific embodiments in which the subject matter may be practiced. Other embodiments may be utilized and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of the present disclosure. Such embodiments of the inventive subject matter may be referred to herein, individually or collectively, by the term "invention" for convenience only, and are not intended to spontaneously limit the scope of this application to any single invention or inventive concept when multiple inventions are in fact disclosed. Thus, although specific embodiments have been shown and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiment shown. The present disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above embodiments, as well as other embodiments not specifically described herein, will be apparent to those skilled in the art upon reviewing the above description. [Explanation of symbols]

[0269] 1 processor 10 Sensory Augmentation System 100, 108, 110 Visual markers 102 Processing Unit 104 Display Devices 106 users 206 Tracking Camera 402 Camera 1 404 Camera 2 406 Camera N 408 Inertial Measurement Unit 410 Radio 416 Microphone 418 Speaker 420 Haptic Feedback 1100 Hip Impactor Assembly 1102 Acetabulum shell 1104 Optical Marker 1106 Acetabulum impactor 1108 First Fiducial 1110 Second Fiducial 1112 Third Fiducial 1300 Anatomical Marker Assembly 1302 Clamp Assembly 1304 Optical Marker 1400 base 1416 Marker body 3600 AR headset 3800 replaceable battery 3904 Stereoscopic Camera 6210 Computer Module 6212 Support Module 6302 Battery Connector 6304 DC / DC step-down circuit 6306 Telephone USB Connector 6308 CPU / Radio 6310 DC / DC LDO Regulator 6312 Wireless Charger 6314 Load Switch 6316 NP FET switch 6318 Headset USB Connector

Claims

1. 1. A self-contained surgical navigation system configured for use with a helmet and a face shield, comprising: A head-mounted display device worn by a user during surgery, comprising: a display generator for generating a visual display on the display device; a sensor suite having at least one tracking camera; Visible light and Infrared light and a head-mounted display device comprising: a processor unit configured to receive data from the sensor suite and to calculate the position and orientation of at least one marker; a shroud disposed around the infrared light source and including a plurality of side walls defining an opening through which light from the infrared light source is emitted; the at least one tracking camera, the visible light, and the infrared light are positioned behind a face shield when the head-mounted display device is attached to a helmet; and A self-contained surgical navigation system, wherein the plurality of side walls contact the face shield when the head-mounted display device is attached to the helmet to prevent light emitted by the infrared light from being reflected back to the at least one tracking camera and passing only through the face shield.

2. 10. The system of claim 1, further comprising an infrared light filter coupled to the visible light, the infrared light filter preventing the visible light from emitting infrared light when coupled to the visible light.

3. and at least two markers affixed to the object of interest for tracking the object of interest, a first marker being within a field of view of the at least one tracking camera and a second marker being outside a field of view of the at least one tracking camera; The processor unit further comprises: determining a position of the first marker within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to guide the user to a position of the second marker relative to the first marker; and determining a position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

3. A system according to any one of claims 1 to 2.

4. Further comprising a support module, the support module comprising: a user-replaceable modular battery removably inserted into the housing of the support module; a processor unit configured to receive data from the sensor suite and to calculate a position and orientation of at least one marker; the support module is electrically coupled to the head-mounted display device to provide power and data to the head-mounted display device, the support module being worn on the user's body at a location other than the user's head; 4. The system of claim 1, wherein the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electrical equipment.

5. 5. The system of claim 1, wherein a front surface coupled to the side walls is adjacent to the face shield and has a radius of curvature that generally matches a radius of curvature of the face shield.

6. The system of claim 1 , wherein one or more of the side walls are angled at an angle of about 10 degrees to about 20 degrees relative to a central axis of the infrared light.

7. 1. A self-contained surgical navigation system, comprising: A head-mounted display device worn by a user during surgery, comprising: a display generator for generating a visual display on the display device; a head-mounted display device including a sensor suite having at least one tracking camera; A support module comprising: a user-replaceable modular battery removably inserted into the housing of the support module; a processor unit configured to receive data from the sensor suite and calculate a position and orientation of at least one marker; the support module is electrically coupled to the head-mounted display device to provide power and data to the head-mounted display device, the support module being worn on the user's body at a location other than the user's head; A system wherein the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electrical equipment.

8. The system of claim 7 , further comprising one or more of a face shield and a helmet, wherein the display device is mounted to the face shield or the helmet.

9. 9. The system of claim 7, wherein the head-mounted display device further comprises an infrared light.

10. 10. The system of claim 7, wherein the head-mounted display device further comprises a visible light and an infrared light filter coupled to the visible light, the infrared light filter preventing the visible light from emitting infrared light when coupled to the visible light.

11. a shroud disposed around the infrared light and including a plurality of side walls defining an opening through which light from the infrared light is emitted; when the head-mounted display device is attached to the helmet, the at least one tracking camera and the infrared light are disposed behind the face shield; 10. The system of claim 7, wherein the plurality of side walls are in close proximity to the face shield when the head-mounted display device is attached to the helmet to prevent light emitted by the infrared light from being reflected back to the at least one tracking camera and to pass only through the face shield.

12. at least one marker affixed to the object of interest for tracking the object of interest, the at least one marker being outside the field of view of the at least one tracking camera; and the processor unit further tracking the angle of the user's head using one or more sensors in the sensor suite; calculating a relative position of the at least one marker based on a last known position of the at least one marker when the at least one marker was located within a field of view of the at least one tracking camera, the last known position being relative to an angle of the head; 12. The system of claim 7, configured to display a virtual guide to the user on the display device to guide the user to the location of the at least one marker.

13. and at least two markers affixed to the object of interest for tracking the object of interest, a first marker being within a field of view of the at least one tracking camera and a second marker being outside a field of view of the at least one tracking camera; The processor unit further comprises: determining a position of the first marker within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to guide the user to a position of the second marker relative to the first marker; The system of claim 7 , configured to determine the position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

14. further acquiring initial positions of the first marker and the second marker; 14. The system of claim 7, further comprising: estimating a position of the second marker relative to the first marker based on the obtained initial position when the second marker is not within a field of view of the at least one tracking camera.

15. further obtaining initial positions of the first marker and the second marker relative to known anatomical markers; calculating distances between the known anatomical markers; 15. The system of claim 7, further comprising: if the second marker is not within a field of view of the at least one tracking camera, estimating a position of the second marker relative to the first marker based on the calculated distance.

16. further tracking head movements of the user using one or more sensors in a sensor suite; 16. The system of claim 7, further comprising: calculating the position of the second marker based on the last known position of the second marker when the second marker was within the field of view of the at least one tracking camera.

17. 17. The system of claim 7, wherein the housing of the support module further comprises a base including a circuit board arranged to conduct power from the battery to the processor unit and the head-mounted display device.

18. 18. The system of claim 7, wherein the housing of the support module further comprises a bracket configured to securely and removably restrain the battery and the processor unit when placed on the bracket.

19. further comprising at least two markers affixed to the object of interest for tracking the object of interest, one or both of the at least two markers being outside the field of view of the at least one tracking camera; The processor unit further comprises: displaying a virtual control between the at least two markers; displaying a user input control configured to be aligned with the virtual control based on user input; adjusting the position of the virtual control when the user turns their head to align the user input control with the virtual control; 19. The system of claim 7, configured to track the at least two markers within a field of view of the at least one tracking camera when the at least two markers are both within a field of view of the at least one tracking camera.

20. 1. A self-contained head-mounted surgical navigation system, comprising: a display generator for generating a visual display on a display device; a sensor suite having at least one tracking camera; a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least two markers by: determining a position of a first of the at least two markers within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to direct the user to a position of the second marker relative to a first marker of the at least two markers; determining a position of the second marker with the at least one tracking camera based on guidance from the virtual guide; A self-contained head-mounted surgical navigation system.

21. 1. A self-contained surgical navigation system configured for use with a helmet and a face shield, comprising: A head-mounted display device is provided which is worn by a user during surgery, and the head-mounted display device includes: a display generator for generating a visual display on the display device; a sensor suite having at least one tracking camera; a visible light source; an infrared light source; and a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least one marker; a shroud including a plurality of side walls disposed around the infrared light source, the plurality of side walls defining an opening through which light from the infrared light source is emitted; When the head-mounted display device is attached to the helmet, the shroud, the at least one tracking camera, the visible light source, and the infrared light source are positioned behind a face shield; a plurality of side walls that contact the face shield when the head-mounted display device is attached to the helmet to prevent light emitted by the infrared light source from being reflected back to the at least one tracking camera and pass only through the face shield;

22. 22. The system of claim 21, further comprising an infrared light filter coupled to the visible light source such that the infrared light filter prevents the visible light source from emitting infrared light when coupled to the visible light source.

23. further comprising at least two markers affixed to the object of interest for tracking the object of interest, a first marker being within a field of view of the at least one tracking camera and a second marker being outside a field of view of the at least one tracking camera; The processor unit further comprises: determining a position of the first marker within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to guide the user to a position of the second marker relative to the first marker; 23. The system of claim 21, configured to determine the position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

24. Further comprising a support module, the support module comprising: a user-replaceable modular battery removably inserted into the housing of the support module; a processor unit configured to receive data from the sensor suite and to calculate the position and orientation of at least one marker; the support module is electrically coupled to the head-mounted display device to provide power and data to the head-mounted display device, the support module being worn on the user's body at a location other than the user's head; 24. The system of any one of claims 21 to 23, wherein the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electrical equipment.

25. 25. The system of any one of claims 21-24, wherein a front surface coupled to the plurality of side walls is adjacent to the face shield and has a radius of curvature that generally matches a radius of curvature of the face shield.

26. 26. The system of any one of claims 21 to 25, wherein one or more of the side walls are at an angle of about 10 to about 20 degrees relative to a central axis of the infrared light source.

27. 1. A self-contained surgical navigation system, comprising: A head-mounted display device is provided which is worn by a user during surgery, and the head-mounted display device includes: a display generator for generating a visual display on the display device; a sensor suite having at least one tracking camera; a support module, the support module comprising: a user-replaceable modular battery removably inserted into the housing of the support module; a processor unit configured to receive data from the sensor suite and calculate a position and orientation of at least one marker; the support module is electrically coupled to the head-mounted display device to provide power and data to the head-mounted display device, the support module being worn on the user's body at a location other than the user's head; A self-contained surgical navigation system, wherein the display device and the support module cooperate to provide all sensing and computing capabilities of the system without the need for external sensors, cameras, computers, or other electrical equipment.

28. 28. The system of claim 27, further comprising one or more of a face shield and a helmet, wherein the display device is mounted to the face shield or the helmet.

29. 29. The system of any one of claims 27 to 28, wherein the head-mounted display device further comprises an infrared light source.

30. 30. The system of any one of claims 27 to 29, wherein the head-mounted display device further comprises a visible light source and an infrared light filter coupled to the visible light source, the infrared light filter preventing the visible light source from emitting infrared light when coupled to the visible light source.

31. a shroud disposed around the infrared light source and including a plurality of side walls defining an opening through which light from the infrared light source is emitted; When the head-mounted display device is attached to the helmet, the shroud, the at least one tracking camera, and the infrared light source are disposed behind the face shield; 31. The system of any one of claims 27 to 30, wherein the plurality of side walls are in close proximity to the face shield when the head-mounted display device is attached to the helmet to prevent light emitted by the infrared light source from being reflected back to the at least one tracking camera and to pass only through the face shield.

32. The system further comprises at least one marker affixed to an object of interest for tracking the object of interest, the at least one marker being outside the field of view of the at least one tracking camera, and the processor unit further comprises: tracking the angle of the user's head using one or more sensors in the sensor suite; calculating a relative position of the at least one marker based on a last known position of the at least one marker when the at least one marker was located within a field of view of the at least one tracking camera, the last known position being relative to an angle of the head; 32. The system of any one of claims 27 to 31, configured to display a virtual guide for the user on the display device to guide the user to the location of the at least one marker.

33. further comprising at least two markers affixed to the object of interest for tracking the object of interest, a first marker being within a field of view of the at least one tracking camera and a second marker being outside a field of view of the at least one tracking camera; The processor unit further comprises: determining a position of the first marker within a field of view of the at least one tracking camera; displaying a virtual guide to the user on the display device to guide the user to a position of the second marker relative to the first marker; 33. The system of any one of claims 27 to 32, configured to determine the position of the second marker with the at least one tracking camera based on guidance from the virtual guide.

34. further acquiring initial positions of the first marker and the second marker; 34. The system of claim 27, further comprising: if the second marker is not within a field of view of the at least one tracking camera, estimating a position of the second marker relative to the first marker based on the obtained initial position.

35. further obtaining initial positions of the first marker and the second marker relative to known anatomical markers; calculating distances between the known anatomical markers; 35. The system of claim 27, further comprising: if the second marker is not within the field of view of the at least one tracking camera, estimating a position of the second marker relative to the first marker based on the calculated distance.

36. further tracking head movements of the user using one or more sensors in the sensor suite; 36. The system of claim 27, further comprising: calculating the position of the second marker based on the last known position of the second marker when it was within the field of view of the at least one tracking camera.

37. 37. The system of any one of claims 27 to 36, wherein the housing of the support module further comprises a base including a circuit board arranged to conduct power from the battery to the processor unit and the head-mounted display device.

38. 38. The system of any one of claims 27 to 37, wherein the housing of the support module further comprises a bracket configured to securely and removably restrain the battery and the processor unit when placed on the bracket.

39. further comprising at least two markers affixed to the object of interest for tracking the object of interest, one or both of the at least two markers being outside the field of view of the at least one tracking camera; The processor unit further comprises: displaying a virtual control between the at least two markers; displaying a user input control configured to be aligned with the virtual control based on user input; adjusting the position of the virtual control when the user turns their head to align the user input control with the virtual control; 39. The system of claim 27, configured to track the at least two markers within the field of view of the at least one tracking camera when the at least two markers are both within the field of view of the at least one tracking camera.

40. 1. A self-contained head-mounted surgical navigation system, comprising: a display generator for generating a visual display on a display device; a sensor suite having at least one tracking camera; a processor unit configured to receive data from the sensor suite and calculate the position and orientation of at least two markers by the following steps: displaying a virtual guide to the user on the display device to guide the user to a position where the at least two markers are within the field of view of the at least one tracking camera, the virtual guide being projected into the user's field of view at a specific position relative to at least one marker, the virtual guide indicating a direction in which one or both markers are located; determining positions of the at least two markers using the at least one tracking camera; A self-contained head-mounted surgical navigation system.

41. 1. A self-contained head-mounted surgical navigation system, comprising: a display generator for generating a visual display on a display device; a sensor suite having at least one tracking camera; at least two markers affixed to one or more objects of interest for tracking the one or more objects of interest; a processor unit, the processor unit comprising: Receive data from the sensor suite and calculate the positions of the at least two markers by: projecting a virtual control into the user's field of view at a specific location relative to the at least one marker; displaying a user input control configured to be aligned with the virtual control based on user input; positioning the virtual control such that the at least two markers are located in the field of view of the at least one tracking camera when the user rotates their head to align the user input control with the virtual control; activating the virtual control by aligning the user input control with the virtual control; tracking the at least two markers within a field of view of the at least one tracking camera; A self-contained head-mounted surgical navigation system configured to:

42. 1. A head-mounted surgical navigation system for determining joint centers, comprising: a display generator for generating a visual display on a display device; a sensor suite having at least one tracking camera; at least one fiducial marker affixed to a bone for tracking the bone, the at least one fiducial marker being positioned relative to the bone as it pivots at or relative to the joint; at least one fixed reference marker arranged to be substantially fixed relative to the joint; a processor unit, the processor unit comprising: registering the points on the bone to a reference coordinate frame; creating a bone coordinate frame based on the registered points; transforming from the reference coordinate frame to a bone coordinate frame; acquiring a point of the at least one fixed marker in the reference frame using the at least one tracking camera, wherein during acquisition, a position of at least a portion of the visual representation moves synchronously with movement of the head-mounted surgical navigation system; a head-mounted surgical navigation system configured to determine a joint center in the bone coordinate frame.

43. 43. The system of claim 42, wherein determining comprises calculating a position of the joint center in the bone coordinate system.

44. 43. The system of claim 42, wherein determining comprises substantially continuously processing through an optimal estimation filter to determine the joint centers.

45. 43. The system of claim 42, wherein determining comprises performing batch processing to determine the joint centers after all points have been acquired.

46. 46. ​​The system of any one of claims 42 to 45, wherein the bone is one of the femur, tibia, humerus, radius, or vertebral body.

47. 47. The system of any one of claims 42 to 46, wherein the joint is one of a hip, knee, shoulder, elbow, ankle, and vertebral body.

48. 48. The system of any one of claims 42 to 47, wherein stationary further comprises fixed in inertial space.

49. 1. A method for registering condylar surfaces prior to setting a resection angle, the method being performed by a head-mounted surgical navigation system, the method being performed by a processor unit; displaying a target including one or more regions on a display of the head-mounted surgical navigation system; providing a movable icon on said display representing in real time one or more angles received from the condylar guide; receiving one or more user inputs to adjust the position of the movable icon relative to the one or more regions within the target; and outputting a visual marker on the display in any of the one or more areas of the target with which the movable icon interacts during adjustment of the position of the movable icon, the visually marked area indicating a valid depth reference point that has been captured.

50. 50. The method of claim 49, further comprising the step of: restricting movement of said movable icon to prevent recording of a previously captured valid depth reference point.

51. 51. A method according to any one of claims 49 to 50, further comprising the step of forming a database in which the captured valid depth reference points are stored.

52. 52. The method of any one of claims 49 to 51, wherein the target is a grid or a bullseye.

53. 53. The method of any one of claims 49 to 52, wherein each of the one or more regions is sequentially highlighted such that the method includes a step of outputting instructions to the user on the display to move the condyle guide relative to the condyle until the movable icon at least partially overlaps the highlighted region.

54. 54. The method of any one of claims 49 to 53, wherein any one of the one or more regions is highlighted such that the method includes a step of outputting instructions to the user on the display to move the condyle guide relative to the condyle until the movable icon at least partially overlaps the highlighted region.

55. 55. The method of claim 53 or 54, further comprising deactivating the highlighted region and highlighting a second region of the one or more regions when the highlighted region at least partially overlaps the movable icon.

56. 56. The method of any one of claims 49 to 55, comprising prompting a user to remove the condylar guide and attach a cutting guide.

57. 57. The method of claim 56, further comprising calculating the resection depth based on a distance from a current resection plane defined by the cutting guide to one of the effective depth reference points corresponding to the depth reference plane.

58. The method further includes providing a condylar guide, the condylar guide comprising: a body having a first end and a second end; at least one planar surface extending from at least a portion of a lateral region of the first end, the planar surface configured to rest on one or more femoral condyles and establish a zero-depth plane for calculating resection depth; at least one tracker disposed on the at least one planar surface for tracking a pose of the condylar guide; a connector extending from the second end of the body and configured to couple to a cutting guide; 58. The method of any one of claims 49 to 57, comprising:

59. 59. The method of claim 58, wherein the condylar guide further comprises an elongated handle extending from the first end of the body.

60. 60. The method of any one of claims 58 to 59, wherein the body further defines an opening configured to receive a pin therein for insertion into bone.

61. 61. The method of claim 60, wherein the diameter of the opening is sized to allow the condylar guide to tilt when a pin is inserted through the opening.

62. 62. The method of any one of claims 58 to 61, further comprising a release mechanism extending from the second end of the body in a direction opposite to the connector, the release mechanism configured to couple the condylar guide to the bone prior to pinning the cutting guide to the bone.

63. 63. The method of any one of claims 58 to 62, wherein at least a portion of the second end of the body defines a slot configured to receive a slider within which the connector and the release mechanism are inserted on opposite sides of the slider.

64. 64. The method of any one of claims 58 to 63, wherein the at least one planar surface is configured to simulate a plane tangent to a femoral condyle.

65. 65. The method of any one of claims 49 to 64, further comprising tracking the condylar guide using the at least one tracker positioned on the at least one planar surface to determine one or more valid depth reference points.

66. 66. The method of any one of claims 49 to 65, further comprising pinning the cutting guide only after determining the one or more valid depth reference points using the condylar guide coupled to the cutting guide.

67. 59. The method of claim 58, wherein the connector is detachable.

68. 1. A method for registering condylar surfaces prior to setting a resection angle, the method being performed by a head-mounted surgical navigation system, the method being executed by a processor, and comprising: displaying a target including one or more regions on a display of the head-mounted surgical navigation system; receiving in real time the one or more angles received from the condylar guide and displaying them on a display; receiving one or more user inputs to adjust the condylar guide relative to the one or more regions within the target; outputting a visual marker on the display at any of the one or more regions of the target, the visually marked region indicating a captured and valid depth reference point.

69. 69. The method of claim 68, further comprising the step of limiting recording of previously captured valid depth reference points.

70. 70. A method according to any one of claims 68 to 69, further comprising forming a database in which the captured valid depth reference points are stored.

71. 71. The method of any one of claims 68 to 70, wherein the target is a grid or a bullseye.

72. 72. The method of any one of claims 68 to 71, wherein each of the one or more regions is sequentially highlighted such that the method includes outputting instructions to a user on the display to move the condyle guide relative to the condyle until the angle of the condyle guide at least partially overlaps the highlighted region.

73. 73. The method of any one of claims 68 to 72, wherein any one of the one or more regions is highlighted such that the method includes outputting instructions to a user on the display to move the condyle guide relative to the condyle until the angle of the condyle guide at least partially overlaps with the highlighted region.

74. 74. The method of any one of claims 68 to 73, further comprising deactivating the highlighted region and highlighting a second region of the one or more regions when the highlighted region and the angle of the condylar guide at least partially overlap.

75. 75. The method of any one of claims 68 to 74, comprising prompting a user to remove the condylar guide and attach a cutting guide.

76. 76. The method of any one of claims 68 to 75, further comprising a step of calculating a resection depth based on a distance from a current resection plane defined by the cutting guide to one of the valid depth reference points corresponding to a depth reference plane.

77. The method further includes providing a condylar guide, the condylar guide comprising: a body having a first end and a second end; at least one planar surface extending from at least a portion of a lateral region of the first end, the planar surface configured to rest against one or more femoral condyles to establish a zero-depth plane for calculating resection depth; at least one tracker disposed on the at least one planar surface for tracking the orientation of the condylar guide; a connector extending from the second end of the body and configured to couple to the cutting guide; 77. The method of any one of claims 68 to 76, comprising:

78. 78. The method of claim 77, wherein the condylar guide further comprises an elongated handle extending from the first end of the body.

79. 79. The method of any one of claims 77 to 78, wherein the body further defines an opening configured to receive a pin therein for insertion into the bone.

79. 79. The method of claim 78, wherein the diameter of the opening is sized to allow the condylar guide to tilt when a pin is inserted through the opening.

80. 80. The method of any one of claims 77 to 79, further comprising a release mechanism extending from the second end of the body in a direction opposite the connector, the release mechanism configured to couple the condylar guide to the bone before pinning the cutting guide to the bone.

81. 81. The method of any one of claims 77 to 80, wherein at least a portion of the second end of the body defines a slot configured to receive a slider, and wherein the connector and the release mechanism are inserted into the slot on opposite sides of the slider.

82. 82. The method of any one of claims 77 to 81, wherein the at least one planar surface is configured to simulate a plane tangent to a femoral condyle.

83. 83. The method of any one of claims 77 to 82, further comprising tracking the condylar guide using the at least one tracker positioned on the at least one planar surface to determine one or more valid depth reference points.

84. 84. The method of any one of claims 77 to 83, further comprising pinning the cutting guide only after determining one or more valid depth reference points using the condylar guide coupled to the cutting guide.

85. 1. A head-mounted surgical navigation system for determining a hip joint center, comprising: a display generator for generating a visual display on a display device; a sensor suite having at least one tracking camera; at least one fiducial marker affixed to the femur for tracking the femur, the at least one fiducial marker being positioned so that the femur pivots at or relative to the hip joint; at least one fixed reference marker positioned to be substantially fixed relative to the hip joint; a processor unit, the processor unit comprising: registering points on the femur to a reference coordinate frame; creating a femoral coordinate frame based on the registered points; transforming from the reference coordinate frame to the femoral coordinate frame; acquiring a point of the at least one fixed marker in the reference frame using the at least one tracking camera, wherein during acquisition, a position of at least a portion of the visual representation moves synchronously with movement of the head-mounted surgical navigation system; a head-mounted surgical navigation system configured to determine a hip joint center in the femoral coordinate frame.

86. 86. The system of claim 85, wherein determining includes calculating a location of the hip joint center in the femoral coordinate system.

87. 86. The system of claim 85, wherein determining comprises substantially continuously processing through an optimal estimation filter to determine the hip joint center.

88. 86. The system of claim 85, wherein determining comprises performing batch processing to determine the hip joint center after all points are acquired.

89. 89. The system of any one of claims 85 to 88, wherein stationary further comprises fixed in inertial space.

Images