Augmented reality guidance for spinal and joint surgery

Abstract

An apparatus and method for performing a surgical procedure or surgical procedure by visual guidance using an optical head-mounted display is disclosed.

Description

[0001] Divisional Description

[0002] This application is a divisional application of Chinese Invention Patent Application No. 202010160584.4, filed on March 10, 2017, entitled “Augmented Reality Guidance for Spinal and Joint Procedures”; which is a divisional application of Chinese Invention Patent Application No. 201780029093.2, filed on March 10, 2017, entitled “Devices and Methods for Surgery”.

[0003] Related Applications

[0004] This patent application claims the benefit of and priority to U.S. Provisional Application No. 62 / 307,476, filed March 12, 2016; U.S. Provisional Application No. 62 / 318,157, filed April 4, 2016; U.S. Provisional Application No. 62 / 323,716, filed April 17, 2016; U.S. Provisional Application No. 62 / 331,995, filed May 5, 2016; U.S. Provisional Application No. 62 / 354,780, filed June 26, 2016; U.S. Provisional Application No. 62 / 378,242, filed August 23, 2016; U.S. Provisional Application No. 62 / 393,054, filed September 11, 2016; U.S. Provisional Application No. 62 / 406,379, filed October 10, 2016; U.S. Provisional Application No. 62 / 425,019, filed November 21, 2016; U.S. Provisional Application No. 62 / 445,691, filed January 12, 2017; U.S. Provisional Application No. 62 / 453,484, filed February 1, 2017, the entire contents of which are incorporated by reference herein in their entireties. TECHNICAL FIELD

[0005] Aspects of the present disclosure relate to devices and methods for using optical head-mounted displays for visual guidance to perform surgical procedure steps or surgical treatments. BACKGROUND

[0006] With computer-assisted surgery, such as surgical navigation or robotics, preoperative imaging of a patient can be used. The imaging can be displayed on an external computer monitor in the operating room (OR), and patient anatomy, such as a point of interest, can be registered with the information displayed on the monitor. Eye-hand coordination is challenging for a surgeon because the surgical site is at a different location, and the surgeon’s line of sight is different from the view coordinate system of the external computer monitor. SUMMARY

[0007] Further, aspects of the present disclosure are directed to enabling simultaneous visualization of patient real-time data (e.g., a patient's spine or joint) and a digital representation of virtual data (e.g., a virtual cut of an optical head mounted display (OHMD) and / or a virtual surgical guide of a cutting block or drill guide). In some embodiments, the surgical site including the patient real-time data, the OHMD, and the virtual data are all registered in a common coordinate system. In some embodiments, the virtual data is superimposed on and aligned with the patient real-time data. Unlike a virtual reality head mounted system that blends real-time data, the OHMD allows a surgeon to view the patient real-time data (e.g., a surgical field) while simultaneously viewing virtual data of the patient and / or virtual surgical instruments or implants having predetermined positions and / or orientations using a display of the OHMD unit.

[0008] Aspects of the present disclosure describe novel devices for performing surgical procedures or surgical treatments using an OHMD, for example, by displaying virtual representations of one or more virtual surgical tools, virtual surgical instruments including virtual surgical guides or cutting blocks, virtual trial implants, virtual implant components, virtual implants or virtual devices, predetermined starting points, predetermined starting positions, predetermined starting orientations or alignments, predetermined intermediate points, predetermined intermediate positions, predetermined intermediate orientations or alignments, predetermined end points, predetermined end positions, predetermined end orientations or alignments, predetermined paths, predetermined planes, predetermined cutting planes, predetermined contours or profiles or cross-sections or surface features or shapes or projections, predetermined depth markers or depth gauges, predetermined angles or orientations or rotational markers, predetermined axes (e.g., rotational axes, bending axes, extension axes, predetermined axes of virtual surgical tools, virtual surgical instruments including virtual surgical guides or cutting blocks, virtual trial implants, virtual implant components, implants or devices, etc.), non-visualized portions of one or more devices or implants or implant components or surgical instruments or surgical tools, and / or one or more predetermined tissue changes or alterations, etc.

[0009] Aspects of the present disclosure are directed to a device including at least one OHMD configured to generate a virtual surgical guide. In some embodiments, the virtual surgical guide is a three-dimensional representation in digital format corresponding to at least a portion of a physical surgical guide, a placement indicator of the physical surgical guide, or a combination thereof. In some embodiments, the at least one OHMD is configured to display the virtual surgical guide superimposed on a physical joint based at least in part on predetermined position coordinates of the virtual surgical guide, and the virtual surgical guide is configured to align the physical surgical guide or physical saw blade with the virtual surgical guide to guide a bone cut of the joint.

[0010] In some embodiments, the device includes one, two, three, or more optical head-mounted displays.

[0011] In some embodiments, the virtual surgical guide is configured to guide bone cuts in a knee replacement, a hip replacement, a shoulder replacement, or an ankle replacement.

[0012] In some embodiments, the virtual surgical guide includes a virtual slot for a virtual or physical saw blade.

[0013] In some embodiments, the virtual surgical guide includes a planar area for aligning a virtual or physical saw blade.

[0014] In some embodiments, the virtual surgical guide includes two or more virtual guide holes or paths for aligning two or more physical drill bits or pins.

[0015] In some embodiments, the predetermined locations of the virtual surgical guide include anatomical information and / or alignment information of a joint. For example, the anatomical and / or alignment information of the joint can be based on at least one of a joint coordinate, an anatomical axis of the joint, a biomechanical axis of the joint, a mechanical axis, or a combination thereof.

[0016] In some embodiments, the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined limb alignment. For example, the predetermined limb alignment can be a normal mechanical axis alignment of a leg.

[0017] In some embodiments, the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined femoral or tibial component rotation. In some embodiments, the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined flexion of a femoral component or a predetermined slope of a tibial component.

[0018] In some embodiments, the virtual surgical guide is configured to guide a proximal femoral cut based on a predetermined leg length.

[0019] In some embodiments, the virtual surgical guide is configured to guide a distal tibial or talus bone cut in an ankle replacement, and the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined ankle alignment, wherein the predetermined ankle alignment includes a coronal plane implant component alignment, a sagittal plane implant component alignment, an axial plane component alignment, an implant component rotation, or a combination thereof.

[0020] In some embodiments, the virtual surgical guide is configured to guide bone cuts of a proximal humerus in a shoulder joint replacement, and the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined humeral implant component alignment, wherein the humeral implant component alignment comprises a coronal implant component alignment, a sagittal implant component alignment, an axial plane component alignment, an implant component, or a combination thereof.

[0021] In some embodiments, the predetermined position of the surgical guide is based on a preoperative or intraoperative imaging, one or more intraoperative measurements, intraoperative data, or a combination thereof.

[0022] Aspects of the present disclosure relate to an apparatus comprising two or more optical head-mounted displays for two or more users, wherein the apparatus is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a three-dimensional representation in digital format, the three-dimensional representation corresponding to at least one of a portion of a physical surgical guide, a placement indicator of a physical surgical guide, or a combination thereof, wherein the optical head-mounted displays are configured to display the virtual surgical guide superimposed onto a physical joint based at least in part on coordinates of a predetermined position of the virtual surgical guide, and wherein the virtual surgical guide is configured to align a physical surgical guide or a saw blade to guide bone cuts of the joint.

[0023] Aspects of the present disclosure relate to an apparatus comprising at least one optical head-mounted display and a virtual bone cut plane, wherein the virtual bone cut plane is configured to guide bone cuts of a joint, wherein the virtual bone cut plane corresponds to at least a portion of a bone cut plane, and wherein the optical head-mounted display is configured to display the virtual bone cut plane superimposed on a physical joint based at least in part on coordinates of a predetermined position of the virtual bone cut plane. In some embodiments, the virtual bone cut plane is configured to guide bone cuts in a predetermined varus or valgus orientation or a predetermined tibial slope or a predetermined femoral flexion of an implant component or a predetermined leg length.

[0024] Aspects of the present disclosure relate to methods of preparing a joint of a prosthesis for use in a patient. In some embodiments, the method includes registering one or more optical head-mounted displays worn by a surgeon or surgical assistant in a coordinate system, obtaining one or more intraoperative measurements from a physical joint of a patient to determine one or more intraoperative coordinates, registering the one or more intraoperative coordinates from the physical joint of the patient in the coordinate system, generating a virtual surgical guide, determining a predetermined position and / or orientation of the virtual surgical guide based on the one or more intraoperative measurements, displaying and superimposing the virtual surgical guide onto the physical joint using the one or more optical head-mounted displays based at least in part on coordinates of the predetermined position of the virtual surgical guide, and aligning a physical surgical guide or a physical saw blade to the virtual surgical guide for guiding bone cuts of the joint.

[0025] In some embodiments, the one or more optical head-mounted displays are registered in a common coordinate system. In some embodiments, the common coordinate system is a shared coordinate system.

[0026] In some embodiments, the virtual surgical guide is used to guide bone cuts in a knee arthroplasty, a hip arthroplasty, a shoulder arthroplasty, or an ankle arthroplasty.

[0027] In some embodiments, the predetermined position of the virtual surgical guide determines a tibial slope for implanting one or more tibial implant components in a knee arthroplasty. In some embodiments, the predetermined position of the virtual surgical guide determines a varus or valgus correction angle of femoral and / or tibial components in a knee arthroplasty.

[0028] In some embodiments, the virtual surgical guide corresponds to a physical distal femoral guide or cutting block, and the predetermined position of the virtual surgical guide determines a femoral component flexion.

[0029] In some embodiments, the virtual surgical guide corresponds to a physical anterior or posterior femoral surgical guide or cutting block, and the predetermined position of the virtual surgical guide determines a femoral component rotation.

[0030] In some embodiments, the virtual surgical guide corresponds to a physical chamfer femoral guide or cutting block.

[0031] In some embodiments, the virtual surgical guide corresponds to a physical multi-cut femoral guide or cutting block, and the predetermined position of the virtual surgical guide determines one or more of an anterior cut, a posterior cut, a chamfer cut, and a femoral component rotation.

[0032] In some embodiments, the virtual surgical guide is used in a hip arthroplasty, and the predetermined position of the virtual surgical guide determines a post-implant leg length.

[0033] In some embodiments, the virtual surgical guide is a virtual plane for aligning a physical saw blade to guide a bone cut of a joint.

[0034] In some embodiments, the one or more intraoperative measurements include detecting one or more optical markers attached to a patient joint, a surgical table, a fixed structure in the operating room, or a combination thereof. In some embodiments, one or more cameras or image capture or video capture systems in the optical head-mounted display detect at least one or more of the position and orientation, alignment, direction of movement, or speed of movement of the one or more optical markers including their coordinates (x, y, z).

[0035] In some embodiments, the registration of the one or more optical head-mounted displays, the surgical site, the joint, the spine, the surgical instruments, or the implant assembly can be performed using a spatial mapping technique.

[0036] In some embodiments, the registration of the one or more optical head-mounted displays, the surgical site, the joint, the spine, the surgical instruments, or the implant assembly can be performed using a depth sensor.

[0037] In some embodiments, the virtual surgical guide is used to guide a bone cut of a distal tibia or talus in an ankle joint replacement, and the one or more optical head-mounted displays are used to align the virtual surgical guide based on a predetermined tibial or talus implant assembly alignment, wherein the predetermined tibial or talus implant assembly alignment includes a coronal plane implant assembly alignment, a sagittal plane implant assembly alignment, an axial plane assembly alignment, an implant assembly rotation, or a combination thereof.

[0038] In some embodiments, the virtual surgical guide is used to guide a bone cut of a proximal humerus in a shoulder joint replacement, and wherein the one or more optical head-mounted displays are used to align the virtual surgical guide based on a predetermined humeral implant assembly alignment, wherein the humeral implant assembly alignment includes a coronal plane implant assembly alignment, a sagittal plane implant assembly alignment, an axial plane assembly alignment, a humeral implant assembly rotation, or a combination thereof.

[0039] Aspects of the present disclosure relate to a system comprising at least one optical head-mounted display and an implant virtual library, wherein the implant virtual library comprises at least one virtual implant assembly, wherein the virtual implant assembly has at least one dimension corresponding to or having substantially the same dimension as a dimension of an implant assembly, wherein the at least one optical head-mounted display is configured to display the virtual implant assembly in substantial alignment with a tissue where the implant assembly is intended to be placed, wherein the placement of the virtual implant assembly is intended to achieve a predetermined implant assembly position and / or orientation. BRIEF DESCRIPTION OF DRAWINGS

[0040] The embodiments of the illustrative, non-limiting examples will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0041] Figure 1 Multiple optical head mounted displays are illustrated for multiple observers, such as a primary surgeon, a second surgeon, a surgical assistant and / or a nurse in accordance with certain embodiments of the application.

[0042] Figure 2 Workflows for segmentation and selection of subsequent steps are illustrated in accordance with certain embodiments of the application.

[0043] Figure 3 Registration of digital holograms for initial surgical steps, performance of surgical steps and re-registration of one or more digital holograms for subsequent surgical steps are illustrated in accordance with certain embodiments of the application.

[0044] Figure 4A B and C are examples of any virtual plane in the hip and femoral neck cut planes in accordance with certain embodiments of the application.

[0045] Figure 5 is an example of any virtual plane in the knee extending through the medial and lateral joint spaces in the knee in accordance with certain embodiments of the application.

[0046] Figure 6 is an illustrative flowchart of different methods of addressing errors between changes caused by surgical steps and expected, projected or scheduled changes in patient virtual data in accordance with certain embodiments of the application.

[0047] Figure 7A -H is an example of a femoral neck cut and a corrective femoral neck cut technique in accordance with certain embodiments of the application.

[0048] Figure 7A -H is an example of a femoral neck cut and a corrective femoral neck cut technique in accordance with certain embodiments of the application.

[0049] Figure 8A -H is an example of a distal femur cut and a corrective distal femur cut technique in accordance with certain embodiments of the application.

[0050] Figure 9A -G is an example of a distal femur cut and a corrective distal femur cut technique in accordance with certain embodiments of the application.

[0051] Figure 10A -G is an example of a distal femur cut and a proximal tibia cut and a corrective cut technique in accordance with certain embodiments of the application.

[0052] Figure 11 is an example of how to use intra-operative data, such as intra-operative measurements, to generate a virtual surgical plan, such as in accordance with certain embodiments of the present application.

[0053] Figure 12 is an example of an exemplary workflow for generating a virtual surgical plan, such as in accordance with certain embodiments of the present application.

[0054] Figure 13 is an example of how to modify a virtual surgical plan using intra-operative data, such as intra-operative measurements, in accordance with certain embodiments of the present application.

[0055] Figure 14 is an example of how to use multiple optical head-mounted displays, such as by a first surgeon, a second surgeon, a surgical assistant, and / or one or more nurses, during a surgical procedure, and how to modify and display a surgical plan during a procedure using multiple optical head-mounted displays, while preserving correct perspective of virtual data and corresponding real-time data for each operator, in accordance with certain embodiments of the present application.

[0056] Figure 15 is an example of how to use or apply two-dimensional to three-dimensional morphing data.

[0057] Figure 16A and B are flowcharts outlining model generation, registration, and view projection of one or more optical head-mounted displays, such as for a primary surgeon, a second surgeon, a surgical assistant, a nurse, or others, in accordance with certain embodiments of the present application.

[0058] Figure 17A -D is an illustrative flowchart for selecting a protocol and approach for performing a spinal procedure in a mixed reality environment, in accordance with certain embodiments of the present application.

[0059] Figure 18A -F is an example of how to use one or more optical head-mounted displays to display a virtual acetabular reamer axis and align a physical acetabular reamer with the virtual reamer axis for placement of an acetabular cup having a predetermined cup angle, offset, medial or lateral position, and / or anteversion angle, in accordance with certain embodiments of the present application.

[0060] Figure 19AThe non-limiting illustrative example of C illustrates how a virtual surgical guide (e.g., an anterior-posterior (AP) femoral cutting block displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical AP cutblock for knee replacement) can be used in accordance with certain embodiments of the application.

[0061] Figure 20A The non-limiting illustrative example of C illustrates how a virtual surgical guide (e.g., an anterior-posterior (AP) femoral cutting block displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical AP cutblock for knee replacement) can be used in accordance with certain embodiments of the application.

[0062] Figure 21A The non-limiting illustrative example of F illustrates how a virtual surgical guide (e.g., a virtual proximal tibial cutting guide displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical proximal tibial cutting guide in accordance with certain embodiments of the application) can be used.

[0063] Figure 22A The anterior and lateral views of A and B illustrate an exemplary normal anterior cruciate ligament (ACL) that includes an anterior-medial and a posterior-lateral fiber.

[0064] Figure 22C The anterior and lateral views of C and D illustrate exemplary anterior cruciate ligament tunnels (straight solid lines) on the femoral side and the tibial side.

[0065] Figure 22E The anterior and lateral views of E and F illustrate exemplary virtual anterior cruciate ligament tunnels on the femoral side and the tibial side (straight dashed lines) in accordance with certain embodiments of the application.

[0066] Figure 22G The anterior and lateral views of G and H illustrate exemplary virtual anterior cruciate ligament grafts that extend through the intercondylar space (straight solid lines) between the femur and the tibia on the femoral side and the tibial side in accordance with certain embodiments of the application.

[0067] Figure 23 The illustrative non-limiting flowchart of FIG. 1 illustrates different methods for planning the positioning, location, orientation, alignment, and / or direction of one or more femoral or tibial tunnels (e.g., for single or double bundle techniques) or for placing an anterior cruciate ligament graft in accordance with certain embodiments of the application. DETAILED DESCRIPTION

[0068] Various example embodiments will be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments are shown. The

[0069] The term "patient real-time data" as used herein includes the surgical site, anatomy, anatomical or tissue and / or pathology, pathological structure or tissue of the patient as seen by the eyes of the surgeon or observer without information from virtual data, virtual data or stereoscopic views of imaging studies. The term "patient real-time data" does not include internal or subsurface tissue or structures or hidden tissue or structures that can only be seen with the aid of a computer monitor or optical head-mounted display.

[0070] The terms "real surgical instrument", "actual surgical instrument", "physical surgical instrument" and "surgical instrument" are used interchangeably throughout the application; the terms "real surgical instrument", "actual surgical instrument", "physical surgical instrument" and "surgical instrument" do not include virtual surgical instruments. For example, a physical surgical instrument can be a surgical instrument provided by a manufacturer or supplier for a spinal surgery, pedicle screw fixation, anterior spinal fusion, knee replacement, hip replacement, ankle replacement and / or shoulder replacement; for example, a physical surgical instrument can be a chopper, a guide rod, a awl, a reamer, a hammer, a broach. A physical surgical instrument can be reusable or disposable or a combination thereof. A physical surgical instrument can be patient specific. The term "virtual surgical instrument" does not include real surgical instruments, actual surgical instruments, physical surgical instruments and surgical instruments.

[0071] The terms "real surgical tool", "actual surgical tool", "physical surgical tool" and "surgical tool" are used interchangeably throughout the application; the terms "real surgical tool", "actual surgical tool", "physical surgical tool" and "surgical tool" do not include virtual surgical tools. A physical surgical tool can be a surgical tool provided by a manufacturer or supplier. For example, a physical surgical tool can be a staple, a drill, a saw blade, a retractor, a frame for separating tissue and other tools for orthopedic, neurosurgery, urology or cardiovascular surgery. The term virtual surgical tool does not include real surgical tools, actual surgical tools, physical surgical tools and surgical tools.

[0072] ​The terms real implant or implant component, actual implant or implant component, physical implant or implant component, and implant or implant component are used interchangeably throughout the application; the terms real implant or implant component, actual implant or implant component, physical implant or implant component, and implant or implant component do not include virtual implant or implant component. The physical implant or implant component can be an implant or implant component provided by a manufacturer or supplier. For example, the physical surgical implant can be a pedicle screw, a spinal fixation rod, a spinal fusion cage, a femoral or tibial component in a knee replacement, an acetabular cup or femoral stem and head in a hip replacement. The terms "virtual implant" or "implant component" do not include real implant or implant component, actual implant or implant component, physical implant or implant component, and implant or implant component.

[0073] With surgical navigation, a first virtual instrument can be displayed on a computer monitor that is an image of a physical instrument tracked with navigation markers, such as infrared or radio frequency markers, and the position and / or orientation of the first virtual instrument can be compared to the position and / or orientation of a corresponding second virtual instrument generated in a virtual surgical plan. Thus, with surgical navigation, the position and / or orientation of the first and second virtual instruments are compared.

[0074] Aspects of the present application relate to apparatus, systems, and methods for positioning a virtual path, virtual plane, virtual tool, virtual surgical instrument, or virtual implant component in a mixed reality environment using a head-mounted display device, which is optionally coupled to one or more processing units.

[0075] With guidance in the mixed reality environment, a virtual surgical guide, tool, instrument, or implant can be superimposed on a physical joint, spine, or surgical site. In addition, the physical guide, tool, instrument, or implant can be aligned with a virtual surgical guide, tool, instrument, or implant displayed or projected by the optical head-mounted display. Thus, guidance in the mixed reality environment does not require the use of multiple virtual images of the guide, tool, instrument, or implant, and does not require the comparison of the position and / or orientation of multiple virtual images of the virtual guide, tool, instrument, or implant.

[0076] In various embodiments, the optical head-mounted display can display one or more virtual surgical tools, including virtual surgical instruments of a virtual surgical guide or virtual cutting block, a virtual trial implant, a virtual implant assembly, a virtual implant or virtual device, a predetermined starting point, a predetermined starting location, a predetermined starting direction or alignment, a predetermined intermediate point, a predetermined intermediate location, a predetermined intermediate direction or alignment, a predetermined end point, a predetermined end location, a predetermined end direction or alignment, a predetermined path, a predetermined plane, a predetermined cutting plane, a predetermined shape or contour or cross-section or surface feature or shape or projection, a predetermined depth marker or depth gauge, a predetermined angle or direction or rotational marker, a predetermined axis, such as a rotational axis, a bending axis, an extension axis, a virtual surgical tool, including a virtual surgical instrument of a virtual surgical guide or cutting block, a virtual trial implant, a virtual implant assembly, an implant or device, one or more more devices or implants or implant assemblies or surgical instruments or surgical tools, and / or an estimate or prediction of one or more predetermined tissue changes or alterations, or non-visualized portions thereof.

[0077] A preoperative imaging study, preoperative data, preoperative measurements, intraoperative imaging study, intraoperative data and / or intraoperative measurements can be used to determine a position, location, orientation, alignment, direction, movement speed, applied force, or virtual or physical of a surgical instrument or tool.

[0078] A preoperative imaging study, preoperative data, preoperative measurements, intraoperative imaging study, intraoperative data and / or intraoperative measurements can be used to determine any position, location, direction, alignment, sagittal plane alignment, coronal plane alignment, axial plane alignment, rotation, implantation inclination, implantation angle, flexion of an implant assembly; position, location, direction, alignment relative to one or more anatomical landmarks; position, location, direction, alignment relative to one or more anatomical planes; position, location, direction, alignment relative to one or more anatomical axes; position, location, direction, alignment relative to one or more biomechanical axes; position, location, direction, alignment relative to a mechanical axis of a trial implant, implant assembly or implant, virtual and / or physical. Intraoperative measurements can include measurements for registration purposes, such as a joint, a spine, a surgical site, a bone, a cartilage, an optical head-mounted display, a surgical tool or instrument, a trial implant, an implant assembly or an implant.

[0079] In some embodiments, multiple coordinate systems can be used instead of a common or shared coordinate system. In that case, coordinates can be transformed from one coordinate system to another, such as for registration purposes of an optical head-mounted display, patient real-time data, including a surgical site, virtual instruments and / or virtual implants and physical instruments and physical implants, etc.

[0080] Optical head-mounted display

[0081] In some embodiments of the application, a pair of glasses is used. The glasses can include an optical head-mounted display. An optical head-mounted display (OHMD) can be a wearable display that has the ability to reflect projected images as well as allow the user to see through. Various types of optical head-mounted displays can be used to implement the application. These include curved mirror or curved combiner optical head-mounted displays as well as waveguide optical head-mounted displays or lightpipe optical head-mounted displays. The optical head-mounted display can optionally use diffractive optics, holographic optics, polarized optics, and reflective optics.

[0082] Conventional input devices that can be used with the optical head-mounted display include, but are not limited to, touchpads or buttons, smartphone controllers, voice recognition, and gesture recognition. Advanced interfaces can be available, such as, for example, brain-computer interfaces.

[0083] Optionally, the computer or server or workstation can send data to the optical head-mounted display. The data transmission can be through a cable, Bluetooth, WiFi, optical signal, and any other method or data transmission mode known in the art. The optical head-mounted display can display virtual data, such as, for example, patient virtual data, in uncompressed form or in compressed form. The patient virtual data can optionally be reduced in resolution when transmitted to and displayed by the optical head-mounted display.

[0084] When virtual data are transmitted to the optical head-mounted display, they can exist in compressed form during transmission. The optical head-mounted display can then optionally decompress them so that the optical head-mounted display displays the virtual data in uncompressed form.

[0085] Alternatively, when virtual data are transmitted to the optical head-mounted display, they can be reduced in resolution during transmission, such as, for example, by increasing the layer thickness of the image data prior to transmission. The optical head-mounted display can then optionally increase the resolution, such as, for example, by re-interpolating to the original layer thickness or even a thinner layer thickness of the image data, so that the resolution of the virtual data is equal to or greater than the original virtual data or at least higher than the resolution of the transmitted data displayed by the optical head-mounted display.

[0086] In some embodiments, the optical head-mounted display can send data back to the computer, server, or workstation. Such data can include, but is not limited to:

[0087] - position, location, or orientation information about the optical head-mounted display or the wearer of the optical head-mounted display, the surgeon or the operator

[0088] - changes in the position, location, or orientation of the optical head-mounted display

[0089] - data generated by one or more inertial measurement units (IMUs)

[0090] - Data from markers connected to, integrated to or coupled to the optical head-mounted display

[0091] - Data generated by a surgical navigation system connected to, integrated to or coupled to the optical head-mounted display

[0092] - Data generated by an image and / or video capture system connected to, integrated to or coupled to the optical head-mounted display

[0093] - Parallax data, such as using two or more image and / or video capture systems connected to, integrated to or coupled to the optical head-mounted display, such as a first image and / or video capture system located above or below or near the left eye and a second located above or below or near the right eye

[0094] - Distance data, such as parallax data generated by two or more image and / or video capture systems, assessing changes in distance between the optical head-mounted display and the surgical field or objects

[0095] - Motion parallax data

[0096] - Data related to calibration or registration phantoms (see other sections of this specification)

[0097] - Any type of real-time data of the patient captured by the optical head-mounted display, including image and / or video capture systems connected, integrated or coupled to the optical head-mounted display

[0098] o Such as changing the live surgical site

[0099] o Such as using certain surgical instruments detected by the image and / or video capture system

[0100] o Such as using certain medical devices or trial implants detected by the image and / or video capture system

[0101] - Any type of modification to the surgical plan

[0102] o Partial or multi-faceted live surgical plan

[0103] o Partial or multi-faceted virtual surgical plan

[0104] The radio frequency tags used throughout the embodiments can be active or passive types, with or without a battery.

[0105] Exemplary optical head-mounted displays include ODG R-7, R-8, and R-8 Smartglasses from ODG (Osterhout Group, San Francisco, CA), NVIDIA SHIELD™ 9423-D Vision Wireless Goggles (NVIDIA, Santa Clara, CA), and Microsoft HoloLens (Microsoft, Redmond, WI).

[0106] Microsoft HoloLens is manufactured by Microsoft. It is a pair of augmented reality smartglasses. Microsoft HoloLens can use the Windows 10 operating system. The front of the Microsoft HoloLens includes sensors, related hardware, several cameras, and a processor. The visor includes a pair of transparent combiner lenses for displaying projected images. The Microsoft HoloLens can be adjusted for interpupillary distance (IPD) using an integrated program that recognizes gestures. A pair of speakers is also integrated. The speakers do not exclude outside sounds and allow the user to hear virtual sounds. A USB 2.0 micro-B receptacle and an A 3.5 mm headphone receptacle are integrated.

[0107] Microsoft HoloLens has an inertial measurement unit (IMU) with an accelerometer, a gyroscope, and a magnetometer, four environmental mapping sensors / cameras (two on each side), a depth camera with a 120° x 120° field of view, a 2.4 megapixel photographic camera, a four-microphone array, and an ambient light sensor.

[0108] Microsoft HoloLens has an Intel Cherry Trail SoC containing a CPU and a GPU. Microsoft HoloLens also includes a custom Microsoft Holographic Processing Unit (HPU). The SoC and HPU each have 1 GB of LPDDR3 and share 8 MB of SRAM, and the SoC also controls 64 GB of eMMC and runs the Windows 10 operating system. The HPu processes and integrates data from the sensors, as well as handles tasks such as spatial mapping, gesture recognition, and speech and language recognition. Microsoft HoloLens includes IEEE 802.11ac Wi-Fi and Bluetooth 4.1 low energy (LE) wireless connectivity. The headset uses Bluetooth LE to connect to a Clicker, which is a finger-operated input device that can be used to select menus and functions.

[0109] Many applications are available for Microsoft HoloLens, such as Holographic Gallery, HoloStudio, a three-dimensional modeling application with Microsoft 3D printing capabilities, Autodesk Maya, a three-dimensional creation application, FreeForm, which connects Microsoft HoloLens with Autodesk Fusion 360, a cloud-based three-dimensional development application, and others.

[0110] Microsoft HoloLens using HPU can use intuitive and natural interface commands - voice, hand gestures and gestures. Gaze commands, such as head tracking, can let users command applications to focus on anything the user perceives. Any virtual application or button can be selected using an air tap method, similar to clicking a virtual computer mouse. A tap can be held to simulate a drag, moving the display. Voice commands can also be used.

[0111] The Microsoft HoloLens shell employs many components or concepts from the Windows desktop environment. The bloom gesture to open the main menu is accomplished by opening one hand, palm up and fingers extended. Windows can be dragged to specific locations, locked and / or resized. Virtual windows or menus can be pinned to multiple locations or multiple physical objects. Virtual windows or menus can move with the user or can be fixed in association with the user. Alternatively, they can follow the user as the user moves.

[0112] Developers can use the Microsoft HoloLens app for Windows 10 PC and Windows 10 Mobile devices to run applications and view real-time streams from the perspective of a Microsoft HoloLens user, as well as capture augmented reality photos and videos.

[0113] Almost all Universal Windows Platform apps can run on Microsoft HoloLens. These applications can be projected in two dimensions. Microsoft HoloLens currently supports selected Windows 10 APIs. Microsoft HoloLens apps can also be developed on Windows 10 PC. Holographic apps can use Windows Holographic APIs. Unity and Vuforia are applications that can be used. Applications can also be developed using DirectX and Windows API's.

[0114] Computer image viewing approach

[0115] In some embodiments of the invention, the optical head-mounted display uses a computer image viewing approach that includes the following steps to display three-dimensional objects or two-dimensional objects or other computer-generated objects and models located in three-dimensional space.

[0116] Figure 16B :

[0117] 1. Registration

[0118] 2. Viewing projection

[0119] Registration:

[0120] Different objects to be displayed by an optical head-mounted display computer image system, such as virtual anatomical models, instrument virtual models, geometric and surgical references and guides, are initially defined in respective independent model coordinate systems. In a registration process, spatial relationships between the different objects are defined and each object is transformed from its respective model coordinate system to a common overall coordinate system. Different techniques, described below, can be applied to the registration process.

[0121] For augmented reality optical head-mounted displays that superimpose computer generated objects with real-time views of the physical environment, the overall coordinate system is defined by the environment. A procedure, described below, called spatial mapping, creates an environment computer image that allows merging and registration with computer generated objects, thereby defining spatial relationships between computer generated objects and the physical environment.

[0122] View projection:

[0123] Once all objects to be displayed have been registered and transformed into the same overall coordinate system, they are ready for viewing on the display by transforming the coordinates of the display objects from the overall coordinate system to the view coordinate system and subsequently projecting them onto the display plane. This view projection step uses the viewpoint and view direction to define the transformation applied in this step. For stereoscopic displays such as optical head-mounted displays, two different view projections can be used, one for the left eye and one for the right eye. For augmented reality optical head-mounted displays, the position of the viewpoint and view direction relative to the physical environment can be known in order to correctly superimpose computer generated objects with the physical environment. When the viewpoint and view direction change, such as due to head movement, the view projection is updated so that the computer generated display follows the new view changes.

[0124] Eye tracking system

[0125] The present invention provides methods for using the human eye, including eye movement and eyelid movement as well as movements caused by the muscles surrounding the eye socket, to execute computer commands. The present invention also provides methods for executing computer commands through facial movement and head movement.

[0126] Command execution caused by eye movement and eyelid movement as well as movements caused by the muscles surrounding the eye socket, facial movement and head movement, are most advantageous in environments where the operator cannot use their hands to type on a keyboard or execute commands on a touchpad or other hand-held computer interface. These situations include, but are not limited to, industrial applications, including specifically automobile and airplane manufacturing, chip manufacturing, medical or surgical procedures, and many other potential applications.

[0127] In some embodiments, the optical head-mounted display can include an eye tracking system. Different types of eye tracking systems can be used. The embodiments provided below are in no way to be considered limiting to the present invention. Any eye tracking system known in the art can now be used.

[0128] Eye movements can be divided into fixation and saccades - i.e. when the eye is momentarily fixated on a location, and when it moves to another location, respectively. The resulting series of fixations and saccades can be defined as a scan path. The center of the visual field provides most of the visual information; the amount of input information from the periphery is less. Therefore, the fixation locations along the scan path illustrate the information locations processed during an eye tracking session, such as during a surgical procedure.

[0129] Eye trackers can measure the rotation or movement of the eye in a number of ways, such as by measuring the motion of an object attached to the eye, such as in the form of a contact lens, by optical tracking without direct contact with the eye, and by measuring the potential of electrodes placed around the eye.

[0130] If an eye attachment is used, it can be a special contact lens, such as one with an embedded mirror or magnetic field sensor. The motion of the attachment can be measured without the attachment sliding significantly as the eye rotates. Measuring a closely fitting contact lens can provide very accurate eye movement measurements. In addition, a magnetic search coil can be used to measure the horizontal, vertical, and torsional directions of movement of the eye.

[0131] Alternatively, non-contact optical methods of measuring eye movement can be used. The eye can reflect light, optionally infrared light; this is sensed using optical sensors or video cameras using this technology. The measurements are then used to infer the rotation and / or movement of the eye from changes in the reflections. Optical sensor or video-based eye trackers can use the corneal reflection (the so-called first Purkinje image) and the center of the pupil as tracking features, optionally changing over time. A more sensitive eye tracker, the dual Purkinje eye tracker, uses reflections from the front of the cornea (the first Purkinje image) and the back of the lens (the fourth Purkinje image) as tracking features. An even more sensitive tracking method is to image visual features from inside the eye, such as retinal blood vessels, and track these features as the eye rotates and / or moves. Optical methods, particularly those based on optical sensors or video registration, can be used for gaze tracking.

[0132] In some embodiments, optical or video-based eye trackers can be used. As the observer performs a function such as surgery, a camera focuses on one or both eyes and tracks their movement. An eye tracker can use the center of the pupil for tracking. An infrared or near-infrared non-collimated light can be utilized to produce a corneal reflection. The vector between the pupil center and the corneal reflection can be used to calculate the point of regard on the surface or gaze direction. Optionally, a calibration procedure can be performed at the beginning of eye tracking.

[0133] Bright pupil and dark pupil tracking can be employed. Their difference lies in the position of the illumination source relative to the optics. If the illumination is coaxial to the optical path, the effect of the eye is retro-reflective, as the light reflects from the retina, producing a bright pupil effect similar to red-eye. If the illumination source is off-axis to the optical path, the pupil appears dark, as the back-reflection from the retina is away from the optical sensor or camera.

[0134] Bright pupil tracking can achieve greater iris / pupil contrast, making it more reliable to track the eye with iris pigmentation. It can also reduce interference from eyelashes. Tracking can be performed in dark and very bright lighting conditions.

[0135] As mentioned above, optical tracking methods can include tracking eye movement including the pupil. Optical tracking methods can also include tracking movement of the eyelids as well as the peri-orbital and facial muscles.

[0136] In some embodiments, eye tracking devices are integrated in optical head-mounted displays. In some embodiments, head movement can be tracked simultaneously, for example, using an inertial measurement unit (see below) formed by a combination of accelerometers and gyroscopes.

[0137] In some embodiments, electrical potentials can be measured with electrodes placed around the eye. The eye produces an electrical potential field that can also be detected if the eye is closed. The electrical potential field can be modeled as being generated by a dipole with a positive pole at the cornea and a negative pole at the retina. It can be measured by placing two electrodes on the skin around the eye. The electrical potential measured in this way is called electrooculography.

[0138] If the eye moves from a central position to the periphery, the retina approaches one electrode while the cornea approaches the opposite electrode. Changes in the direction of the dipole and consequent changes in the electrical potential field cause changes in the measured electrooculogram signal. By analyzing these changes, eye movements can be assessed. Two separate directions of movement can be identified, horizontal and vertical. If a posterior electrode is used, an electrooculogram (EOG) component in the radial direction can be measured. The mean value of the electrooculogram channel referenced to the posterior electrode is typically used. The radial electrooculogram channel can measure the saccadic peak potential originating from the extraocular muscles at the beginning of a saccade.

[0139] Electrooculography can be limited to measure slow eye movements and detect gaze direction. However, electrooculography is well suited to measure fast or saccadic eye movements associated with gaze shifts and detect blinks. Unlike optical or video-based eye trackers, electrooculography can register eye movements even with the eyes closed. The main disadvantage of electrooculography compared to optical or video trackers is its relatively poor accuracy in gaze direction. Alternatively, both optical or video tracking and electrooculography can be used in combination in selected embodiments of the present application.

[0140] Sampling rates of 15, 20, 25, 30, 50, 60, 100, 120, 240, 250, 500, 1000 Hz or higher can be used. Any sampling frequency is possible. In many embodiments, sampling rates greater than 30 Hz are preferred.

[0141] Measuring position, direction, acceleration

[0142] The position, direction and acceleration of a human head, of parts of the human body such as hands, arms, legs or feet, and of parts of the body of a patient such as the head or limbs of a patient including the hips, knees, ankles, feet, shoulders, elbows, hands or wrists, and of any other body part, can be measured by using gyroscopes and accelerometers in combination. In selected applications, magnetometers can also be used. Such a measurement system utilizing any of these components can be defined as an Inertial Measurement Unit (IMU).

[0143] As used herein, the term Inertial Measurement Unit relates to an electronic device that measures and transmits information about the specific forces, angular rates and optionally the magnetic field around the body, using accelerometers and gyroscopes, and optionally magnetometers, in combination. The Inertial Measurement Unit or components thereof can be coupled or registered with a navigation system or robot, such as by registering the body or parts of the body within a shared coordinate system. Optionally, the Inertial Measurement Unit can be wireless, such as using a WiFi network or a Bluetooth network.

[0144] Pairs of accelerometers spread over a spatial area can be used to detect the difference (gradient) in proper acceleration of the frame of reference associated with these points.

[0145] Single- and multi-axis accelerometer models can be used to detect the magnitude and direction of acceleration as a vector, and can be used to detect direction (as the direction of weight changes), coordinate acceleration (as long as it produces gravity or a change in gravity), vibration, shock. In some embodiments, micro-machined accelerometers can be utilized to detect the position of the device or the surgeon's head.

[0146] Piezoelectric, piezoresistive and capacitive devices can be used to convert mechanical motion into electrical signals. Piezoelectric accelerometers rely on piezoelectric ceramics, or single crystal piezoresistive accelerometers can also be used. Capacitive accelerometers typically use a silicon micromachined sensing assembly.

[0147] In some embodiments, the accelerometers used can include small microelectromechanical systems (MEMS), including cantilevered beams with a detection mass, for example.

[0148] Optionally, the accelerometers can be integrated in the optical head-mounted device, and the output from both the eye tracking system and the accelerometers can be used to command execution.

[0149] With the inertial measurement units, the following exemplary information about the surgeon and the patient and various body parts can be captured: velocity, speed, acceleration, spatial location, position change, alignment, direction and / or motion direction (e.g., by continuous measurement)

[0150] Body parts of the surgeon and / or patient whose information is transmitted with the inertial measurement units include, but are not limited to: head, chest, torso, shoulder, elbow, wrist, hand, finger, arm, hip, knee, ankle, foot, toe, leg, internal organs such as brain, heart, lung, liver, spleen, intestine, bladder, etc.

[0151] Any number of inertial measurement units can be placed on the optical head-mounted display, the operator and / or the patient and, optionally, these inertial measurement units can be referenced to one another in a single or multiple coordinate systems or, alternatively, they can be referenced to one another in relation to the optical head-mounted display, a second and third or more optical head-mounted displays, a navigation system or robot and one or more coordinate systems used by such navigation system and / or robot. A navigation system can be used in conjunction with an optical head-mounted display without the use of inertial measurement units. For example, navigation markers including infrared markers, retro-reflective markers, radio frequency markers can be attached to the optical head-mounted display and, optionally, to the patient or various parts or segments of the patient's anatomy. The optical head-mounted display and the patient or patient's anatomy can be referenced to one another or registered in one or more coordinate systems used by the navigation system in this manner and the motion of the optical head-mounted display or the operator wearing the optical head-mounted display can be registered in one or more coordinate systems in relation to the patient. Once the virtual data of the patient and the real-time data of the optical head-mounted display are registered in the same coordinate system, any change in the position of the optical head-mounted display measured in relation to the patient in this manner, for example, using inertial measurement units, optical markers, navigation markers including infrared markers, retro-reflective markers, radio frequency markers and any other registration method described in the specification or known in the art, can be used to change the relationship between the virtual data of the patient and the real-time data of the patient so that the visual image of the virtual data of the patient and the real-time data of the patient viewed through the optical head-mounted display are always aligned regardless of the movement of the optical head-mounted display and / or the head of the operator and / or the operator wearing the optical head-mounted display. Similarly, when multiple optical head-mounted displays are used, for example, one for the primary surgeon and another, for example, two, three, four or more, for other surgeons, assistants, residents, researchers, nurses and / or visitors, other staff than the primary surgeon wearing the optical head-mounted display will also display a virtual image of the virtual data of the patient that is consistent with the corresponding real-time data of the patient viewed through the optical head-mounted display where the perspective of the virtual data of the patient and / or the surgical site is used for the location, position and / or orientation of the viewer using the optical head-mounted display and the eyes of each viewer. The foregoing embodiments can be implemented because the inertial measurement units, optical markers, radio frequency markers, infrared markers and / or navigation markers placed on the operator and / or the patient and any spatial positioning can be registered in the same coordinate system of the primary optical head-mounted display and any other optical head-mounted displays.The remaining patient and / or surgical site positions, orientations, collimations, and changes in position, orientation, and collimation can be monitored separately to align and / or superimpose the corresponding structures of the patient real-time data and patient virtual data of the additional optical head-mounted displays regardless of the patient and / or surgical site positions, orientations, and / or collimations.

[0152] With reference to Figure 1 , the system 10 is used for multiple observers using multiple optical head-mounted displays 11, 12, 13, 14, such as, for example, a primary surgeon, a second surgeon, a surgical assistant, and / or a nurse. The multiple optical head-mounted displays can be registered in a common coordinate system 15 using anatomical structures, anatomical landmarks, calibration phantoms, reference phantoms, optical markers, navigation markers, and / or spatial anchor points, such as, for example, the spatial anchor points used by Microsoft HoloLens. Preoperative data 16 of the patient can also be registered in the common coordinate system 15. Patient real-time data 18, such as, for example, from the surgical site, the spine, a hip arthrotomy site, a knee arthrotomy site, a bone cut, a changed surface, optionally with a minimally invasive portal, can be measured using one or more inertial measurement units, optical markers, navigation markers, image or video capture systems, and / or spatial anchor points. The patient real-time data 18 can be registered in the common coordinate system 15. Intraoperative imaging findings 20 can be registered in the common coordinate system 15. Or references, such as, for example, using optical markers, inertial measurement units, navigation markers, or spatial mapping 22, surgical room tables or room fixtures can be registered in the common coordinate system 15. The preoperative data 16 or the real-time data 18 including intraoperative measurements or a combination thereof can be used to generate, create, or modify a virtual surgical plan 24. The virtual surgical plan 24 can be registered in the common coordinate system 15. The optical head-mounted displays 11, 12, 13, 14 can project virtual data or digital holograms of virtual data into a left eye view using a view position and orientation of a left eye 26 and can project virtual data or digital holograms of virtual data into a right eye view using a view position and orientation of a right eye 28 of a user to create a shared digital holographic experience 30. Using virtual or other interfaces, a surgeon wearing an optical head-mounted display 11 1 can execute a command 32, such as, for example, to display a next planned bone cut, such as, for example, from a virtual surgical plan or imaging findings or intraoperative measurements, which can trigger the optical head-mounted displays 11, 12, 13, 14 to project a digital hologram of a superimposition of a next surgical step 34 onto the surgical site and aligned with the surgical site to a planned position / orientation.

[0153] By registering the patient real-time data, such as the surgical field, the patient virtual data, and each optical head-mounted display in the same shared coordinate system, they can superimpose the patient virtual data onto each viewer's patient real-time data from their respective perspective using each optical head-mounted display. Thus, the patient virtual data, including aspects of the virtual surgical plan, can remain superimposed and / or aligned with the patient real-time data regardless of the viewer's perspective and can remain collimated and / or superimposed as the viewer moves his or her head or body.

[0154] Novel user interface

[0155] One subject of the present invention is to provide a novel user interface in which human eye movements, including eye ball movements, and eyelid movements, including movements caused by the eye socket and the muscles surrounding the eye socket and selected cranial muscles, are detected by an eye tracking system and processed to execute predefined, actionable computer commands.

[0156] An exemplary list of eye movements and eyelid movements that can be detected by the system is provided in Table 1.

[0157] Table 1: Exemplary list of eye movements and eyelid movements detected by eye tracking software

[0158] o 1 blink

[0159] o 2 blinks

[0160] o 3 blinks

[0161] o Fast blink, e.g., less than 0.5 seconds

[0162] o Slow blink, e.g., more than 1.0 second

[0163] o 2 or more blinks in a short time interval, e.g., less than 1 second

[0164] o 2 or more blinks in a long time interval, e.g., more than 2 seconds (typically chosen to be less than the natural time interval between blinks)

[0165] o Left eye only

[0166] o Right eye only

[0167] o Both left and right eyes

[0168] o Left eye first, then right eye in a short time interval, e.g., less than 1 second

[0169] o Right eye first, then left eye in a short time interval, e.g., less than 1 second

[0170] o Left eye first, then right eye in a long time interval, e.g., more than 2 seconds

[0171] o First blink right eye, then left eye in a long time interval, e.g., more than 2 seconds

[0172] o Fast eye movement left

[0173] o Fast eye movement right

[0174] o Fast eye movement up

[0175] o Fast eye movement down

[0176] o Wide open eyes, hold for a short time interval, e.g., less than 1 second

[0177] o Wide open eyes, hold for a long time interval, e.g., more than 2 seconds

[0178] o Close both eyes for 1 second

[0179] o Close both eyes for 2 seconds or more

[0180] o Close both eyes, hold, then open and blink quickly

[0181] o Close left eye only for 1 second, 2 seconds

[0182] o Close right eye only for 1 second, 2 seconds

[0183] o Close left eye, then right eye

[0184] o Close right eye, then left eye

[0185] o Blink left eye, then right eye

[0186] o Blink right eye, then left eye

[0187] o Stare at live scene, virtual button for 1 second, 2 seconds, 3 seconds or more; activate function, e.g., zoom in or out

[0188] Any combination of blinks, eye movements, sequences and time intervals can be used to encode various types of commands. These commands, for example, can be computer commands that direct or manipulate surgical instruments or robots.

[0189] The present invention also provides methods of executing commands through facial movements and head movements.

[0190] An exemplary list of facial movements and head movements that can be detected by the system is provided in Table 2. (This list is just an example and is by no means exhaustive; any number or combination of movements is possible.)

[0191] Table 2: Exemplary list of facial actions and head actions detected

[0192] o Move head quickly right and hold

[0193] o Move head quickly left and hold

[0194] o Move head quickly down and hold

[0195] o Move head quickly down and hold

[0196] o Move head quickly right and back

[0197] o Move head quickly left and back

[0198] o Move head quickly down and back

[0199] o Move head quickly down and back

[0200] o Tilt head left and hold

[0201] o Tilt head right and hold

[0202] o Tilt head left and back

[0203] o Tilt head right and back

[0204] o Open mouth and hold

[0205] o Open mouth and close

[0206] o Twinkle nose once

[0207] o Twinkle nose twice

[0208] Exemplary commands performed using eye movements, eyelid movements, facial movements, and head movements are listed in Table 3.

[0209] Table 3: Exemplary command line table that can be performed by tracking eye movements, eyelid movements, facial movements, and head movements (this list is merely exemplary and in no way meant to be exhaustive; any number or combination of commands is possible; special purpose commands can also be performed in this manner).

[0210] o Click

[0211] o Point

[0212] o Move pointer

[0213] ■ Slow

[0214] ■ Fast

[0215] o Scroll, e.g., through images

[0216] ■ Fast scroll

[0217] ■ Slow scroll

[0218] o Scroll up

[0219] o Scroll down

[0220] o Scroll left

[0221] o Scroll right

[0222] o Drag

[0223] o Spin

[0224] o Register

[0225] o 2D and 3D toggle

[0226] o Toggle imaging modality

[0227] o Overlay image

[0228] o Fuse image

[0229] o Register image

[0230] o Crop

[0231] o Paste

[0232] o Copy

[0233] o Undo

[0234] o Redo

[0235] o Delete

[0236] o Purchase

[0237] o Provide credit card information

[0238] o Authorize

[0239] o Go to shopping cart

[0240] o Optical head mounted display on

[0241] o Optical head mounted display off

[0242] o Eye tracking on

[0243] o Eye tracking off

[0244] o Execute eye command on

[0245] o Execute eye command off

[0246] o Execute face command on

[0247] o Execute face command off

[0248] o Turn on surgical instrument (e.g., jigsaw, laser, etc.)

[0249] o Turn off surgical instrument

[0250] o Increase strength, speed, energy of surgical instrument

[0251] o Decrease strength, speed, energy of surgical instrument

[0252] o Change direction of surgical instrument

[0253] o Change orientation of surgical instrument

[0254] o Change any type of set-up surgical instrument

[0255] In some embodiments of the application, eye movement, eyelid movement, facial movement, head movement, alone or in combination, can be used as a signal digital code or digital sequence or machine operation sequence. For example, such digital sequence can be used to perform certain machine operation sequences.

[0256] Head movement controls movement of surgical instrument

[0257] In some embodiments of the application, head movement can be used to control a surgical instrument. For example, in a robotically assisted procedure with robotic haptic feedback, a surgeon can use his hands to control the direction of a surgical instrument. The surgeon can move his head forward. This forward movement is captured by an inertial measurement unit and translated into forward movement of a robotic arm that controls the direction of the surgical instrument along the surgical instrument. A backward movement of the head can be captured by the inertial measurement unit and translated into backward movement of the robotic arm that controls the direction of the surgical instrument along the surgical instrument.

[0258] In some embodiments of the application, eye movement, eyelid movement, facial movement, head movement, or combinations thereof, can be used to send Morse code signals. The International Morse Code encodes the Latin alphabet using a small set of punctuation and procedural signals as standardized sequences of longer and shorter signals called dots and dashes. Each character (letter or number) is represented by a single sequence of dots and dashes. The duration of a dash is three times the duration of a dot. Each dot or dash is followed by a brief silence equal to the duration of a dot. Word letters are separated by spaces equal to three dots (one dash), and words are separated by spaces equal to seven dots.

[0259] An example of using eye commands to perform Morse code is as follows; this is by no means limiting. Many different implementations are possible. For example, a quick blink of both eyes (typically less than 1 second) can be used to perform a dot, while a line can be performed by closing only the right eye (e.g., for 1 second). The letter A in Morse code is a dot followed by a line. Using this encoding of Morse code, the letter A can be performed by a quick blink of both eyes (dot) followed by closing the right eye for only one second (line). The letter B (line, three dots), can be performed by closing the right eye for only one second (line) followed by three quick blinks of both eyes (three dots), etc. For example, the letters can be separated by maintaining a two second or greater interval between the eye commands. Or, in another example, the letters can be separated by closing only the left eye for about one second.

[0260] Binary code can also optionally be performed using eye commands. For example, a quick blink of both eyes can represent the number 0, while closing only the right eye for about one second can represent the number 1. Or, closing only the right eye for about one second can represent the number 0, while closing only the left eye for about one second can represent the number 1. Many different types of encoding are possible. Other numbers can also be performed using some of the eye, eyelid, face and / or head movements shown in Tables 1 and 2, for example.

[0261] Many different languages can be performed in this manner. Computer languages are also optionally performed, such as Fortran, Pascal, C, C++, C--, Basic and many other computer languages known in the art.

[0262] In some embodiments of the present invention, eye, eyelid, face and head movements commands can be used in conjunction with or paired with voice commands, hand commands, gesture commands, keyboard commands, trackpad commands, mouse commands, graphical user interface commands and any other command input device known in the art. The optical head-mounted display also optionally one or more touch-sensitive sensors.

[0263] Another benefit of eye commands in a selected environment is the ability to maintain privacy or confidentiality of commands while navigating screens or performing commands. For example, in a hospital environment, with other patients or visitors nearby, eye commands can be used to view a patient's case or schedule a lab test or other diagnostic test, without onlookers knowing that these records are being reviewed or these tests are being scheduled.

[0264] In a meeting, the wearer of the optical head-mounted display can use eye commands to turn on a video or audio recording function or transmit to a remote site or remote conference room, without exposing that the recording function has been activated. This is in contrast to manually activating the recording function, such as the user pressing a button or touch-sensitive sensor on the optical head-mounted display to activate the recording function.

[0265] In some embodiments, a user can utilize eye motion, facial motion or head motion to direct a digital camera to take a photograph or video. Commands can include, but are not limited to, zoom in, zoom out, move target area left, right, up, down, take a photograph, take a series of photographs, turn flash on / off, start video recording, stop video recording, change resolution, increase resolution, decrease resolution.

[0266] Any other camera commands known in the art can be executed in this manner using eye motion, facial motion or head motion based commands. By utilizing one or more commands of this type, a user can maintain privacy while obtaining image information about the surrounding environment.

[0267] Eye commands are very useful for surgeons or operating room personnel to execute commands without using their hands, whereby a sterile field is maintained.

[0268] Fusing the physical world with a patient's image and other data

[0269] In some embodiments of the present invention, a surgical person, such as a surgeon, can view a patient's physical data or information, such as a surgical site or changes induced on a surgical site, through an optical head mounted display, while pre-existing patient data is superimposed on the physical visual image of the patient in situ.

[0270] The patient's pre-existing data can be imaging studies or image data or other types of data including metabolic or functional information.

[0271] The pre-existing patient data, including one or more imaging studies, or other types of data including metabolic or functional information, can be obtained outside of the time of a surgical procedure. For example, the pre-existing patient data can be obtained one, two, three or more days or weeks prior to a surgical procedure.

[0272] The pre-existing patient data, including one or more imaging studies, or other types of data including metabolic or functional information, is typically obtained from a patient or a surgical site in a location or in an object coordinate system that is different from the location or the object coordinate system of the pre-existing data when compared to the location or object coordinate system of the patient or the surgical site in situ. Thus, the pre-existing data of the patient or the surgical site is typically in a first object coordinate system and the real-time data of the patient or the surgical site is typically in a second object coordinate system; the first and second object coordinate systems are typically different from each other. The first object coordinate system with the pre-existing data needs to be registered with the second object coordinate system for the real-time data of the patient, such as the surgical site in situ.

[0273] Scanning techniques

[0274] The following is an exemplary list of scanning and imaging techniques that can be used or applied in aspects of the present application; this list is not exhaustive, but is exemplary. Those skilled in the art can identify other scanning or imaging techniques that can be used to implement the present application.

[0275] For detailed descriptions of these different scanning and imaging techniques, see, for example, Bushberg et al., The Essential Physics of Medical Imaging, 3rd Edition, Wolters, Kluwer, Lippincott, 2012.

[0276] X-ray imaging, two-dimensional, three-dimensional, supine, upright or other body position and posture, including analog and digital X-ray imaging

[0277] Digital tomosynthesis

[0278] Cone-beam computed tomography (CT)

[0279] Ultrasound

[0280] Doppler ultrasound

[0281] Elastic imaging, such as, using ultrasound or nuclear magnetic resonance

[0282] Computed tomography

[0283] Magnetic resonance imaging (MRI)

[0284] including, for example, functional magnetic resonance imaging, diffusion imaging, stroke imaging, magnetic resonance imaging with contrast agents

[0285] Functional magnetic resonance imaging (fMRI), such as, for brain imaging and functional brain mapping

[0286] Magnetic resonance spectroscopy

[0287] Positron emission tomography (PET)

[0288] Single photon emission computed tomography - computed tomography (SPECT-CT)

[0289] Positron emission tomography - computed tomography (PET-CT)

[0290] Positron emission tomography - magnetic resonance imaging (PET-MRI)

[0291] Vertical scanning, optionally in multiple planes or three-dimensionally, using any of the aforementioned modalities (including X-ray imaging, ultrasound, etc.)

[0292] Contrast agents

[0293] • such as, iodinated contrast agents for x-ray and computed tomography scans, or nuclear magnetic resonance contrast agents.

[0294] • The contrast agent can include antigens or antibodies for specific targeting of cells or tissues

[0295] • Other targeting technologies, such as the use of liposomes, can also be applied

[0296] • Molecular imaging

[0297] • To highlight metabolic abnormalities in the brain and to align surgical instruments to the metabolic abnormality area

[0298] • Any contrast agent known in the art can be used in conjunction with the present application.

[0299] Multi-dimensional imaging, reconstruction, and visualization

[0300] Various embodiments of the present application can be implemented in one, two, three, or more dimensions. The following is an exemplary list of potential dimensions, views, projections, angles, or reconstructions that can be applied; this list is not exhaustive, but is exemplary. One skilled in the art can identify additional dimensions, views, projections, angles, or reconstructions for implementing the present application. Exemplary dimensions are listed in Table 4.

[0301] Table 4: Exemplary list of potential dimensions, views, projections, angles, or reconstructions that can be displayed using virtual image display with optical head-mounted display, optionally stereoscopic

[0302] First dimension: superior-inferior, such as patient's physical data

[0303] Second dimension: medial-lateral, such as patient's physical data

[0304] Third dimension: anterior-posterior, such as patient's physical data

[0305] Fourth-sixth dimensions: 1, 2, or 3 dimensional head motion (and glasses motion / optical head-mounted display)

[0306] Seventh-ninth dimensions: 1, 2, or 3 dimensional instrument motion, such as related to the surgical field, organ, or head, including head motion

[0307] Tenth-thirteenth dimensions: 1, 2, or 3 dimensional arm or hand motion, such as related to the surgical field, organ, or head, including head motion

[0308] Fourteenth-sixteenth dimensions: virtual 3-dimensional data of the patient, such as obtained from scans or intraoperative measurements

[0309] Dimensions 17-19: Blood flow; in 1, 2 or 3 dimensions, e.g., related to surgical field, organ or head, including head motion

[0310] Dimensions 20-22: Temperature map (including changes caused by low or high temperature), thermal imaging, in 1, 2 or 3 dimensions, e.g., related to surgical field

[0311] Dimensions 25-28: Metabolic map (e.g., using MRS, positron emission tomography - computed tomography, single photon emission tomography - computed tomography), in 1, 2 or 3 dimensions, e.g., related to surgical field

[0312] Dimensions 29-32: Functional map (e.g., using functional magnetic resonance, positron emission tomography - computed tomography, single photon emission tomography - computed tomography, positron emission tomography, motion imaging), in 1, 2 or 3 dimensions, e.g., related to surgical field or patient

[0313] Any perspective projection is possible. Any angle of inclination is possible. Any curved surface is possible. Any curved perspective projection is possible. Any combination of one, two and three dimensional data between different types of data is possible.

[0314] Registration of real-time data and virtual data through optical head-mounted display

[0315] In some embodiments, patient virtual data can be superimposed on real-time data seen through the optical head-mounted display. The virtual data can be raw data in unprocessed form, e.g., preoperative images of the patient, or can be processed data, e.g., filtered data or segmented data.

[0316] Segmentation of data

[0317] When patient images are overlaid onto real-time data seen through an optical head-mounted display, image segmentation can be required in many embodiments. Any known algorithm in the art can be used for this purpose, such as thresholding, seed point techniques, firewire, deformable models, statistical models, active shape models, level set methods, marching cubes algorithm, artificial neural networks, deep learning techniques, or combinations thereof, among others. Many of these algorithms are part of open source or commercial libraries, such as the Insight Segmentation and Registration Toolkit (ITK), the Open Source Computer Vision Library OpenCV, G'MIC (GREYC's Magic for Image Computing), Caffe, or MATLAB (MathWorks, Natick, Mass.). Figure 2 A representative workflow for segmentation and follow-up is provided in FIG. 6. An optional preoperative imaging study 40 can be obtained. An optional intraoperative imaging study 41 can be obtained. The preoperative 40 or intraoperative 41 imaging study can be segmented 42, extracting, for example, surfaces, volumes, or key features. An optional three-dimensional reconstruction or three-dimensional rendering 43 can be generated. The preoperative 40 or intraoperative 41 imaging study and any three-dimensional reconstruction or three-dimensional rendering 43 can be registered in a common coordinate system 44. The preoperative 40 or intraoperative 41 imaging study and any three-dimensional reconstruction or three-dimensional rendering 43 can be used to generate a virtual surgical plan 45. The virtual surgical plan 45 can be registered in the common coordinate system 44. The surgical site 46 can be registered in the common coordinate system 44. Intraoperative measurements 47 can be obtained and can be used to generate the virtual surgical plan 45. An optical head-mounted display 48 can project or display virtual data or a digital hologram of virtual data 49 overlaid on and aligned with the surgical site. The optical head-mounted display 48 is configured to use a built-in camera or image capture or video capture system 50 to optionally detect and / or measure the position and / or orientation and / or alignment of one or more optical markers 51, which can be used for coordinate measurements 52 and can be part of the intraoperative measurements 47.

[0318] Registration software and algorithms

[0319] Registration of real-time data and virtual data can be performed using various techniques known in the art. These include, but are not limited to, surface registration algorithms such as Iterative Closest Point algorithms, statistical models, Active Shape Models, mutual information based or other volume registration algorithms, object recognition, pattern recognition or computer vision techniques, deep learning or other artificial intelligence methods. The data processed can consist of mesh data, parametric surface data, point cloud data, volume data or combinations thereof, for example. These methods are known in the art and have been implemented in publicly and / or commercially available code libraries and application programming interfaces such as Insight Segmentation and Registration Toolkit (ITK), the open source computer vision library OpenCV, Elastix, Plastimatch or the Medical Image Registration Toolkit (MIRTK).

[0320] Optical head-mounted display superimposes virtual data and real-time data

[0321] In some embodiments, segmented data or raw data can be superimposed on the patient real-time data seen through the optical head-mounted display. This superimposition can occur in an unregistered form, i.e., the patient virtual data can not be aligned with the real-time data seen through the optical head-mounted display. In this case, the surgeon wearing the optical head-mounted display can move the head in the direction of the corresponding features of the superimposable virtual data and patient real-time data. The surgeon or operator can also move and reorient the virtual data using other means such as a trackball or virtual display interface displayed in the optical head-mounted display, independent of the surgeon / operator head motion. The operator can adjust the magnification of the real-time data so that the size, shape, length, thickness of certain features of the virtual data match the size, shape, length, thickness of the actual data at a given distance from the subject / patient.

[0322] For example, during a brain surgery, a surgeon can view exposed gyri and sulci of a patient's brain while viewing real-time data on a heads-up display. The optical head-mounted display can display a virtual three-dimensional model of the patient's gyri and sulci. The surgeon can selectively adjust the magnification of the three-dimensional model to match the size or width or length of the corresponding gyri and sulci in the real-time data. The surgeon can select to adjust the transparency or opacity of the virtual data displayed in the optical head-mounted display. The ratio of virtual data to real-time data transmitted through the optical head-mounted display can be 1 : 10, 1 : 9, 1 : 8, 1 : 5, 1 : 2, 1 : 1, 2 : 1, 3 : 1, 5 : 1, 8 : 1, 10 : 1, and fractions or multiples thereof. Any combination of transparency or opacity of the virtual data and real-time data is possible. The direction of movement of the surgeon's head can superimpose virtual features, such as the patient's gyri and sulci, and the patient's real-time data.

[0323] Once the data is superimposed, the surgeon can select to register the virtual data with the real-time data. Registration is as simple as the surgeon visually confirming whether the virtual data and real-time data substantially match or substantially superimpose, as described above. At this point, the surgeon can optionally reference the real-time data and / or coordinate system of the real-time data to the virtual data and / or coordinate system of the virtual data (2, 3, or more dimensions). Once the data is registered, the surgeon can move his head to any desired position or direction, such as to view the patient's brain or lesion and adjacent structures, such as sensitive structures, from a different perspective. The inertial measurement unit of the optical head-mounted display will register the head motion, the direction of the head motion, the latest head position, and the head direction. Changes in the position and direction of the surgeon's head can be applied to the virtual data simultaneously or, if desired, non-simultaneously. The virtual data, which can now be superimposed with the new position and direction of the real-time data that results, can additionally be enlarged or reduced in size according to the distance from the inertial measurement unit, such that in some embodiments the size, shape, and dimensions of the virtual data will approximate or match the size, shape, and dimensions of the real-time data, regardless of the distance, position, and direction of the surgeon's head.

[0324] For the purpose of registering virtual data and real-time data, the optical head-mounted display is optionally placed in a fixed position, such as mounted on a stand or a tripod. When the optical head-mounted display is placed in a fixed position, the surgeon can view the real-time data and can choose to register and / or display on a monitor with the camera. Then, the virtual data can be superimposed and the matching and registration of the virtual data and real-time data can be performed. At this point, the surgeon or operator can remove the optical head-mounted display from the fixed position and the surgeon can wear the optical head-mounted display during the surgical procedure.

[0325] The virtual data is optionally displayed in different colors, such as red, green, yellow, etc. Optionally, only the outlines of selected features of the virtual data are displayed. For example, these features can be the sulci of the patient's brain (such as black lines or lines of black or other colors) and the gyri that border these sulci are not visualized. Or, for example, only one lesion is displayed, such as a tumor, a glioblastoma, in the case of a brain. Or, the virtual data of normal and pathological tissue can be displayed in combination.

[0326] The virtual data can be registered with the real-time data seen through the optical head-mounted display. The registration can be performed using any method known in the art in order to register or refer to each other in 2, 3 or higher dimensions the virtual data and the real-time data.

[0327] In some embodiments, the registration of the virtual data and the real-time data is to be maintained throughout the surgical procedure. In some embodiments, the registration of the virtual data and the real-time data will be maintained during the selection of the surgical procedure or a surgical procedure plan, which can be or can include a virtual surgery plan, such as a preoperatively generated surgery plan.

[0328] In some embodiments of the present application, the superimposition of the virtual data and the real-time data by the optical head-mounted display occurs simultaneously. In some embodiments, the superimposition of the virtual data and the real-time data by the optical head-mounted display is not simultaneous. For example, the virtual data can be superimposed intermittently.

[0329] The virtual data can be transparent, semi-transparent or opaque. If the virtual data are opaque, they can be displayed intermittently so that the operator or surgeon can see how they project in relation to the patient real-time data.

[0330] If the combination of virtual data is displayed simultaneously with the real-time data, different types of virtual data can be displayed in different colors. Representative combinations of virtual and real-time data are provided below. The following is merely illustrative and in no way meant to limit the present application:

[0331] Real-time data: patient's brain; surgically exposed gyri and sulci.

[0332] Real-time data: surgical instruments, such as a biopsy needle or a cutting tool

[0333] Virtual data: a patient's brain with gyri and sulci derived from an imaging modality and optionally segmented from the imaging modality, such as a computed tomography scan or a magnetic resonance scan

[0334] Virtual data: a brain tumor deep within the brain

[0335] Virtual data: the same surgical instrument currently used by the surgeon, in a virtual image of the instrument, the virtual data indicates a desired direction, position or orientation of the surgical instrument.

[0336] Any of the foregoing virtual data can be displayed in two dimensions or in three dimensions. Multi-dimensional displays, as outlined elsewhere in the specification, are also possible.

[0337] For example, normal tissue of the patient, such as normal brain tissue, can optionally be displayed in two dimensions, such as using a grayscale image, while abnormal tissue of the patient, such as a stroke, a hemorrhage or a tumor, can be displayed in three dimensions. The optical head-mounted display can display any combination of two-dimensional, three-dimensional and multi-dimensional images; the optical head-mounted display can superimpose any combination of two-dimensional, three-dimensional and multi-dimensional images on the real-time data of the patient.

[0338] The virtual two-dimensional, three-dimensional and multi-dimensional data can be generated or acquired by different data acquisition techniques, such as different imaging studies, etc.

[0339] Locking or moving virtual data

[0340] In some embodiments of the invention, the virtual data can be associated with locking the surgeon or the operator, or associated with locking the patient or a certain target anatomical structure within the patient. This means that the virtual data will not move in the OHMD display even if the surgeon moves his head or body or part of the patient's anatomy is moving. For example, the optical head-mounted display can display a virtual image of a target tissue or adjacent tissue once registered. The virtual image of the target tissue or adjacent tissue can be, for example, an image through a tumor or other type of pathological tissue. When the surgeon or operator moves his head or body during the surgical procedure, the virtual data will not move but will be displayed within the same location.

[0341] In some embodiments of the present application, the virtual data can be associated with movement of the surgeon or operator, or with movement of the patient or some target anatomy within the patient. This means that if the surgeon moves his head or is moving the patient's body or part of the anatomy, the virtual data will move in the OHMD display. For example, once registered, the optical head-mounted display can display a virtual image of the target tissue or adjacent tissue. The virtual image of the target tissue or adjacent tissue can be, for example, an image through a tumor or other type of pathological tissue. As the surgeon or operator moves his head or body during the surgical procedure, the virtual data will move and change its position and orientation in the same way as the surgeon moves his head or body, typically reflecting changes in perspective or view angle created by the surgeon moving his head or body.

[0342] Optionally, the virtual data moves with a greater virtual distance or a greater angle, or a smaller virtual distance or a smaller angle than the movement of the surgeon's head or body.

[0343] Improving accuracy of movement or reorientation of virtual data

[0344] Once registration between the virtual data and the physical data is complete, movement or reorientation of the virtual data follows, for example, movement of the surgeon's head or body or movement of the operating arms or hands, or movement of the patient or some body part, for example, by using an inertial measurement unit of the optical head-mounted display to monitor movement and changes in position and / or orientation of the surgeon's head.

[0345] In some embodiments, optical or radio frequency (RF) trackers or other tracking devices known in the art can be applied to the optical head-mounted display and / or the patient, including selected body parts or target tissue of the patient, for example, the patient's knee. Using standard surgical navigation techniques known in the art, the spatial position of the optical or RF trackers can be registered, for example, for the initial pose or position or location. Movement of the trackers, for example, caused by movement of the surgeon's head or body or by movement of at least a portion of the patient, can be tracked using the navigation system. Then, using information about changes in position, changes in orientation or changes in direction of movement of the surgeon's head or the patient or both, the virtual data is updated accordingly, or the virtual data displayed by the optical head-mounted display is updated accordingly, or both. In this way, the virtual data and the real-time data can be superimposed by the optical head-mounted display, typically in a precise manner.

[0346] Optionally, position, orientation, direction, etc. data generated by the inertial measurement unit can be used in conjunction with such data generated by the surgical navigation system. The combination of data can be beneficial to more accurately measure changes in position or direction of the surgeon's head, body, operating arm, hand, or patient.

[0347] Using virtual data in 2 or more dimensions

[0348] In some embodiments of the application, the optical head-mounted display can display two-dimensional virtual images of the patient. The images can be transmissive images, such as X-ray or computed tomography scout scans. The images can be cross-sectional images of selected anatomical structures of the patient. The images can be original images or reformat, reconstructed, or partially segmented images of the patient.

[0349] In some embodiments of the application, the surgeon will view real-time data of the patient through the optical head-mounted display, such as exposed brain surface with patient gyri and sulci. The surgeon can register the virtual data of the patient with the real-time data of the patient, such as a magnetic resonance scan of the patient's brain. The registration can be in 2, 3 or more dimensions. The registration of real-time data and virtual data can include registering different types of virtual data, such as different types of normal or diseased tissue, using different imaging modalities, different sizes for different types of normal or diseased tissue, etc. Multiple two-dimensional scan planes can be displayed simultaneously. These two-dimensional scan planes can be variable angle parallel or non-parallel planes, orthogonal or non-orthogonal planes.

[0350] Scrolling, moving virtual data over real-time data

[0351] In some embodiments of the application, the surgeon or operator can optionally scroll through a set of continuous or non-continuous virtual two-dimensional image data and three-dimensional image data being superimposed on real-time data of the patient, typically from the same anatomical site, such as the brain, spine, hip, knee, etc. The scrolling can be guided through any type of user interface known in the art. For example, the surgeon can use a virtual interface projected by the optical head-mounted display in which he or she can move a virtual arrow up or down or left or right to scroll the image backward or forward, or for example, rotate the image, or display the image at different multiplanar angles or change the perspective or projection angle.

[0352] The surgeon can optionally scroll or move the virtual image data by moving his or her head back and forth, such as to scroll backward or forward in a virtual image volume. For example, the surgeon can move his or her head left or right to rotate the image, or display the image at different multiplanar angles or change the perspective or projection angle of the three-dimensional image.

[0353] The surgeon can optionally scroll the virtual image data by moving his or her hand or finger or any other body part forward or backward, such as to scroll back or forward in a virtual image volume. The surgeon can move his or her hand or finger or any other body part left or right to rotate the image or to display the image at different multiplanar angles or to change the perspective or projection angle. The surgeon can move his or her hand or finger in a rotating or turning motion to rotate or turn the virtual data. Any combination of head or hand or eye and other body signals can be used to change the display of the virtual data.

[0354] These screen changes of the virtual data are optionally performed in the optical head-mounted display using the same variable values of the location, position, orientation, angle, direction, and motion of the surgeon's body part used to trigger the screen change. Alternatively, any one of the related changes in location, position, orientation, angle, direction, and motion of the virtual data can be performed using a magnification factor or a reduction factor related to the change in location, position, orientation, angle, direction, and motion of the surgeon's body part. These magnification factors or reduction factors can be linear or non-linear, such as exponential or logarithmic. In some embodiments, the further the surgeon's body part controlling the movement of the virtual data in the OHMD display is from its original position, the greater the change in the virtual data in the optical head-mounted display caused by the movement. In some embodiments, the further the surgeon's body part controlling the movement of the virtual data in the OHMD display is from its original position, the smaller the change in the virtual data in the optical head-mounted display caused by the movement.

[0355] Using virtual data in 3 or more dimensions

[0356] In some embodiments of the application, the optical head-mounted display can display a three-dimensional virtual image of the patient. The three-dimensional image of the patient can include three-dimensional images of different types of anatomy, such as in the area of the intended surgery or surgical site.

[0357] A three-dimensional reconstruction of the image data or other data of the patient can be generated pre-operatively, intra-operatively, and / or post-operatively. The virtual three-dimensional image can include the entire anatomical area or a select tissue or anatomical area. For example, the optical head-mounted display can display different tissues virtually in three-dimensions using different colors. Normal and pathological tissues can be displayed in this manner.

[0358] For example, normal tissues can include brain tissue, heart tissue, lung tissue, liver tissue, vascular structures, bone, cartilage, spinal tissue, intervertebral discs, nerve roots. The optical head-mounted display can virtually visualize any tissue.

[0359] For example, registering patient virtual data and real-time data at the surgical site

[0360] In some embodiments of the application, the patient virtual data displayed by the optical head mounted display and the patient real-time data seen through the optical head mounted display are spatially co-registered with each other, such as in a common coordinate system, with one or more optical head mounted displays in the same coordinate system. The virtual and physical surgical instruments and implant components can also be co-registered in the common coordinate system. The spatial co-registration has the advantage that the simultaneous display of the patient's virtual and real-time data is not affected or less affected when the surgeon moves his head or body, when the optical head mounted display moves or when the patient moves. Thus, the perspective of the patient real-time data seen by the surgeon's eyes through the optical head mounted display, such as the real-time surgical field, can remain the same as the perspective of the patient virtual data seen by the surgeon's eyes through the display of the optical head mounted display device, such as the virtual surgical field, virtual surgical plane, virtual path, virtual cutting path or plane, projected into the surgeon's eyes, even if the surgeon moves his head or body. In this way, the surgeon does not need to re-think or adjust his hand-eye coordination because the patient real-time data seen through the surgeon's eyes and the patient virtual data seen through the OHMD display are superimposed, which is fundamentally different from other approaches, such as surgical navigation approaches in operating rooms with separate computer monitors where the surgeon's perspective is different from his or her perspective of the real-time data of the patient and the surgical field. Furthermore, with surgical navigation, a first virtual instrument can be displayed on a computer monitor that is an image of a physical instrument tracked with navigation markers, such as infrared or radio frequency markers, and the position and / or orientation of the first virtual instrument can be compared to the position and / or orientation of a corresponding second virtual instrument generated in a virtual surgical plan. Thus, with surgical navigation, the position and / or orientation of the first and second virtual instruments are compared.

[0361] With guidance provided in a mixed reality environment, such as with a stereoscopic display similar to an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed on a joint, spinal column or surgical site. Furthermore, a physical guide, tool, instrument or implant can be aligned with a two- or three-dimensional image of the virtual surgical guide, tool, instrument or implant. Thus, guidance in a mixed reality environment does not require the use of multiple virtual images of a guide, tool, instrument or implant and does not require the comparison of the position and / or orientation of multiple virtual images of a virtual guide, tool, instrument or implant.

[0362] In some embodiments, the virtual data can move in association with the surgeon or operator, or in association with the patient or some target anatomical structure within the patient. This means that if the surgeon moves his head or is moving the patient's body or a portion of the anatomy, the virtual data will move in the OHMD display. For example, once the optical head mounted display, the patient virtual data and the patient real-time data are registered in the common coordinate system, the optical head mounted display can display a virtual image of the target tissue or adjacent tissue. The virtual image of the target tissue or adjacent tissue can be, for example, an image of a tumor or other type of pathological tissue or a spinal column or a pedicle of a vertebra. As the surgeon or operator moves his head or body during the surgical procedure, the virtual data will move and change its position and orientation in the manner that the surgeon moves his head or body, typically reflecting the changes in perspective or view angle that the surgeon creates by moving his head or body. The virtual data can include a three-dimensional image of a surgical tool or instrument, such as a needle for kyphoplasty or vertebroplasty, where the virtual image of the needle shows its intended position, orientation or path in relation to the spinal column and / or the pedicle. The virtual data can also include a medical device, such as a pedicle screw, where the virtual data of the pedicle screw shows its intended position, orientation or path in relation to the spinal column, and / or the pedicle, and / or the vertebral body.

[0363] In some embodiments, the registration is performed using at least three or more points that can be superimposed or fused into a common object coordinate system for the virtual data and the real-time data. The registration can also be performed using surfaces or three-dimensional shapes of anatomical structures that exist in the patient virtual data and the real-time data. In this case, the virtual surfaces can be moved until they substantially match the active surfaces of the patient, or the virtual shapes can be moved until they substantially match the live shapes of the patient.

[0364] The registration of the patient virtual data and the patient real-time data can be achieved using different means. The following is by no means intended to limit the present application, but is merely exemplary.

[0365] Registration of patient virtual data and patient real-time data using directly or indirectly connected object coordinate systems

[0366] If virtual data (e.g., image data of a patient) is acquired from the patient in a first object coordinate system and real-time data (e.g., during surgery) is observed or acquired from the patient in a second target coordinate system, where the first and second target coordinate systems can be directly connected (e.g., physically) or indirectly connected (e.g., non-physically), then registration of the virtual data and real-time data of the patient can be performed. Direct connection of the first and second object coordinate systems can be, for example, a physical connection between the first and second object coordinate systems. For example, the patient can be moved from the first object coordinate system to the second object coordinate system along the length of a tape measure. Alternatively, the patient can be scanned with a scanner, such as a computed tomography scanner or a nuclear magnetic resonance scanner, and then the scanner table can be moved out of the scanner for surgery to be performed on the patient who remains on the scanner table. In this case, the scanner table can be a form of physical connection between the first and second target coordinate systems, and the length of the table movement between the scanning position and the scanner position outside (for real-time data, e.g., surgery) can define the coordinate transformation from the first object coordinate system to the second object coordinate system.

[0367] Indirect connection between the first (virtual data) object and the second (real-time data) object can be established, provided that the patient moves between acquiring the virtual data (e.g., using an imaging examination) and acquiring the real-time data (e.g., when performing surgery along a defined path, where the orientation and angle of the path are known, such that the first and second object coordinate systems can be referenced to each other, and using the defined path of the patient and the known information of the virtual data, an object coordinate transformation can be applied, and the real-time data of the patient and the optical head-mounted display can be registered in a common coordinate system) in one or more unknown orientations, lengths, or magnitudes. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0368] Registration of the virtual data and real-time data of the patient can also be performed without direct or indirect connection of the object coordinate systems using other means and methods that will be explained in the following paragraphs and columns, such as when the patient performs one or more unknown orientations, lengths, or magnitudes of movement. Combinations of all the different registration methods described in this specification are possible, such as switching registration methods during an operation or using multiple registration methods simultaneously in order to improve the accuracy of the registration.

[0369] Registration using spatial mapping

[0370] Real-time data can be static, or more desirably, continuously updated with additional information to incorporate changes in the position or surface of one or more objects or environmental structures. Additional information can be acquired through depth sensors, laser scanners, structured light sensors, time-of-flight sensors, infrared sensors, or tracking probes.

[0371] Real-time data can be static, or more desirably, continuously updated with additional information to incorporate changes in the position or surface of one or more objects or environmental structures. Additional information can be acquired through depth sensors, laser scanners, structured light sensors, time-of-flight sensors, infrared sensors, or tracking probes.

[0372] For initial spatial mapping and updates to the mapping data, commonly available software code libraries can be used. For example, this functionality can be provided by the Microsoft HoloToolkit or Google Project Tango platform. Various techniques for spatial mapping and tracking have been described, including those described in US 9,582,717, which is expressly incorporated herein by reference.

[0373] Using visual anatomical features to register patient virtual data and patient real-time data

[0374] a) Surgeon or operator can visualize registration of patient virtual data relative to patient real-time data

[0375] In some embodiments, the surgeon or operator can visually align or match the patient virtual data with the patient real-time data. This visual alignment or matching of the patient virtual data and the patient real-time data can be performed by moving the optical head mounted display, such as by the head movement of the operator wearing the optical head mounted display. In this example, the patient virtual data is displayed in a fixed manner, rather than changing the perspective as the operator moves the optical head mounted display. The operator will move the optical head mounted display until the patient real-time data is aligned or superimposed on the fixed projection of the patient virtual data. Once a satisfactory alignment, match or superimposition of the patient data with the patient virtual data is achieved, the surgeon can execute a registration command, such as by a voice command or a keyboard command. The patient virtual data and the patient real-time data are now registered. At this point, after registration is complete, the patient virtual data will move with the movement of the optical head mounted display, such as by integrated inertial measurement units, image and field of view tracking movement; such as with image and / or video capture systems, and / or using additional navigation systems with optical or radio frequency or other trackers that can be attached to the patient, the surgical site, the skeleton or any other tissue of the patient, the surgeon, the surgeon's arms, the surgeon's head or the optical head mounted display worn by the surgeon, while measuring in the image or field of view using anchor points.

[0376] Thus, once a satisfactory alignment or match is achieved, the surgeon can execute a command for successful registration. The registration can include a change in at least one of the position, orientation and magnification of the virtual data and the real-time data in order to achieve the alignment or match. The magnification applied to the virtual data can be indicative of the distance from the optical head mounted display or the surgeon's head to the matching tissue. As a means of maximizing registration accuracy, the estimated distance between the optical head mounted display and the target tissue or skin surface or other reference tissue can be confirmed by an optional distance physical measurement, particularly if the optical head mounted display is in a fixed position, such as on a stand or tripod, which can be optionally used during the initial registration. After successful alignment or match, the surgeon's command can register in the same common coordinate system, such as the patient virtual data and the patient real-time data or the optical head mounted display. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0377] In some embodiments of the present application, the visual anatomical data can be gyri or osteophytes or bone spurs or pathologic bone deformities or tumor nodules or nodules, such as on the surface of the liver or the brain.

[0378] In some embodiments of the present application, the method described herein can be used to register patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or shape, such as the shape of a bone after milling or reaming, or the perimeter of tissue, such as the perimeter of a bone cut or tissue volume or other tissue feature of the live patient, can be matched, superimposed, and / or aligned with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue feature in the patient virtual data. For example, in a virtual surgical plan developed for a patient, the surgeon's eyes see the patient virtual data through the optical head-mounted display device display, and the perspective of the patient real-time data seen by the surgeon's eyes through the optical head-mounted display device is substantially the same. After the surgical tissue alteration, matching, superimposing, and / or aligning the patient real-time data and the patient virtual data can use the same methods described in the foregoing or any other registration method described in this specification or any other registration method known in the art. It is to be understood that the foregoing description is exemplary of the application only, and is intended to provide an overview for the understanding of others skilled in the art rather than as a Figure 3 , Figure 3The example illustrates registering a digital hologram or virtual data for an initial surgical step, performing the surgical step and re-registering one or more holograms for subsequent surgical steps. An optical head mounted display can project or display virtual data or a digital hologram of virtual data of the patient 55. The digital hologram is optionally fixed to the optical head mounted display so that it moves with movement of the optical head mounted display 56. The surgeon can move the optical head mounted display until the digital hologram of virtual data or the patient virtual data superimposes and aligns with the patient real time data, such as the surgical site 57. The virtual data or digital hologram of virtual data can then be registered using the same or similar coordinates as the real time data with which the digital hologram was superimposed 58. The surgeon can then perform one or more predetermined surgical steps, such as bone cuts 59. The virtual data or digital hologram of virtual data is optionally registered or re-registered after a surgical change using real time data 60. The virtual data or digital hologram of virtual data after the surgical change is optionally displayed by the optical head mounted display 61. The virtual data or digital hologram of virtual data after the surgical change is optionally fixed relative to the optical head mounted display so that it moves with movement of the optical head mounted display 62. The surgeon can move the optical head mounted display until the patient virtual data or digital hologram of virtual data after the surgical change superimposes and aligns with the patient real time data after the surgical change 63. The virtual data or digital hologram of virtual data can then be registered using the same or similar coordinates as the real time data after the surgical change with which the digital hologram was superimposed 64. The surgeon can then perform one or more predetermined subsequent surgical steps, such as bone cuts, milling or drilling 65. The foregoing steps are optionally repeated until the surgical procedure is complete 66. The virtual surgical plan 67 can be used. Optionally, the optical head mounted display 68 can display the patient’s native anatomy, including after the first surgical change. The optical head mounted display optionally displays a digital hologram of the subsequent surgical steps 69.

[0379] b) automatically or semi-automatically registering the patient real time data and the patient virtual data using image processing and / or pattern recognition and matching techniques

[0380] c) In some embodiments of the application, image processing techniques, pattern recognition techniques or deep learning / artificial neural network based techniques can be used to match the patient virtual data and the patient real time data. Optionally, image processing and / or pattern recognition algorithms can be used to identify certain features, such as gyri or sulci on the surface of the brain of the patient virtual data. Ears, which include a unique shape, can also be used to match the patient virtual data and the patient real time data.

[0381] For example, during a brain surgery, a patient can be placed on an operating table. Optionally, a cleaning or disinfecting solution can be applied to the shaved skull, such as with povidone-iodine. An optical head-mounted display can be placed on the patient, or on a tripod or worn by the surgeon, such as with the patient's head turned to the side with the patient's ear and side of the skull in view. The optical head-mounted display will be placed so as to include the portion of the patient in view that the patient virtual data is to be displayed.

[0382] The patient virtual data can be displayed in the optical head-mounted display. The patient virtual data can include a visualization of the patient's skin, ear, or nose, or other data, such as from pre-operative MRI data. The virtual data of the patient's skin or other structure, such as the patient's ear or nose, can be displayed simultaneously with the patient real-time data. The patient virtual data can then be moved, re-oriented, re-aligned, and optionally zoomed in or out until the alignment, match, or overlay is satisfactory. Optionally, the optical head-mounted display can also be moved during this process to achieve a satisfactory size match between the patient virtual data and the patient real-time data, optionally without zooming in or out of the patient virtual data.

[0383] Once a satisfactory alignment, match, or overlay between the patient virtual data and the real-time data is achieved, the surgeon can execute a command indicating a successful registration. By integrating the position, orientation, or changes in orientation of the optical head-mounted display as measured by an inertial measurement unit, image and field of view tracking (e.g., with anchor points in the image or field of view using an image and / or video capture system), and / or a navigation system connected to the optical head-mounted display, the view of the patient real-time data obtained through the optical head-mounted display can be used to move the patient virtual data using substantially the same object coordinates as the patient virtual data and the patient real-time data, thereby maintaining registration during the procedure regardless of how the optical head-mounted display is moved, such as by the surgeon wearing the optical head-mounted display making head movements, and ensuring that the patient virtual data correctly overlays the patient real-time data when projected into the surgeon's field of view.

[0384] After the patient virtual data is successfully registered to the patient skin or other structure such as an ear or nose, the surgeon or assistant can place a marker or calibration or registration phantom or device on the patient, such as near the intended site of a craniotomy. The marker or calibration or registration phantom or device cannot be covered by any subsequent draping or surgical cover. Then, using the real-time marker or calibration or registration phantom or device placed on the patient, the patient virtual data and patient real-time data are re-registered by cross-referencing, such as the real-time data of the patient skin or other structure such as an ear or nose, by registering the patient skin or other structure such as an ear or nose in the same coordinate system as the marker or calibration or registration phantom or device placed on the patient, such as by co-registering the patient virtual data of the patient skin or other structure such as an ear or nose or bony excrescences or spurs or other bony anatomy or deformities, and the real-time data of the marker or calibration or registration phantom or device. The distance, offset, angular offset or overall difference in coordinates between the patient skin or other structure such as an ear or nose or bony excrescences or spurs or other bony anatomy or deformities and the marker or calibration or registration phantom or device attached to the patient can be measured and used to switch registration of the patient virtual data to the patient real-time data of the patient skin or other structure such as an ear or nose, and the real-time data of the marker or calibration or registration phantom or device. Optionally, the real-time data of the patient skin or other structure such as an ear or nose, and the real-time data of the marker or calibration or registration phantom or device can be kept registered. Alternatively, the system can assess whether registration to the real-time data of the patient skin or other structure such as an ear or nose, or the real-time data of the marker or calibration or registration phantom or device is more accurate, and the system can switch back and forth between the two. For example, if the distance from the optical head-mounted display to the patient's skin or other structure such as an ear or nose or bony excrescences or spurs or other bony anatomy or deformities increases or decreases beyond a certain level, such as an optionally pre-defined threshold, or is partially covered by a drape, the system can switch registration to the real-time data of the marker or calibration or registration phantom or device. The reverse is also possible. Or, if the angle of the patient's skin or other structure such as an ear or nose or bony excrescences or spurs or other bony anatomy or deformities to the optical head-mounted display increases or decreases beyond a certain level, such as an optionally pre-defined threshold, the system can switch registration to the real-time data of the marker or calibration or registration phantom or device. The reverse is also possible.

[0385] The surgeon or assistant can then place a sterile drape or surgical cover over the site, but preferably not over the marker or calibration or registration phantom or device. Registration can be maintained by the real-time data of the marker or calibration or registration phantom or device attached to the patient, such as near or inside the craniotomy site.

[0386] The patient real-time data can then be image processed and / or pattern recognized in real-time through the optical head-mounted display. For example, using an in-built image capture device to capture real-time data of the patient or image and / or video capture system attached to, integrated with, or coupled to the optical head-mounted display.

[0387] The virtual and real-time data features or patterns can then be matched. To successfully register the patient real-time data and the overlay of both, the matching can include movement and / or re-orientation and / or magnification and / or reduction of the virtual data. The virtual data and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0388] A combination of (a) and (b) (e.g., an automatic registration function with a manual adjustment option), such as, after the image processing software and / or pattern recognition software and / or matching software identifies a potential match or performs an initial match, the virtual image data can be moved by comparing the real-time image data, and then manual / surgeon-based adjustments can be made. Alternatively, manual / surgeon-based matching and registration can be performed first, and then fine-tuned through software or algorithms (image processing, pattern recognition, etc.) based on the matching and registration. The virtual data and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0389] In some embodiments of the present application, the methods described herein can be used to register patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, overlaid, and / or registered with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the patient virtual data. Matching, overlaying, and / or registering the patient real-time data and the patient virtual data after the surgical tissue alterations can use the same methods described in the foregoing or any other registration methods described in this specification or any other registration methods known in the art.

[0390] Registering patient virtual data and patient real-time data using anatomical landmarks

[0391] In some embodiments, the surgeon can identify a selected anatomical landmark in the patient virtual data, such as, for example, the patient's electronic preoperative plan, and the anatomical landmark in the patient real-time data. For example, the surgeon can identify the anatomical landmark by placing a cursor or marker on the electronic image of the patient virtual data, and clicking on the anatomical landmark once the cursor or marker is in the desired location. In the spine, such anatomical landmarks can be, for example, the posterior tip of a spinous process, a spinal lamina, the inferior articular facet on the patient's left side, the superior articular facet on the patient's left side, the inferior articular facet on the patient's right side, the superior articular facet on the patient's right side, the tip of a facet joint, a bone spur, an osteophyte, etc. In the hip, such landmarks can be the most anterior point of the acetabulum, such as, for example, osteophytes at the acetabular rim, within the acetabulum, adjacent to the acetabulum, at the femoral head, at the femoral neck or neck shaft junction, at the center of the femoral head for two-dimensional or three-dimensional images, at the most anterior point of the femoral head, the anterior superior iliac spine, the anterior inferior iliac spine, the pubic symphysis, the greater trochanter, the lesser trochanter, etc. In the knee, such landmarks can be the femoral condyle, the femoral notch, the intercondylar notch, the medial or lateral epicondyle, the femoral shaft, the epicondylar shaft, the trochlear shaft, the mechanical axis, the trochlear groove, the femoral osteophytes, the marginal femoral osteophytes, the central femoral osteophytes, the patellar dome, the superior or inferior patellar or femoral articular surface, the medial, lateral, anterior, posterior tibial articular surface, the tibial osteophytes, the marginal tibial osteophytes, the central tibial osteophytes. The surgeon can then identify the same landmarks in the patient. For example, while viewing through the optical head-mounted display, the surgeon can point to the corresponding anatomical landmark in the real-time data with a finger or with a pointing device. The tip of the pointer or the tip of the finger can optionally include a tracker that locates the tip of the pointer or the finger in space. This location can also be accomplished visually using image capture, such as, for example, through the optical head-mounted display to more accurately determine the distance and position of the pointer or finger from the optical head-mounted display in a stereoscopic manner. An image and / or video capture system can also be connected to, integrated with, or coupled to the optical head-mounted display. The virtual and real-time data can include osteophytes or bone spurs or other bony anatomy or deformities.

[0392] Representative anatomical landmarks that can be used to register the patient's virtual and real-time data can include, but are not limited to:

[0393] In the spine:

[0394] - a portion or the entire spinous process

[0395] - a portion or the entire spinal lamina

[0396] - a portion or the entire spinal facet

[0397] - a portion or the entire facet joint

[0398] - a portion or the entire transverse process

[0399] - a portion or the entire pedicle

[0400] - a portion or the entire vertebral body

[0401] - a portion or the entire intervertebral disc

[0402] - a portion or the entire spinal osteophyte

[0403] - a portion or the entire spinal bone spur

[0404] - a portion or the entire spinal fracture

[0405] - a portion or the entire vertebral body fracture

[0406] - any combination of the foregoing

[0407] Hip:

[0408] - a portion or the entire acetabulum

[0409] - a portion or the entire rim of the acetabulum

[0410] - multiple portions of the rim of the acetabulum

[0411] - a portion of the iliac wall

[0412] - a portion of the pubic bone

[0413] - a portion of the ischium

[0414] - the anterior superior iliac spine

[0415] - the anterior inferior iliac spine

[0416] - the pubic symphysis

[0417] - a portion or the entire greater trochanter

[0418] - a portion or the entire lesser trochanter

[0419] - a portion or the entire femoral shaft

[0420] - a portion or the entire femoral neck

[0421] - a portion or the entire femoral head

[0422] - the fovea capitis

[0423] - the transverse acetabular ligament

[0424] - the posterior tubercle of the thalamus

[0425] - a ligament

[0426] - the upper lip

[0427] - one or more osteophytes, femur and / or acetabulum

[0428] - any combination of the foregoing

[0429] knee:

[0430] - a portion or the entirety of the medial condyle

[0431] - a portion or the entirety of the lateral condyle of the femur

[0432] - a portion or the entirety of the femoral notch

[0433] - a portion or the entirety of the trochlea

[0434] - a portion of the anterior cortex of the femur

[0435] - a portion of the anterior cortex of the femur with adjacent portions of the trochlea and osteophytes, when present

[0436] - a portion of the anterior cortex of the femur with adjacent portions of the trochlea and osteophytes, when present

[0437] - one or more osteophytes, femur and / or tibia

[0438] - one or more spurs, femur and / or tibia

[0439] - the supracondylar ridge

[0440] - a portion or the entirety of the medial tibial plateau

[0441] - a portion or the entirety of the lateral tibial plateau

[0442] - a portion or the entirety of the central tibial spine

[0443] - a portion or the entirety of the lateral tibial spine

[0444] - a portion of the anterior cortex of the tibia

[0445] - a portion of the anterior cortex of the tibia and a portion of the tibial plateau, medial or lateral or both

[0446] - a portion of the anterior cortex of the tibia and a portion of the tibial plateau, medial or lateral or both and osteophytes, when present

[0447] - a portion or the entirety of the patella

[0448] - the medial edge of the patella

[0449] - the lateral edge of the patella

[0450] - the superior pole of the patella

[0451] - the inferior pole of the patella

[0452] - osteophytes of the patella

[0453] - the anterior cruciate ligament

[0454] - posterior cruciate ligament

[0455] - medial collateral ligament

[0456] - lateral collateral ligament

[0457] - a portion or the entire medial meniscus

[0458] - a portion or the entire lateral meniscus

[0459] - any combination of the foregoing

[0460] Shoulder:

[0461] - a portion or the entire glenoid

[0462] - a portion or the entire coracoid process

[0463] - a portion or the entire acromion

[0464] - a portion of the clavicle

[0465] - a portion or the entire humeral head

[0466] - a portion or the entire humeral neck

[0467] - a portion of the humeral shaft

[0468] - one or more humeral osteophytes

[0469] - one or more glenoid osteophytes

[0470] - a portion or the entire glenoid lip

[0471] - a portion or the entire shoulder ligament, such as the coracoclavicular ligament, superior, middle, or inferior glenohumeral ligament

[0472] - a portion of the scapula

[0473] - any combination of the foregoing

[0474] Skull and brain:

[0475] - a portion of the cranium

[0476] - a portion of the occipital bone

[0477] - a portion of the temporal bone

[0478] - a portion of the occipital bone

[0479] - a portion of the parietal bone

[0480] - a portion of the frontal bone

[0481] - a portion of the facial skeleton

[0482] - a portion of the facial structure

[0483] - a portion or the entire bone structure inside the skull

[0484] - a portion or the entire selected gyrus

[0485] - a portion or the entire selected sulcus

[0486] - a portion of the sinus

[0487] - a portion of the venous sinus

[0488] - a portion of the vessel

[0489] - a portion of the ear

[0490] - a portion of the external ear canal

[0491] Organ:

[0492] - a portion of the organ, such as the upper or lower pole of the kidney

[0493] - the edge or margin of the liver, spleen, lung

[0494] - a portion of the liver lobe

[0495] - a portion of the vessel

[0496] - a portion of the fissure, such as in the liver or spleen

[0497] - a portion of the uterus

[0498] One skilled in the art can identify other anatomical landmarks of hard tissue, soft tissue, and / or organs including the brain that can be used to register virtual data (including optional virtual surgical plans) and real-time data of the patient and optical head-mounted display on a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0499] In some embodiments of the present application, the optical head-mounted display can display an arbitrary virtual plane on the surgical field. The arbitrary virtual plane can be moved using a virtual interface or other interface. For example, the arbitrary virtual plane can include a "touch area" in which gesture recognition software, such as that provided by Microsoft with Microsoft HoloLens, has an integrated virtual "drag function" for the hologram that can be used to move the arbitrary virtual plane. For example, one or more cameras integrated with or attached to the optical head-mounted display can capture movement of the surgeon's finger relative to the touch area; using gesture tracking software, the virtual plane can then be moved by pushing the finger in the desired direction against the touch area.

[0500] The optical head-mounted display can initially display the arbitrary virtual plane at any location, such as projected onto the surgical field or outside of the surgical field, such as the hip joint, the knee joint, the shoulder joint, the ankle joint, or the spine, among others. The optical head-mounted display can optionally display the arbitrary virtual plane at a defined angle, such as orthogonal or parallel to a fixed structure in the operating room, the arbitrary virtual plane can be identified using one or more cameras integrated to the optical head-mounted display, an image capture or video capture system, and spatial recognition software such as Microsoft HoloLens provided by Microsoft, or can be identified using one or more additional optical markers or navigation markers, including infrared or radio frequency markers. For example, one or more optical markers can be attached to an extension of the surgical table. The optical head-mounted display can detect these one or more optical markers and determine their coordinates, and from that determine the horizontal plane of the surgical table. The virtual plane can then be displayed perpendicular or at another angle relative to the operating room table.

[0501] For example, in a hip arthroplasty, the optical head-mounted display can display a virtual plane on the surgical site. The virtual plane can be perpendicular to the surgical table. Using virtual interfaces, such as contact areas and gesture tracking on the virtual surgical plane, the optical head-mounted display can detect how the surgeon moves the virtual plane. Optionally, the virtual plane can maintain its perpendicular (or other desired angle) orientation relative to the surgical table as the surgeon moves and / or reorients the plane; a perpendicular orientation can be desired when the surgeon intends to make a perpendicular femoral neck cut. A different angle can be desired when the surgeon intends to cut the femoral neck with another orientation.

[0502] The surgeon can then move the virtual plane to a desired location, orientation, and / or alignment using the contact areas or other virtual interfaces. The movement of the virtual plane can include translation and rotation in any desired direction using any desired angle or vector, or a combination thereof. The surgeon can move the virtual plane to intersect a selected anatomical landmark or to intersect a selected anatomical or biomechanical axis. The surgeon can move the virtual plane to be tangential to a selected anatomical landmark or to a selected anatomical or biomechanical axis.

[0503] For example, in a hip arthroplasty, the surgeon can move the virtual plane to be tangential to the superior-most surface of the greater trochanter and the superior-most surface of the lesser trochanter. Figure 4A An example illustrates a virtual plane 70 in which the primary surgeon has moved and aligned to be tangential to the superior-most surface of the greater trochanter 71 and the superior-most surface of the lesser trochanter 72. Figure 4BExamples of the same virtual plane 70, where the primary surgeon has moved and aligned to be tangent to the superior-most of the greater trochanter 71 and the superior-most of the lesser trochanter 72, now the view is from the optical head mounted display of a second surgeon or surgical assistant, such as on the other side of the operating table (OR table).

[0504] Alternatively, the surgeon can point to and identify the gingival groove point, and as additional references, the lowest point between the greater trochanter and the femoral neck, such as using a pointer with additional optical markers or additional navigation markers, or using an image or video capture system integrated into the optical head mounted display, and gesture recognition software such as provided by Microsoft in Microsoft HoloLens to detect his fingers, or using a finger with additional optical markers or navigation markers. The line connecting the superior-most of the greater trochanter and the superior-most of the lesser trochanter can then be determined on a preoperative or intraoperative AP radiograph of the hip joint; alternatively, the gingival groove point can also be detected on the AP radiograph. The AP radiograph can include templates for the surgeon to select and size, such as femoral and acetabular component assemblies, and liners and / or femoral heads. The radiographic templates can include femoral neck cut indications. The angle between the line connecting the superior-most of the greater trochanter and the superior-most of the lesser trochanter and the femoral neck cut indication can be determined. Figure 4C Examples of the same virtual plane 70, where the primary surgeon has moved and aligned to be tangent to the superior-most of the greater trochanter 71 and the superior-most of the lesser trochanter 72, now the view is from the optical head mounted display of a second surgeon or surgical assistant, such as on the other side of the operating table (OR table).

[0505] If the radiographic magnification is a point of interest for defining a second virtual plane, such as a virtual cut plane based on the first virtual plane, such as a plane that is tangent to or intersects one or more anatomical landmarks or one or more anatomical or biomechanical axes, the angle can optionally be combined with a measured distance and a correction magnification applied. For example, a distance between one or more anatomical landmarks that are tangent to or intersect a virtual plane can be measured in the patient real-time data and can be measured on the radiograph. If the radiographic distance is greater or less than the distance in the live patient, a magnification correction can be applied, such as a distance between the first virtual plane, such as a plane that is tangent to or intersects one or more anatomical landmarks or one or more anatomical or biomechanical axes, and a second virtual plane, such as a virtual cut plane, can be corrected based on a radiographic magnification factor.

[0506] In another example, an arbitrary virtual plane can be projected or displayed outside or on top of the surgical field in a knee replacement. Optionally, the arbitrary virtual plane is at least initially perpendicular to the surgical table or at an angle to the surgical table. If the mechanical axis of the leg has been determined in a previous step using intraoperative measurements, such as optical markers affixed to the thigh and one or more optical markers affixed to the ankle joint, for determining the center of rotation of the hip joint and the center of the ankle joint using an image capture or video capture system integrated to, connected to, or separate from the optical head-mounted display, the virtual plane can be configured to be perpendicular to the mechanical axis of the leg. Using a virtual interface such as a contact area, and an image or video capture system integrated or attached to the optical head-mounted display and optionally gesture tracking software, the surgeon can move and / or realign the virtual plane, such as to intersect with the medial and lateral joint space of the exposed knee joint, such as to increase the degree of flexion or at 5, 10, 15, 20, 30, 45 degrees of flexion or greater. Figure 5 An example of an arbitrary virtual plane 74 in a knee that intersects with the extended medial 76 and lateral 75 joint space.

[0507] One or more additional virtual planes can then be optionally projected, such as perpendicular to the surgical table or at another angle, or using a desired femoral component bending angle or a desired tibial slope. The surgeon can optionally move these one or more virtual planes to coincide with one or more anatomical axes, such as an anatomical femoral shaft axis or an anatomical tibial shaft axis in the live patient. The surgeon can also move the virtual plane to place and orient in the center of the femoral notch, the virtual plane parallel to the notch wall and extending centrally between the medial and lateral femoral shaft cortices, as a means to estimate the anatomical femoral shaft axis.

[0508] Once the anatomical femoral and / or tibial axes are determined or estimated, a virtual surgical plan is designed that includes femoral and tibial resections to achieve a desired femoral mechanical axis correction, such as, for example, a correction from the patient's mechanical axis alignment, such as, for example, 5, 10, 15 degrees of varus or valgus, to normal mechanical axis alignment or any desired residual, such as, for example, can create or result in a congenital varus or valgus. Implant sizes and desired polyethylene thicknesses can be factored into the virtual surgical plan. An optical head-mounted display can then project virtual surgical cutting planes based on the virtual surgical plan and / or intraoperative measurements, desired varus and / or valgus correction, desired slope, and / or desired implant rotation. The surgeon can then align a physical saw blade with the projected or displayed virtual saw blade. Alternatively, the optical head-mounted display can display a virtual guide or virtual cutting block that has at least one or more dimensions that are the same as a physical guide or physical cutting block, and the surgeon can align a physical cutting guide or cutting block with the virtual guide or cutting block, in which a saw blade insert is guided by the physical guide or cutting block and one or more blocks are executed.

[0509] The aforementioned concepts of projecting arbitrary virtual planes and aligning them with one or more anatomical landmarks, anatomical axes, or biomechanical or mechanical axes can be applied to any joint as well as the spine. Similarly, these concepts can be applied to brain surgery, in which one or more virtual planes can be projected or displayed and moved to be tangent to or intersect one or more anatomical landmarks, such as, for example, gyri, the pons, the cerebellum, etc. Similarly, these concepts can be applied to organ surgery, in which one or more virtual planes can be projected or displayed and moved to be tangent to or intersect one or more anatomical landmarks, such as, for example, the porta hepatis and anterior liver margin, one or more heart valves, etc.

[0510] Any arbitrary two-dimensional and / or three-dimensional virtual shape or contour or surface, such as, for example, a cube, a cuboid, a prism, a cone, a cylinder, a sphere, an ellipsoid-derived three-dimensional shape, an irregular shape, etc., can be virtually projected or displayed and automatically or using a virtual or other user interface moved, oriented, or aligned to coincide with, be tangent to, intersect, partially or completely overlap with, patient anatomy, pathology, anatomical axes, biomechanical axes including mechanical axes, anatomical planes, three-dimensional shapes, two-dimensional and / or three-dimensional geometric shapes, three-dimensional surfaces, and / or three-dimensional volumes of any internal organs, patient soft or hard tissue; after being moved, oriented, or aligned, coordinate information of the two-dimensional and / or three-dimensional virtual shape or contour or surface can be measured. Optionally, based on the coordinate information, additional intraoperative measurements are performed, and / or optionally the information is used to develop or modify a virtual surgical plan.

[0511] In some embodiments of the present application, the methods described herein, including the anatomical landmarks, can be used to register the patient virtual data and the patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or registered with the surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue features in the patient virtual data. The matching, superimposing and / or registering of the patient real-time data and the patient virtual data after the surgical tissue alteration can use the same methods described in the foregoing or any other registration methods described in this specification or any other registration methods known in the art. Alternatively, different anatomical landmarks can be used for the first registration and any subsequent registrations. Or, the same anatomical landmarks can be used for the first registration and any subsequent registrations.

[0512] Using a light source to reference live anatomical landmarks

[0513] The tracker or pointing device can also be a light source that can produce a red or green dot on the patient tissue produced by a laser that highlights the anatomical landmark for registration. The light source can be selected to have an intensity and / or color that will easily distinguish it from the live tissue of the patient.

[0514] The laser or other light source is optionally integrated into or attached to the optical head-mounted display, such as the laser or light source can be integrated into or connected to a bridge piece that connects the left and right eye pieces of the optical head-mounted display, such as over the nose region.

[0515] The image capture integrated into or attached to or coupled to the optical head-mounted display, such as, can be used to identify the location of the light on the patient tissue or anatomical landmark of the patient. Once the light is directed to the desired location on the patient real-time data, specifically, the live anatomical landmark of the patient, the registration can be performed by executing a registration command to register the patient real-time data and the virtual data, such as the live anatomical landmark of the laser or other light reflection and the corresponding virtual anatomical landmark of the patient. This process can be repeated for different anatomical landmarks, such as by directing the light source to the next live anatomical landmark of the patient, confirming the accurate location or pointing of the light, such as can be captured by the image capture device and software of the optical head-mounted display, the red or green laser dot reflected from the live patient anatomical landmark, and the next anatomical live anatomical landmark can be registered with the corresponding virtual anatomical landmark of the patient. The virtual data and real-time data can include bone spurs or bone nodules or other skeletal anatomy or deformities. In this way, the optical head-mounted display, the patient real-time data and the patient virtual data can be registered in a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0516] In some embodiments, more than one real-time and virtual anatomical landmark of the patient will be used, such as two, three, or more.

[0517] In some embodiments of the present application, ultrasound or radiofrequency transmitters can be used to precisely locate certain live anatomical landmarks. For example, an ultrasound transmitter or a radiofrequency transmitter can be incorporated into a point device, such as the tip of a pointer. When the tip contacts the desired live anatomical landmark, the transmitter can emit an ultrasound or radiofrequency signal that can be captured at a receiving station, optionally integrated into an optical head-mounted display. Optionally, multiple receiving stations can be used at spatially distinct locations, such as a means of improving the accuracy of real-time data registration. The virtual and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0518] In some embodiments of the present application, the size of the pointer has been previously scanned and registered with the optical head-mounted display. An image and / or video capture system connected to, integrated with, or coupled to the optical head-mounted display can identify the pointer in the real-time data and can identify the tip of the pointer. When the tip of the pointer touches a live anatomical landmark on the patient that corresponds to an anatomical landmark in the virtual data, the surgeon can click to indicate that the mutual referencing has been successfully accomplished. The two data points can then be optionally fused or superimposed in a common coordinate system. The virtual and real-time data and data points can include bone spurs or osteophytes or other skeletal anatomy or deformities, or can be generated from bone spurs or osteophytes or other skeletal anatomy or deformities. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0519] Anatomical landmarks can include unaltered surface shapes, such as skin, such as the tip of the nose, the distance between the eyes, the location of the ears, the shape of the ears, and other facial features.

[0520] Anatomical landmarks can also be bony anatomical landmarks, such as the medial or lateral malleolus, the tibial tubercle, the medial or lateral epicondyle, the trochlear fossa, the spinous process, and others. The virtual and real-time data and the virtual and live anatomical landmarks can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0521] Optionally, live anatomical surfaces can be used for registration purposes. In this embodiment, real-time anatomical surfaces can be derived during surgery using light scanning, infrared scanning, or ultrasound or ultrasonic scanning technology. The patient live surfaces detected and generated in this manner can be matched or aligned with the patient virtual surfaces obtained preoperatively, such as using imaging studies such as X-ray imaging, ultrasound, computed tomography, or magnetic resonance imaging, or any other technique known in the art. The virtual and real-time data and anatomical surfaces can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0522] In some embodiments of the present application, the method described herein can be used to register the patient virtual data and the patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue feature in the live patient can be matched, superimposed and / or registered with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue feature in the patient virtual data. The matching, superimposing and / or registering of the patient real-time data and the patient virtual data after the surgical tissue alteration can use the same methods described in the foregoing or any other registration method described in this specification or any other registration method known in the art.

[0523] The patient virtual data and the patient real-time data are registered using implantable or attachable markers or, calibration or registration phantoms or devices including optical markers.

[0524] In some embodiments of the present application, the surgeon optionally uses implantable or attachable markers to register the patient real-time data and the virtual data. This embodiment is useful, for example, if the surgical range is very extensive and results in tissue in the surgical site being removed, such as in brain surgery to remove brain tumors, liver surgery such as to remove liver tumors, joint replacement surgery and many other types of surgery. The virtual data and the real-time data can include bone spurs or osteophytes or other bone anatomy or deformities.

[0525] The implantable markers, the attachable markers, the skin markers, the soft tissue markers, the calibration or registration phantoms or devices, and the image capture markers and like terms used throughout this application, include optical markers such as optical markers having different geometric shapes or patterns, optical markers with two-dimensional codes (QR codes), bar codes, alphanumeric codes. The implantable or attachable markers or the calibration or registration phantoms or devices can be implanted or attached prior to the actual surgery and can be included in preoperative, intraoperative and / or postoperative images. The implantable or attachable markers or the calibration or registration phantoms or devices can be implanted or attached to bone spurs or osteophytes or other bone anatomy or deformities.

[0526] If there are implanted or attached markers or calibration or registration models or devices in the virtual image data, the surgeon can identify the implanted or attached markers or calibration or registration models or devices at any time after the cut because the surgeon can access the target tissue and the implanted markers placed next to or within the target tissue. Such implanted or attached markers or calibration or registration phantoms or devices can include radiation beets or metallic beets, such as, also used for stereoscopic imaging or registration.

[0527] Alternatively, implanted or attached markers or calibration or registration phantoms or devices can be placed during the surgery, such as, through optical head-mounted display image capture, or through attached optical head-mounted display, integrated or coupled optical head-mounted display image capture, the position of the implanted or attached markers or calibration or registration models or devices can be determined. The position of the implanted or attached markers or calibration or registration models or devices in the patient entity data can then be matched to the position of the anatomy of the implanted or attached markers or calibration or registration models or devices in the patient virtual data. Such as, the anatomy in the virtual and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities. In some embodiments, a pointer or indicating device can optionally be placed on the implanted or attached markers or calibration or registration models or devices, then, image capture through the optical head-mounted display and other image capture devices connected to and in connection or coupling with the optical head-mounted display and registration of the pointer tip. In this way, the optical head-mounted display, the implanted or attached markers or calibration or registration phantoms or devices including optical markers, and the anatomy, pathological structures, instruments, implant components and any other objects to which one or more implanted or attached markers or calibration or registration phantoms or devices including optical markers are attached through the use of the implanted or attached markers or calibration or registration phantoms or devices including optical markers, and the patient virtual data can be registered in a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0528] Implanted or attached markers or calibration or registration phantoms or devices can include rigid or fixed registration markers. Such rigid or fixed registration markers can be used to maintain registration when the surgical field is changed. The rigid or fixed registration markers can be screws or pins. The virtual data and real-time data can include bone spurs or osteophytes or other bone anatomy or deformities. The rigid or fixed registration markers can be attached to the bone spurs or osteophytes or other bone anatomy or deformities. In some embodiments, a medical device or component thereof being implanted has been temporarily or permanently attached to the patient tissue, and during the procedure, such as at the completion of a subsequent step of the procedure, a bone spur or osteophyte or bone anatomy or deformity, or anatomical site or surgical site can be used as an implanted or attached marker or calibration or registration phantom or device. These subsequent steps can include implantation of additional components of the medical device. For example, in a spinal fusion procedure, a first pedicle screw can be implanted. The real-time data and virtual data of the first pedicle screw can be registered. By using the registered first pedicle screw to maintain registration between the real-time data and virtual data, subsequent pedicle screws or other components, including their intended path, location, position or orientation, can be virtually displayed in the optical head-mounted display. Any other rigid or fixed registration markers or implanted devices can be used in this manner for different types of surgical procedures on the human body.

[0529] One or more implanted or attached markers or calibration or registration phantoms or devices are attached to bone, cartilage, soft tissue, organs or pathological tissue, such as bone spurs or osteophytes or other bone anatomy or deformities, etc.

[0530] One or more implanted or attached markers or calibration or registration phantoms or devices can optionally include optical markers, retro-reflective markers, infrared markers or radio frequency markers or any other marker devices described in the art.

[0531] Optical markers are markers that reflect light in the visible spectrum, i.e., the portion of the electromagnetic spectrum that is visible to the human eye, with wavelengths typically between about 390 and 700 nm or a frequency band of about 430-770 THz. Optical markers can also reflect a mixed light that includes different wavelengths within the visible spectrum. The light reflected by the optical markers can be detected by image and / or video capture systems integrated into, attached to, or separate from the optical head-mounted display. By using image and / or video capture systems that can be correlated image processing, and optionally pattern recognition software and systems, integrated into, attached to, or separate from the optical head-mounted display, the location, position, orientation, collimation, and / or direction of movement of the optical markers can be detected. Optical markers can include markers with selected geometric patterns and / or geometric shapes that can be recognized by image and / or video capture systems, such as those integrated into, attached to, or separate from the optical head-mounted display, such as using image processing and / or pattern recognition techniques. Optical markers can include markers composed of selected alphanumeric or pattern and / or numeric or pattern and / or alphanumeric or pattern or other codes, such as barcodes or two-dimensional codes that can be recognized by image and / or video capture systems, such as those integrated into, attached to, or separate from the optical head-mounted display, such as using image processing and / or pattern recognition techniques. Two-dimensional codes or quick response codes include any current or next generation matrix code including barcodes. Barcodes and two-dimensional codes are machine-readable optical labels on which information is stored, such as with patient identifiers, patient conditions, surgical types, patient information, surgical site information, surgical spinal segment information if a spinal surgery is performed, patient surgical side information, one or more surgical instrument information, one or more trial implant information, one or more implant component information, including implant type and / or implant size used, polyethylene type, acetabular liner type (e.g., standard, lip, offset, other), if a hip arthroplasty is intended. Two-dimensional codes can store data using different standardized encoding modes, such as numeric, alphanumeric, byte / binary, and / or Kanji, among others. Other encoding modes can be used. Any current and / or future versions of two-dimensional codes can be used. Monochrome or multicolor encoded two-dimensional codes can be used. Other graphical markers, such as those supported by the Vuforia (PTC, Needham, Mass) augmented reality platform can also be used.

[0532] Barcodes, two-dimensional codes, or other graphical markers can be optical markers. Barcodes, two-dimensional codes, or other graphical markers can be part of or integrated into optical markers. The same two-dimensional codes or barcodes or other graphical markers

[0533] - information related to the patient and / or the surgical site, such as, patient identifier, age, gender, BMI, medical history, risk factors, allergies, site and side (left, right), spinal segment to be operated

[0534] - information related to inventory management, including surgical instruments and / or implants or implant components, such as, left and right components, selected component size (matching to the virtual surgical plan and / or template and / or size)

[0535] and can be used to obtain information about the surgical site, the surgical altered tissue, the location, position, orientation, collimation and / or movement direction (if applicable) of the one or more surgical instruments and the one or more trial implants and / or implant components.

[0536] The geometric patterns, geometric shapes, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes included in one or more of the optical markers or a portion thereof can be predefined and, optionally, stored in a database accessible by the image and / or video capturing system and the associated image processing software and pattern recognition software. The geometric patterns, geometric shapes, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes included in one or more of the optical markers or a portion thereof can be two-dimensional images, while some are three-dimensional images. For example, one or more planar or two-dimensional patterns can be used in selected embodiments. Alternatively, selected three-dimensional geometric shapes can be used, such as, a cube, a cuboid, a prism, a cone, a cylinder, a sphere. Any three-dimensional shape can be used, including irregular shapes and / or asymmetric shapes. The three-dimensional geometric shapes can include two-dimensional geometric patterns and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes on one or more surfaces. For example, if a cuboid is used, the same or different geometric patterns and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes can be included, affixed to or integrated into one or more surfaces, such as two opposite surfaces or two adjacent surfaces, such as, vertically oriented. The two-dimensional geometric patterns and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes included in, affixed to or integrated into one or more surfaces of the three-dimensional geometric shape can be used to determine the orientation of the surface of the geometric shape including the optical marker, as well as the orientation and / or alignment of the surface, and the geometric shape, such as, in relation to the surgical site and in relation to the surgical alteration, in particular the cut bone surface or the reamed bone surface, the surgical instruments and / or the one or more implant components including the trial implant.

[0537] The geometric patterns and / or geometric shapes, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes can be colored or black and white. The geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes can include colored and black and white portions, only colored portions and only black and white portions. The geometric shapes can include include colored and black and white surfaces, only black and white surfaces, and only colored surfaces. Different colors and different colored codes can be used for different surfaces of the geometric shape portions of the optical markers. Different colors and different colored codes can be used for different geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes. Different colors and different colored codes can be used for different optical markers. Different colors, such as, for example, red, blue, green, orange, cyan, etc., can be used for different geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes. Different colors, such as, for example, red, blue, green, orange, yellow, pink, cyan, etc., can be used for different optical markers. Different optical markers can optionally be associated with different surgical steps and / or different surgical instruments and / or different implant components; the use of a particular marker can be recognized by integration into the optical head-mounted display, attachment to the optical head-mounted display or a separate image and / or video capture system from the optical head-mounted display, optionally including a pattern database, such as, for example, an image database associated with a particular surgical step and / or surgical instrument, using standard image processing and / or pattern recognition software. As the image and / or video capture system recognizes a particular optical marker in the field of view, such as, for example, based on a particular geometric pattern and / or geometric shape and / or letter, number, alphanumeric and other code or pattern including a barcode. It can then selectively display the corresponding surgical step and / or surgical instrument and / or implant component associated with that optical marker.

[0538] Two-dimensional geometric patterns, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes or combinations thereof, optionally color and / or black and white coded, included, added or integrated into the surface of one or more three-dimensional geometric shapes, can be used to determine the orientation and / or alignment of the selected surface of the geometric shape and / or the orientation and / or alignment of the geometric shape and / or optical markers, such as, for example, in relation to anatomical landmarks, surgical sites, surgical alterations, including, in particular, cut bone surfaces or reamed bone surfaces, surgical instruments and / or one or more implant components including trial implants, etc. One or more two-dimensional geometric patterns, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes or combinations thereof, optionally color and / or black and white coded, included, added or integrated into optical markers, can be used to determine the orientation and / or alignment of the optical markers fixed to or integrated into anatomical landmarks, surgical sites, surgical alterations, such as, for example, cut bone surfaces or reamed bone surfaces, surgical instruments and / or one or more implant components including trial implants, etc. Optical markers can be fixed to anatomical landmarks, surgical sites, surgical alterations (such as, for example, cut bone surfaces or reamed bone surfaces, or patient drill holes and corresponding anatomical landmarks, surgical sites or surgical alterations, which can be identified in the patient’s virtual data), enabling registration of the patient’s virtual data and real-time data in the same coordinate system. If multiple optical headsets are used during surgery, optical markers can also be attached to the optical headsets, including multiple optical headsets. Optionally, optical markers (such as, for example, using QR codes) can be used to distinguish between a first, third, fourth and / or more optical headsets. Optionally, one or more optical markers can be attached to the surgical table and they can be registered in a coordinate system, such as, for example, in the same coordinate system, one or more optical headsets, the patient and parts of the surgical site can be registered. One or more optical markers can be optionally attached to other structures in the surgical room, including fixed structures (such as, for example, walls) and movable structures (such as, for example, surgical room lights), which can be registered in a coordinate system, such as, for example, in the same coordinate system, one or more optical headsets, the patient and parts of the surgical site can be registered. In this example, optical markers can also be mounted to fixed structures on holding arms or extenders, optionally mobile, such as, for example, having known dimensions, orientation, length and angle.

[0539] Optical markers attached to fixed structures such as operating room walls (OR walls) can be used to enhance the accuracy of room identification and spatial mapping, especially when the coordinates and / or angles and / or distances between different optical markers are known. Optical markers attached to fixed structures such as operating room walls can also be used to enhance the determination of the position and pose of one or more optical head-mounted displays and changes in position or pose or coordinates and changes in coordinates, which can help increase the accuracy of virtual data displays and the accuracy of superimposition on corresponding real-time data.

[0540] Optical markers attached to movable structures can be used to track their position in the operating room. Optical markers attached to operating room lights can be used to estimate the direction of the light and the direction and / or trajectory of the shadow in the operating room or room. If the direction and / or trajectory of the shadow in the operating room or room is known, a virtual shadow with the same or similar direction or trajectory can be applied to the virtual data display of the optical head-mounted display.

[0541] For example, one or more optical markers comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof can be attached to the medial femoral condyle, for example, using pins or screws or adhesives. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can be used to monitor the position, and / or orientation, and / or alignment, and / or direction of movement of the optical markers relative to the image and / or video capture system; as the distal femur moves, the image and / or video capture system can detect the markers (e.g., based on their preprogrammed geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes) and can monitor and optionally register the motion. If, in the same example, a second optical marker comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof is attached to the lateral femoral condyle, the image and / or video capture system can also monitor and optionally register the position, and / or orientation, and / or alignment, and / or direction of movement of the second optical marker relative to the image and / or video capture system; by monitoring the position and / or orientation and / or alignment and / or direction of movement of the first optical marker on the medial femoral condyle and the position and / or orientation and / or alignment and / or direction of movement of the second optical marker on the lateral femoral condyle, the image and / or video capture system and associated image processing and pattern recognition software can also monitor and optionally register the motion of the femoral condylar axis (e.g., during flexion and extension of the knee). One or more optical markers comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof can be attached to the proximal tibia (e.g., the proximal tibia), the tibial anterior edge, the medial and / or lateral tibial spine, the lowest point of the medial tibial plateau, and / or the highest point of the lateral tibial plateau. For example, in the same example. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can be used to monitor the position, and / or orientation, and / or alignment, and / or direction of movement of the optical markers attached to the tibia relative to the image and / or video capture system and to one or more femoral optical markers, thereby monitoring and optionally registering tibiofemoral motion, for example, during surgery. One or more optical markers comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof can be attached to the patella (e.g., the uppermost, lowermost, lateralmost, and / or medialmost in the same example).An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, and / or orientation, and / or movement direction of optical markers attached to the patella relative to the image and / or video capture system and to one or more femoral optical markers, and thereby monitor and optionally register patellofemoral motion, such as during surgery. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, and / or orientation, and / or movement direction of optical markers attached to the patella relative to one or more tibial optical markers, and thereby monitor and optionally register tibial related patellar motion, such as during tibial adduction or abduction.

[0542] In some embodiments of the application, optical markers of one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns including bar codes and two-dimensional codes or combinations thereof can be assigned to virtual surgical steps. The markers can include written text defining a surgical step or corresponding to a surgical step, which can be the immediately preceding surgical step or the next surgical step, such as in a virtual surgical plan. In some embodiments, the text can be a number (such as a digit corresponding to a particular surgical step), such as 1 - for distal femoral cut, 2 - for femoral anterior cut, 3 - for posterior femoral cut, 4 - for first chamfer cut, 5 - for second chamfer cut. The number can be recognized by the image and / or video capture system, and then the image and / or video capture system can display a virtual view of the corresponding surgical step, such as a cutting plane for a distal femoral cut or a virtual outline of a corresponding physical distal femoral cutting block for a distal femoral cut of 1. A combination of numbers and text can be used, and the image and / or video capture system and related software and optional pattern recognition software and systems can recognize the number and text and trigger a command to display the corresponding virtual view of the corresponding virtual surgical step, such as 1F - distal femoral cut, 2F - femoral anterior cut, 1T - proximal tibial cut, 2T - tibial keel, etc.

[0543] In another example, an optical marker having one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes or combinations thereof can be assigned to the "distal femoral cutting" step in a virtual surgical plan for a total knee arthroplasty patient; the optical marker can include the text "distal femoral cut". For example, the surgeon can affix the marker to the cut bone surface of the distal femur or somewhere adjacent thereto. An image and / or video capture system integrated into, attached to or separate from the optical head-mounted display can detect the optical marker having one or more specific geometric patterns and / or specific geometric shapes assigned to the "distal femoral cut" indicating that the distal femoral cut has been completed; the image capture signal can then command the optical head-mounted display to display the next surgical step (e.g., the anterior cut plane or the outline of the anterior cut block or cutting guide) such as the femoral anterior cut in this example, when the surgeon is ready to perform the next cut.

[0544] In some embodiments, an optical marker using one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes or combinations thereof can be integrated into, included in or attached to a surgical instrument used for a surgical step in a virtual surgical plan. For example, the optical marker can be included in, integrated into or attached to a surgical cutting block or cutting tool such as for a proximal tibial cut. Optionally, the marker can include written text defining the surgical step or corresponding to the surgical step such as in a virtual surgical plan. In the previous example, an optical marker having one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns including barcodes and QR codes or combinations thereof can be assigned to the "proximal tibial cut" step in a virtual surgical plan for a total knee arthroplasty patient; the optical marker can include the text "proximal tibial cut" readable by the surgeon and ensure that the correct marker is used for the next surgical step he or she is considering, in this example, the proximal tibial cut.

[0545] When the optical marker enters the field of view of the surgeon, the image and / or video capture system integrated into or connected to the optical head-mounted display on the surgeon's head can detect the optical marker and display the next virtual surgical step, such as the outline of a virtual proximal tibial cutting block corresponding to the physical proximal tibial cutting block, so that the surgeon can align or superimpose the physical surgical cutting block or instrument onto the outline of the virtual surgical cutting block or instrument. Alternatively, when the optical marker enters the field of view of the surgeon, the image and / or video capture system integrated into or connected to the optical head-mounted display on the surgeon's head can detect the optical marker and display the next virtual surgical step, such as a virtual cutting plane with a predetermined resection level, varus or valgus angle, and / or slope, so that the surgeon can align or superimpose the physical surgical cutting block and / or physical surgical saw to the virtual cutting plane. Once the surgical step is completed, such as the proximal tibial cut, and the surgeon removes the physical surgical instrument with the integrated, included, or attached optical marker from the surgical field and / or field of view of the image and / or video capture system, the image and / or video capture system can detect that the optical marker is no longer present in the field of view, and the software can generate a command to close the screen of the optical head-mounted display or the screen completing the virtual surgical step. Optionally, a command, optionally automatically generated, can be generated at this time to display the next surgical step, such as the tibial keel punch, such as to set the tibial rotation. Alternatively, the display of the optical head-mounted display device can display the next surgical step of the next surgical instrument while the next surgical instrument, with the corresponding optical marker for the next surgical step, enters the field of view, such as in the surgeon's hand.

[0546] In a similar example, an optical marker can be attached to a reamer for a hip arthroplasty. The image and / or video capture system integrated into or connected to the optical head-mounted display can detect the optical marker when it enters the field of view of the surgeon, triggering a command to display a virtual display of the reamer or the reamer with the intended alignment or orientation of the reaming step; when the optical marker of the surgical instrument exits the field of view of the surgeon, the image and / or video capture system can detect it triggering a command to stop the display of the reamer or the virtual display of the reamer, optionally switching to the next surgical step.

[0547] In some embodiments, one or more optical markers can be included, integrated or attached to an insert for a cutting block or guide plate. The insert can be configured to fit into one or more slots of a cutting block or guide plate or into a guide plate to guide a saw blade. For example, representative cutting blocks or guide plates are cutting blocks or guide plates for knee replacement, shoulder replacement, hip replacement and ankle replacement. For example, these cutting blocks or guide plates are used to remove bone at a joint surface to adapt the patient's bone to the bone-facing side of an implant or implant component. The insert can be designed to partially or substantially fill the entire slot or guide plate, for example, in the X and Y directions or in the X and Z directions or in the Y and Z directions, depending on the shape and / or design of the cutting block or guide plate. If the insert partially or substantially fills the slot or guide plate in the X and Y directions, the insert can be configured to extend beyond the slot or guide plate in the Z direction. If the insert partially or substantially fills the slot or guide plate in the X and Z directions, the insert can be configured to extend beyond the slot or guide plate in the Y direction. If the insert partially or substantially fills the slot or guide plate in the Y and Z directions, the insert can be configured to extend beyond the slot or guide plate in the X direction. Depending on the configuration of the cutting block or guide plate and the configuration of the associated slot or guide plate, any direction is possible, including oblique directions, orthogonal directions and non-orthogonal directions. For example, a beveled slot can be used for chamfer resection in total knee replacement or for oblique talus cutting in total ankle replacement.

[0548] The portion of the insert that extends beyond the slot or guide plate can include one or more integrated or attached optical markers. If more than one optical marker is used, the optical markers can be arranged at predetermined angles and positions, for example, 90 degrees or less than 90 degrees or more than 90 degrees. The insert can have similar dimensions to a representative saw blade for a cutting block or guide plate. The insert can indicate the location, position, direction, alignment and travel direction of a subsequent insert saw blade. The surgeon can place the insert within the slot or guide plate of the physical cutting block or guide plate and align the insert with the virtual cutting plane or virtual contour of the insert or cutting block or guide plate projected by the optical head-mounted display to the surgical site, for example, the distal femur in total knee arthroplasty or the proximal femur in total hip arthroplasty. Once the insert is substantially aligned and / or superimposed with the virtual cutting plane, virtual contour of the insert or cutting block or guide plate, the surgeon can secure the physical cutting block or guide plate to the bone, thereby securing the cutting block or guide plate to the bone at a fixed position where the virtual surgical plan, for example, the virtual cutting plane or virtual contour of the insert or cutting block or guide plate, is substantially aligned with the physical cutting plane and / or the physical insert or cutting block or guide plate. The surgeon can then insert a physical saw blade and perform a physical cut. The insert can be configured to have substantially similar shape to a physical saw blade to serve as a dummy saw blade.

[0549] Alternatively, the surgeon can place a physical saw blade within the slot of a physical cutting block or guide or within the guide and the surgeon can align the physical saw blade with the virtual cutting plane or virtual outline of the saw blade or cutting block or guide that is projected by the optical head-mounted display to the surgical site, such as the distal femur in a total knee arthroplasty or the proximal femur in a total hip arthroplasty. Once the physical saw blade is substantially aligned and / or superimposed with the virtual cutting plane, virtual outline of the saw blade or cutting block or guide, the surgeon can pin the physical cutting block or guide to the bone, thereby fixing the cutting block or guide to the bone in a fixed position where the virtual surgical plan, such as the virtual cutting plane or virtual outline of the saw blade or cutting block or guide, is substantially aligned with the physical cut plane and / or the physical saw blade or cutting block or guide. The surgeon can then advance the physical saw blade and make a physical cut.

[0550] The optical markers can be included in the cutting block or guide or insert, integrated into or attached to the insert or insert, such as a pseudo-saw blade. The optical markers can also be attached or affixed to the saw blade. The optical markers can include text or alphanumeric codes for the surgeon that designate a particular surgical step, such as a particular surgical step, IF - distal femoral cut, 2F - femoral anterior cut, IT - proximal tibial cut, 2T - tibial keel punch, etc. The optical markers can also include one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof. The one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof can be specific to a surgical step, such as corresponding to a letter or alphanumeric code that represents the surgeon's surgical step. When the optical markers enter the field of view, an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can detect the one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof; can use image processing and / or pattern recognition software to identify the specific geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns, such as command for displaying in the optical head-mounted display the corresponding virtual surgical step superimposed onto the surgical field with the surgeon's perspective consistent with the surgical field or target anatomy or bone cut. When the cutting block or guide, insert, such as a pseudo-saw blade, or physical saw blade with optical markers is removed, the image and / or video capture system can detect that the optical markers are no longer present in the field of view, triggering a command to turn off the optical head-mounted display display or complete the display of the surgical step or switch to the display of the next surgical step and the corresponding virtual display.

[0551] In some embodiments of the application, one or more optical markers (whose angle of selection, such as 90 degrees or less or more or parallel or on axis), can be included in the cutting block or guide, integrated into or attached to the cutting block or guide.

[0552] In some embodiments, one or more optical markers can be used in conjunction with spinal surgery, such as vertebroplasty, kyphoplasty, posterior spinal fusion, anterior spinal fusion, lateral spinal fusion, and / or disc replacement. For example, one or more optical markers can be included in, integrated with, or attached to needles, pins, awls, probes, ball handle probes, straight probes, curved probes, taps, ratchets, screwdrivers, rod templates, rod inserter, rod clamps, tube benders, plug starters, compressors, distractors, disruptors, occluders, anti-torque, quick joints, drivers, retractors, retraction frames, implant positioners, calipers, plate holders, plate benders, forceps, and the like. The foregoing list is merely exemplary and should not be construed as limiting the present application. One or more optical markers can be used to designate a patient left side and a patient right side, and / or designate a patient's spinal segment, such as using one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns detected using an image and / or video capture system integrated with, attached to, or separate from an optical head-mounted display, and identified using image processing and / or pattern recognition.

[0553] One or more optical markers can be used to determine a location, position, orientation, alignment, and / or orientation of a needle, pin, awl, probe, ball handle probe, straight probe, curved probe, screw, ratchet, screwdriver, rod template, rod inserter, rod clamp, tube bender, plug starter, compressor, distractor, disruptor, occluder, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, caliper, plate holder, plate bender, forceps, milling cutter, saw, reamer, broach, impactor, cutting or drilling block, and / or other surgical instrument and / or trial implant and / or implant assembly using an image and / or video capture system integrated with, attached to, or separate from an optical head-mounted display. For example, after an initial registration or subsequent registration of a patient, a surgical site, an optical head-mounted display, optionally, an image and / or video capture system integrated with, attached to, or separate from an optical head-mounted display, virtual data and / or real-time data of a patient, the image and / or video capture system can detect optical markers included in, integrated with, and / or attached to a surgical instrument. Since the location, position, alignment, and / or orientation of the optical markers on the surgical instrument is known, and the size (such as at least one of them) or geometry of the surgical instrument is known, the image and / or video capture system can track the location, position, orientation, alignment, and / or motion of the optical markers and the surgical instrument.

[0554] In another example, two or more optical markers can be integrated into or attached to the long axis of a needle, a pin, a spike, a probe, a ball handle probe, a straight probe, a curved probe, a screw, a ratchet, a screwdriver, a rod template, a rod inserter, a rod clamp, a tube bender, a plug initiator, a compressor, a retractor, a circuit breaker, a closure, an anti-torque, a quick joint, a driver, a retractor, a retraction frame, an implant locator, a caliper, a plate fixator, a plate bender, a forceps, a milling cutter, a saw, a reamer, a broach, a striker, a cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant assemblies in different, optionally defined positions, such as instruments or trial implants or implant assemblies in a knee replacement or hip replacement. The image and / or video capturing system can detect the two or more optical markers and can determine their respective positions. By locating the two or more optical markers captured and defined by the image and / or video capturing system, the long axis of the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the surgical instrument and / or the trial implant and / or the implant assembly can be determined; in addition to the long axis or instead of the long axis, other axes can be determined. With the positions of the optical markers on the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the other surgical instrument and / or the trial implant and / or the implant assembly known, the long axis or the other axes of the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the other surgical instrument and / or the trial implant and / or the implant assembly known, and the dimensions of the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the other surgical instrument and / or the trial implant and / or the implant assembly known, parts of the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the other surgical instrument and / or the trial implant and / or the implant assembly that are hidden by tissue, such as under the skin and / or inside the muscle, can be estimated and optionally displayed by the optical head-mounted display in addition to any other aspect of the virtual or intended path or projected path or virtual surgical plan. Instead of using two or more optical markers in the aforementioned embodiments, it is also possible to use an optical marker that is long enough or wide enough or deep enough to define one or more axes of the needle, the pin, the spike, the probe, the tap, the milling cutter, the saw, the reamer, the broach, the striker and / or the other surgical instrument and / or the trial implant and / or the implant assembly.

[0555] Optionally, when using two or more optical markers included in, integrated into, or attached to a surgical instrument, the optical markers can be arranged at the same angle, such as parallel or on the same axis, or at different angles, such as orthogonal angles or non-orthogonal angles. This can be particularly useful when the optical markers include one or more of geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof. By arranging the optical markers and any associated geometric shapes, geometric patterns, letters, numbers, alphanumeric, and other codes or patterns including barcodes and QR codes or combinations thereof in this manner, it is more accurate to determine the angular orientation of the surgical instrument. For example, a geometric shape, geometric pattern, letter, number, alphanumeric, and other code or pattern including barcodes and QR codes or combinations thereof on a first optical marker on a surgical instrument can be only partially visualized or not visible at all from certain viewing angles of an image and / or video capture system integrated into or connected to an optical head-mounted display due to the angular orientation; when a second optical marker is oriented at a different angle, position, and / or orientation on the same surgical instrument, the viewing angle from the image and / or video capture system integrated into or attached to the optical head-mounted display to the second optical marker can allow for full or more complete display of the geometric shape, geometric pattern, letter, number, alphanumeric, and other code or pattern including barcodes and QR codes or combinations thereof, thereby allowing for more accurate determination of the angular orientation of the second optical marker, and thus the surgical instrument. Additionally, the respective projections of the first optical marker and / or the second optical marker measured by the image and / or video capture system, when using two or more cameras, can optionally any parallax information pairing (such as one located near the left eye and the other located near the right eye) can be used to more accurately determine their relative positions and the position of the surgical instrument.

[0556] An image and / or video capture system integrated with, connected to, or detached from an optical head-mounted display detects an optical marker contained in, integrated into, or attached to a needle, a pin, a awl, a probe, a ball handle probe, a straight probe, a curved probe, a screw, a ratchet, a screwdriver, a rod template, a rod inserter, a rod clamp, a tube bender, a plug initiator, a compressor, a retractor, a disconnector, a closer, an anti-torque, a rapid joint, a driver, a retractor, a retraction frame, an implant locator, a caliper, a plate fixator, a plate bender, a forceps, a milling cutter, a saw, a reamer, a puller, a striker, a cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant assemblies, and when the optical marker enters the field of view of the surgeon, triggers a command to display a predetermined path or plane of the needle, the pin, the awl, the probe, the ball handle probe, the straight probe, the curved probe, the screw, the ratchet, the screwdriver, the rod template, the rod inserter, the rod clamp, the tube bender, the plug initiator, the compressor, the retractor, the disconnector, the closer, the anti-torque, the rapid joint, the driver, the retractor, the retraction frame, the implant locator, the caliper, the plate fixator, the plate bender, the forceps, the milling cutter, the saw, the reamer, the puller, the striker, the cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant assemblies, e.g., with the intended position, orientation, and / or alignment and / or direction of the intended surgical step; when the optical marker of the surgical instrument leaves the field of view of the surgeon, the image and / or video capture system can detect it, trigger a command to stop displaying the predetermined path or virtual display of the surgical instrument or other aspect of the virtual surgical plan, and optionally switch to the next surgical step and the corresponding virtual display. In spinal surgery and selected other surgeries, the next surgical step can involve the same side of the same spinal segment of the patient or the opposite side of the same spinal segment of the patient, where the virtual display for the next surgical step corresponding to the given level and side can be initiated by the OHMD display. The next surgical step can involve the same side of an adjacent or different spinal segment of the patient or the opposite side of an adjacent or different spinal segment of the patient, where the corresponding virtual display for the next surgical step corresponding to the given level and side can be initiated by the optical head-mounted display display.

[0557] The optical marker can include one or more two-dimensional codes. The two-dimensional code can be part of or can be embedded in the geometric pattern or figure included in the optical marker. The optical marker can be the two-dimensional code.

[0558] If the optical marker is attached to the surgical instrument, the attachment can occur at a defined position and / or location and / or alignment, such as, for example, at the tip of the surgical instrument. The attachment can include an opening with a stop, thereby defining the position and / or location and / or alignment of the optical marker on the surgical instrument. For example, the optical marker is provided with an opening with a stop that is large enough to accommodate the end of the surgeon's facing the staple or drill bit, such as, for example, a portion of the insertion of the spinous process or facet joint or pedicle. With this type of attachment and other attachments that secure the marker in a defined position, location and / or orientation on the surgical instrument, the image and / or video capture system can detect the optical marker and its position, location and / or orientation, from which the position, location and / or orientation of the surgical instrument, such as, for example, a needle, can be determined, including its tip or front portion in the patient's body due to its defined spatial relationship and the known geometry of the surgical instrument.

[0559] In some embodiments of the application, optical markers can be used to determine or identify the location, position, orientation, alignment, size, axis or axes, plane or planes of a surgical modification. For example, if a bone cut is made during a surgical procedure, one or more optical markers can be attached to the cut bone to determine its location, position, orientation, alignment, size, shape, geometry, axis or axes, plane or planes, or one or more thereof. For example, one, two or more optical markers can be placed at the periphery or edge of the cut bone or surgical modification; an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display can detect the position, location and / or orientation of the optical markers, and software can be used to, for example, analyze the position, location and / or orientation information of the optical markers to obtain information about the periphery and / or edge and / or shape of the cut bone or surgical modification. One, two or more optical markers can be placed near or on the cut bone or surgical modification; an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display can detect the position, location and / or orientation of the optical markers, and software can be used to, for example, analyze the location, position and / or orientation information of the optical markers to obtain information about the shape or geometry of the cut bone or surgical modification. If the bone cut is planar, one or more optical markers can be held or attached to a carrier or instrument (e.g., a plastic sheet) facing the surface of the planar bone, resting against, fixed to or attached to the cut bone surface; an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display can then be used to detect the one or more optical markers, and software can be used to, for example, analyze the position, location and / or orientation information of the one or more optical markers to obtain information about the position and / or location and / or orientation and / or alignment of the bone cut plane, including the relationship to other anatomical landmarks and / or other optical markers. The carrier or instrument for the optical markers can be transparent or translucent so that the surgeon can check or confirm that the carrier or instrument and attached optical markers are flush with the bone cut, e.g., the bone plane, before determining or confirming. Once the bone cut plane is determined or confirmed in this manner, the optical markers attached to the cut bone and / or the determined plane of the bone cut can be used to plan the next surgical modification, e.g., the next bone cut or surgical modification, such as an anterior femoral cut or a posterior femoral cut after a distal femoral cut in a knee arthroplasty, or a chamfer cut after an anterior-posterior femoral cut in a knee arthroplasty, or an onlay cut in a relative articular surface. By determining, confirming and / or referencing a previous surgical modification, e.g., a bone cut, in this manner, the accuracy of subsequent surgical steps can be improved, ultimately improving the overall accuracy of the surgical procedure.

[0560] In some embodiments of the application, one or more optical markers can be attached or affixed to the patient's thigh or distal femur. For example, one or more optical markers can be attached to the skin of the distal thigh, such as, for example, above the knee joint space. The attachment can be performed using, for example, an adhesive that attaches the one or more optical markers to the patient's skin. The one or more optical markers can optionally be sterile. The one or more optical markers can optionally be magnetic. In this example, a magnetic base can optionally be attached to the patient's skin, such as, for example, using an adhesive. A transparent, translucent or opaque surgical drape can then be placed over the magnetic base, and then the magnetic optical markers can be attached to the magnetic base that is attached to the patient's skin. Alternatively, the one or more optical markers can be rigidly attached to one or more bones, such as, for example, the distal femur and / or the proximal tibia, once a skin incision is made. The rigid attachment can be accomplished using pins or screws or other attachment mechanisms.

[0561] An image and / or video capture system integrated into the optical head mounted display, attached to the optical head mounted display or separate from the optical head mounted display can register the position and / or location and / or orientation and / or alignment of the one or more optical markers, such as, for example, when the leg is in a neutral position and / or an extended position and / or any other position, including an arbitrary position or a position selected by the surgeon and / or operator. The surgeon and / or operator can then move the leg and thigh to a plurality of different positions and / or orientations and / or alignments, and / or the surgeon and / or operator can move the leg and thigh in a circular or semi-circular manner. The image and / or video capture system integrated into the optical head mounted display, attached to the optical head mounted display or separate from the optical head mounted display can register the position and / or location and / or orientation and / or alignment of the plurality of optical markers and the plurality of different positions and / or orientations and / or alignments of the leg or thigh, and / or register the position and / or location and / or orientation and / or alignment of the plurality of optical markers during the different circular or semi-circular movements. The resulting information can be used to determine the center of rotation, which in this example can be the center of the hip joint.

[0562] In some embodiments, an ankle clip can be applied to the ankle of a patient's leg. The ankle clip can include one or more optical markers, including one or more two-dimensional codes. The ankle clip and / or the optical markers can be disposable. With an image and / or video capture system integrated into an optical head-mounted display, attached to an optical head-mounted display, or separate from an optical head-mounted display, the ankle clip and integrated or attached optical markers can be used to determine the position of the medial malleolus and the lateral malleolus, such as, for example, using the center or 1 / 3 or 2 / 3 distance point between the malleoli of the ankle joint. Alternatively, one or more optical markers can be applied to the medial malleolus and / or the lateral malleolus. In some embodiments of the present application, magnetic bases can be affixed to the medial malleolus and the lateral malleolus. The ankle can then be prepared and draped with aseptic technique, and one or more aseptic magnetic optical markers can be applied to the drape or surgical cover, affixing the in-draped or surgically covered ankle to the magnetic bases with the one or more optical markers. An image and / or video capture system integrated into an optical head-mounted display, attached to an optical head-mounted display, or separate from an optical head-mounted display can then be used to identify the optical markers on the medial malleolus and the lateral malleolus and the center, 1 / 3 or 2 / 3 distance point of the ankle joint.

[0563] With the center of the hip joint determined using one or more optical markers at the thigh or distal femur, and the center or 1 / 3 or 2 / 3 distance point of the ankle joint determined using the ankle clip and / or one or more optical markers, the system can obtain the mechanical axis of the patient, and a surgical intervention can be planned, such as, for example, cutting the respective femur and / or tibia and / or talus to correct varus or valgus deformity, and subsequently projecting using the optical head-mounted display.

[0564] In some embodiments of the present application, one or more optical markers can be attached or affixed to a patient's arm. For example, one or more optical markers can be attached to the skin of the upper arm, such as, for example, above the elbow. The attachment can be performed using, for example, an adhesive that attaches the one or more optical markers to the patient's skin. The one or more optical markers are optionally aseptic. The one or more optical markers are optionally magnetic. In this example, a magnetic base is optionally attached to the patient's skin, such as, for example, using an adhesive. A transparent, translucent, or opaque surgical drape can then be placed over the magnetic base, and magnetic optical markers can then be attached to the magnetic base that is attached to the patient's skin. Alternatively, once a skin incision is made, one or more optical markers can be rigidly attached to one or more bones, such as, for example, the proximal humerus. The rigid attachment can be accomplished using a pin or screw or other attachment mechanism.

[0565] An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of one or more optical markers, such as when the arms are in a neutral position and / or an extended position and / or an abducted position and / or any other position, including, any position or positions selected by the surgeon and / or operator. The surgeon and / or operator can then move the arms to a plurality of different positions and / or orientations and / or alignments, and / or the surgeon and / or operator can move the arms in a circular or semi-circular manner. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of a plurality of optical markers and a plurality of different positions and / or orientations and / or alignments of the arms, and / or register the position and / or location and / or orientation and / or alignment of a plurality of optical markers during different circular or semi-circular movements. The resulting information can be used to determine a center of rotation, which, in this example, can be the center of rotation of the shoulder joint.

[0566] In some embodiments of the present application, one or more optical markers can be attached to the table of an operating room (OR). If the optical markers are parallel to the surgical table, a single marker can be sufficient to determine the principal plane of the surgical table, such as, for example, the horizontal plane, which can be the plane on which the patient lies, such as, for example, in a supine, prone, lateral or inclined or other position known in the art. This can be assisted by the use of optical markers that include a surface or plane that is parallel or perpendicular or at a defined angle to the surgical table and are large enough to be detected by a camera, image or video capture system that is integrated into, attached to or separate from the optical head mounted display. Such a plane of optical markers can measure, for example, 1x1 cm, 2x2 cm, 2x3 cm, 4x4 cm, 4x6 cm, etc. Alternatively, multiple (such as, for example, two, three or more) optical markers can be used to determine a plane, specifically by determining a plane that corresponds to the principal plane of the surgical table or that is parallel to the principal plane of the surgical table or, for example, that is perpendicular to the plane of the surgical table or, for example, that is at an angle to the surgical table. If the surgical table is covered by a surgical drape, one or more magnetic or attached bases can be attached to the surgical table prior to placement of the surgical drape. After placement of the drape, one or more magnetic or attached optical markers can be affixed to the magnetic bases or attachment mechanisms with the inserted drape. Alternatively, one or more holding arms or extenders of known geometry can be connected to the surgical table and one or more optical markers can be connected to or can be integrated into the holding arms or extenders. An image and / or video capture system that is integrated into, attached to or separate from the optical head mounted display can then recognize the location, position, orientation and / or alignment of the one or more optical markers. The resulting information can be used to determine the principal plane of the surgical table on which the patient lies. The image and / or video capture system that is integrated into or attached to the optical head mounted display can be used to reference one or more optical head mounted displays with the surgical table and / or the attached optical markers. Once the principal plane of the surgical table is determined in the system, virtual surgical steps for the principal plane of the surgical table can be planned in the virtual surgical plan for the patient. For example, one or more bone cuts can be planned and / or performed perpendicular to the principal plane of the surgical table, such as, for example, with the patient in a supine or prone position or any other desired position. One or more bone cuts can be planned and / or performed at a defined angle other than 90 degrees relative to the horizontal plane of the surgical table, such as, for example, with the patient in a supine or prone position or any other desired position. One or more bone cuts can be planned and / or performed in a non-orthogonal plane or direction relative to the principal plane or horizontal plane of the surgical table, such as, for example, with the patient in a supine or prone position or any other desired position, optionally referencing a plane displayed by the optical head mounted display that is perpendicular to the surgical table.The main plane of the surgical table can be used in this way as a reference, including for comparing or referencing patient virtual data and patient real-time data, and including for comparing or referencing virtual surgical plans. Orthogonal or non-orthogonal angle (such as relative to the surgical table or relative to patient anatomy, anatomical landmarks, anatomical axes or biomechanical axes) bone cuts can be performed using one or more virtual surgical guides or cutting blocks and / or one or more physical surgical guides or cutting blocks. Virtual surgical guides or cutting blocks can include one or more dimensions corresponding to physical surgical guides or cutting blocks.

[0567] One or more optical markers affixed or linked to the surgical table can also be used as a fixed reference for one or more optical head-mounted displays during a surgical procedure. This can be useful, for example, when the patient and / or limb and / or surgical site moves during the procedure. A fixed reference to the surgical table can help maintain registration of one or more optical head-mounted displays and virtual surgical plans and patient real-time data and / or the operating room.

[0568] In some embodiments of the present application, one or more optical markers can be placed in the surgical field and / or in areas away from the surgical field or attached to the patient. An image and / or video capture system integrated into, attached to or separate from the optical head mounted display can be used to identify the one or more optical markers and determine their location, position, orientation and / or alignment. The image and / or video capture system can also optionally determine the location, position, orientation and / or alignment of one or more optical markers attached to or referenced to the surgical table. The system can reference the coordinates and / or spatial relationship of the one or more optical markers attached to the patient in the surgical field and / or in areas away from the surgical field and the surgical table to which the one or more optical markers are attached or referenced. In this manner, if the patient's body moves during surgery, such as during proximal femoral reaming or acetabular reaming during hip arthroplasty, or during impaction of femoral or tibial components during knee arthroplasty, or during nailing or cutting of bone, or during placement of spinal devices such as cage subs or pedicle screws, movement between the one or more optical markers attached to the patient in the surgical field and / or in areas away from the surgical field and the one or more optical markers attached to or referenced to the surgical table and coordinate changes of the one or more optical markers attached to the patient in the surgical field and / or in areas away from the surgical field can be detected and the amount of motion, direction of motion and magnitude of motion can be determined; such as the resulting information can be used to update or adjust or modify the virtual surgical plan or update or adjust or modify the display of the virtual surgical plan or virtual surgical steps or virtual displays for the motion of the patient, including by updating, moving or adjusting one or more aspects and / or portions of the virtual surgical plan, including one or more virtual surgical tools, virtual surgical instruments including: virtual surgical guides or cutting blocks, virtual trial implants, virtual implant components, virtual implants or virtual devices, predetermined starting points, predetermined starting positions, predetermined starting orientations or alignments, predetermined intermediate points, predetermined intermediate positions, predetermined intermediate orientations or alignments, predetermined end points, predetermined final positions, predetermined end orientations or alignments, predetermined paths, predetermined planes, predetermined cutting planes, predetermined contours or cross sections or surface features or shapes or projections, predetermined depth markers or depth gauges, predetermined angles or orientations or rotational markers, predetermined axes, such as rotational axes, bending axes, extension axes, predetermined axes of virtual surgical tools, virtual surgical instruments including virtual surgical guides or cutting blocks, virtual trial implants, virtual implant components, virtual implants or devices, non-visualized portions of one or more devices or implants or implant components or surgical instruments or surgical tools, and / or predetermined tissue changes or changes using new patient coordinates or new coordinates of the surgical field.

[0569] In some embodiments of the application, part or the entire optical marker can be radiopaque, such that the optical marker can also be visible in radiographs or other imaging examinations with ionizing radiation, including fluoroscopy, digital tomosynthesis, cone beam computed tomography and / or computed tomography. Different levels or degrees of radiopacity can be present in different parts or regions of the optical marker. Different levels or degrees of radiopacity can be utilized to encode information. For example, different levels of radiopacity can be used to encode information saved in an optically readable alphanumeric code, barcode or QR. Different levels of radiopacity can optionally be arranged like a thickness profile bar, which optionally mirrors part or all of the information contained in a barcode. Different levels of radiopacity can optionally be arranged like a thickness profile in dots or squares, which optionally mirrors part of the information contained in a two-dimensional code. Different levels of radiopacity can be obtained by varying the thickness of the metal, such as lead. Radiopaque optical markers encoding information in this way can be manufactured using three-dimensional metal printers. They can also be CNC machined, such as from bar stock or cast billets.

[0570] The radiopaque part of the optical marker can include laterality information, such as left for left and right for right, visible on a radiograph, such as by different material thickness, such as lead; the same information can be contained in an additional alphanumeric code or text, barcode or two-dimensional code, which can be read by a barcode or two-dimensional code reader or an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display. The radiopaque part of the optical marker can include anatomical site information, such as L5 or L4, T1 or T2, C3 or C7, knee, hip, visible on a radiograph, such as by different material thickness, such as lead; the same information can be contained in an additional alphanumeric code or text, barcode or two-dimensional code, which can be read by a barcode or two-dimensional code reader or an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display. Image processing techniques and / or software can be applied to the radiographic information including the optical marker and the radiographic encoded information such as laterality and / or location, and can compare the information included in the radiograph to the information included on the optical scan. If any discrepancies are detected, an alert can be triggered, such as can be displayed in the optical head-mounted display.

[0571] Multiple partially or fully radiopaque optical markers can be used. Radiopaque optical markers can be applied in different locations and planes around the surgical site. In spinal surgery, for example, one, two, three or more radiopaque optical markers can be applied to the skin around the spinal segment for the intended surgery; one, two, three or more radiopaque optical markers can be attached to the pins, drills or screws inserted into the spinous processes and / or pedicles or other spinal elements; one, two, three or more radiopaque optical markers can be applied to the patient's flank or abdomen. In hip replacement surgery, one, two, three or more radiopaque optical markers can be applied to the anterior superior iliac spine on the patient's intended surgical side, such as with adhesive to the skin or attached to a needle or drill to the bone; one, two, three or more radiopaque optical markers can be applied to the anterior superior iliac spine on the patient's opposite side, such as with adhesive to the skin or attached to a needle or drill to the bone; one, two, three or more radiopaque optical markers can be attached at the pubic symphysis, such as with adhesive to the skin or attached to a needle or drill to the bone; one, two, three or more radiopaque optical markers can be applied to the acetabulum on the patient's intended surgical side, such as with pins or drills attached to the bone. By using multiple radiopaque optical markers in multiple different locations and planes around the surgical site, the accuracy of the three-dimensional spatial registration and cross-referencing of the optical markers in different modalities, such as radiographs, image capture, can be increased, such as obtaining multiple X-rays at different angles from multiple perspectives, such as anteroposterior, lateral and / or oblique, and / or imaging the radiopaque optical markers from multiple perspectives by using image and / or video capture systems integrated into, connected to or separate from the optical head-mounted display. By using multiple optical markers in multiple different locations and planes around the surgical site, the accuracy of the three-dimensional spatial registration of the optical markers can be increased, such as imaging the optical markers from multiple perspectives by using image and / or video capture systems integrated into, connected to or separate from the optical head-mounted display. Additionally, the accuracy of the registration can be better maintained when the perspective or radiographic angle is changed, such as during a surgical procedure or due to patient movement.

[0572] In some embodiments of the application, system performance can be tested. For example, system performance testing can measure a phantom comprising two or more optical markers at a known location, position, orientation and / or alignment. With known coordinates of the two or more optical markers and distances and angles between the markers, using image and / or video capture systems integrated into, attached to or separate from the optical head-mounted display, accuracy of distance measurements and / or angle measurements and / or area measurements and / or volume measurements can be determined. Further, using image and / or video capture systems integrated into, attached to or separate from the optical head-mounted display, repeatability and / or precision of distance measurements and / or angle measurements and / or area measurements and / or volume measurements can be determined by repeating measurements. Using image and / or video capture systems integrated into, attached to or separate from the optical head-mounted display, accuracy and / or repeatability and / or precision of distance measurements and / or angle measurements and / or area measurements and / or volume measurements under static and dynamic conditions can be determined. Static conditions can be where the patient, spine, extremity, joint and / or bone is not moving. Dynamic conditions can be where the patient, spine, extremity, joint and / or bone is moving during image capture. For example, dynamic conditions can be used to determine the center of rotation of a joint. Measurements under static and dynamic conditions can be performed for different angles of view and distances of the image and / or video capture systems integrated into, attached to or separate from the optical head-mounted display. Measurements under static and dynamic conditions can be performed while the optical head-mounted display is stationary, as opposed to moving. Measurements under static and dynamic conditions can be performed while the optical head-mounted display is not stationary, as opposed to moving, such as with the head of the operator.

[0573] Table 5 illustrates exemplary tests with various combinations of test conditions and test parameters where accuracy and repeatability and / or precision of measurements can be determined. Any combination is possible. Other parameters can be measured, such as repeatability of color temperature (e.g., in Kelvin). Other statistical tests can be applied. All measurements and all statistical determinations and parameters under static, dynamic, optical head-mounted display not moving and optical head-mounted display moving conditions can be evaluated, including different angles and distances of the image and / or video capture systems from the target anatomy and / or test device and / or phantom.

[0574]

[0575]

[0576] Once the accuracy and / or reproducibility and / or precision of the distance measurements and / or angle measurements and / or area measurements and / or volume measurements and / or coordinate measurements of the image and / or video capture system integrated into the optical head-mounted display, attached to the optical head-mounted display, or separate from the optical head-mounted display is determined, defining a threshold can indicate when the system is operating outside of a clinically acceptable range of performance. The threshold can be determined using standard statistical methods known in the art. For example, when the angle of view and / or distance or speed of movement of the image and / or video capture system integrated into the optical head-mounted display, attached to the optical head-mounted display, or separate from the optical head-mounted display indicates that the measured values fall outside of two standard deviations of the system performance including the overall system performance, a warning can be triggered to the surgeon that the display of virtual data, such as a portion of a virtual surgical plan, a virtual projected path, or a virtual plane, such as a virtual cutting plane, can not be accurate. A binary, such as yes, no, system can be used to trigger an alarm that the image and / or video capture system and / or OHMD display is operating outside of a clinically acceptable range of performance, such as exceeding a certain angle of view, exceeding or falling below a certain distance to a target anatomical structure, or exceeding an acceptable speed of movement. Alternatively, a sliding scale can be used as the system progressively enters a range outside of a clinically acceptable range of performance. The sliding scale can be a color scale from green to red with mixed colors in between. The sliding scale can be an acoustic signal that can further increase in intensity or frequency as the system operates outside of a clinically acceptable range. The sliding scale can be a vibration signal that can further increase in intensity or frequency as the system operates outside of a clinically acceptable range. In some embodiments of the present invention, the optical head-mounted display optionally turns off the display of patient virtual data, such as virtual plan information, virtual surgical guides or cutting blocks, or virtual planes or intended paths, when one or more test data indicates that the system is running outside of its clinically acceptable range of performance. The OHMD display can turn back on when the test data indicates that the system is running within the clinically acceptable range of performance again. The system testing, including precision testing and reproducibility testing, can be performed intermittently, such as every 3 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, etc. The system testing can be performed continuously. The system testing can be performed intermittently or continuously, but only for the time that the optical head-mounted display is displaying virtual data. The system testing can be performed intermittently or continuously, but only for the time of surgical steps that require high precision or reproducibility. These steps that require high precision or high reproducibility can be identified by the surgeon through voice commands or other commands, or they can be identified in the virtual surgical plan, such as automatically or through selection by the surgeon.

[0577] In some embodiments of the application, radiopaque and non-radiopaque optical markers are optionally attached or applied to extenders that increase the distance of the optical markers from the patient's skin. Such extenders can be anchored in the spinous process, pedicle or other spinal elements by pins, drills or screws, for example. The use of extenders with attached radiographic optical markers can increase the registration accuracy between radiographic data and image capture data, such as when using anteroposterior and lateral radiographs. When using image capture, the use of extenders with additional optical markers can account for defining anatomical or instrument axes and other information. When two or more markers are used with an extender and the markers are separated by a distance greater than the spatial resolution of the image and / or video capture system, the accuracy of determining the axis between the two markers, such as the length of the extender and the distance between the markers, can be increased.

[0578] The optical markers can be made visible with other imaging modalities, such as nuclear magnetic resonance, nuclear scintigraphy, single photon emission tomography or positron emission tomography. For example, the optical markers can be doped with a nuclear magnetic resonance contrast agent, such as Gadolinium-DTPA, so that they appear with nuclear magnetic resonance. The optical markers can be doped with an isotope or positron emitter so that they appear with single photon emission tomography or positron emission tomography.

[0579] When the optical markers include a two-dimensional code or when a two-dimensional code is used as an optical marker, it can also address inventory management and quality issues before, during and after surgery. Errors in the patient's surgical side are a common quality issue associated with surgery that can have devastating consequences for the patient. Similarly, in spinal surgery, operating on the wrong spinal segment can result in severe injury to the patient. The optical markers used to determine the position, location, orientation, alignment and / or direction of travel of the patient, limb, joint, surgical site, surgical instrument, trial implant and / or implant assembly can also include any of the following information, such as information contained in a bar code or two-dimensional code included in or attached to the optical marker:

[0580] - patient identifier

[0581] - patient demographics, such as age, gender, height, body mass index (BMI)

[0582] - patient medical history

[0583] - patient risk factors

[0584] - patient allergies

[0585] - side of the body to be operated on, such as left versus right

[0586] - the site to be operated on, such as the knee and the hip, the spinal segments L1 and L2, etc.

[0587] - the spinal segment to be operated on

[0588] - a part of the virtual surgical plan, such as

[0589] o the amount of resection,

[0590] o the resection level for a given surgical step,

[0591] o the position and / or orientation of a bone cut

[0592] o the slope of a tibial cut,

[0593] o the implant rotation, such as femoral component rotation, tibial component rotation

[0594] o the implant flexion, such as femoral component flexion

[0595] o the predetermined depth, location, position, orientation, coordinates of a deburring

[0596] o the predetermined depth, location, position, orientation, coordinates of a reaming

[0597] o the intended depth, location, position, orientation, coordinates of a milling

[0598] o the femoral neck angle,

[0599] o the acetabular angle,

[0600] o the acetabular anteversion,

[0601] o the femoral anteversion,

[0602] o the offset

[0603] o the femoral stem axis

[0604] o the intended implant component axis / alignment

[0605] o the intended polyethylene composition, thickness (such as hip acetabular liner, knee tibial insert, shoulder

[0606] glenoid insert)

[0607] - information related to a template or a size

[0608] o the size of the selected implant component, such as knee femoral, tibial or patellar component, hip acetabular shell, acetabular liner, femoral stem, femoral head, femoral neck portion with a moveable bearing component

[0609] o the side of the implant component, left and right

[0610] - inventory management information, such as,

[0611] o version, type, model of instrument used

[0612] o lot number of instrument used

[0613] o manufacturing location of instrument used

[0614] o manufacturing date of instrument used

[0615] o first sterilization date of instrument used

[0616] o number of sterilization cycles applied to instrument used

[0617] o last sterilization date of instrument used

[0618] o date instrument was delivered to hospital or surgical center

[0619] o version, type, model of implant set used

[0620] o lot number of implant set used

[0621] o manufacturing location of implant set used

[0622] o manufacturing date of implant set used

[0623] o sterilization date of implant set used

[0624] o date implant set was delivered to hospital or surgical center

[0625] o any other information relevant to inventory management

[0626] Optionally, some of these pieces of information can also be included in a two- dimensional code separate from the optical marker. In some embodiments of the application, a separate barcode and / or two-dimensional code reader can be used to read the information contained on the barcode and / or two-dimensional code before, during and / or after the surgery. In some embodiments of the application, an image and / or video capture system integrated into, attached to or separate from the optical head-mounted display can be used to read the information contained on the barcode and / or two-dimensional code. The information read from the barcode and / or two-dimensional code can then be compared, for example, to the portion of the virtual surgical plan and / or to the patient side prepared for the surgery (e.g., left versus right), to the patient site prepared for the surgery (e.g., spinal segment L4 versus L5 (e.g., as seen in a radiograph)), to the physical surgery performed, to the physical instruments selected, to the physical implant trials selected, to the physical implant set selected.

[0627] When a staple or screw is placed in a surgical site including a joint and / or bone, such as, for example, a spinous process or pedicle of a spinal segment, with an integrated or attached optical marker with a two-dimensional code, or when an instrument, trial implant and / or implant assembly with an integrated or attached optical marker with a two-dimensional code enters the field of view of a barcode and / or two-dimensional code reader and / or image and / or video capture system integrated or attached to an optical head-mounted display, or enters the surgical field or near the tissue altered by the surgery, the information on the bar code or two-dimensional code can read the code or two-dimensional code on the physical staple or screw, the physical instrument, the physical trial implant and / or the physical implant assembly and compare it to the expected surgical site information and / or expected laterality information and / or virtual surgical plan and / or expected size information and / or expected template information. In the example of a spinal segment, the barcode and / or two-dimensional code reader and / or image and / or video capture system integrated or attached to an optical head-mounted display can read the two-dimensional code on the side (left or right) identifying the expected spinal segment and side for the staple or pedicle screw or other device. The information can be compared to the virtual surgical plan and / or X-ray information for the patient. For example, the system can use intraoperative X-rays to automatically or semi-automatically or user-operatively identify the spinal segment (e.g., count up from the sacrum), such as, for example, by detecting the sacral endplate and the opposing endplate and / or pedicles. If the system detects a discrepancy in the spinal segment or laterality between the information read from the staple, screw or device and the integrated or attached optical marker and the barcode or two-dimensional code and the virtual surgical plan and / or radiographic information, an alert can be triggered to check the equipment, check the surgical plan and / or reconfirm the spinal segment and / or laterality. The above example is not limited to radiographic information; other imaging studies known in the art, such as, for example, computed tomography, nuclear magnetic resonance, etc., can be used to determine or identify the anatomical site and laterality, including the spinal segment.

[0628] If the reading of the two-dimensional code indicates a discrepancy in any of the information embedded in the two-dimensional code, such as, for example, the site, laterality, segment, multi-part or multi-faceted, size or template information of the virtual surgical plan, compared to the physical real-time data during the surgery, such as, for example, the physical location of the inserted staple or screw, the physical instrument used, the physical trial implant used and / or the physical implant assembly used or the spinal segment or laterality, an alert can be triggered, such as, for example, in the optical head-mounted display or on a computer monitor used to plan, display or modify the virtual surgical plan. The alert can be visual, such as, for example, a red warning sign or stop sign or displayed alert sign, or audible, or vibratory, or a combination thereof. Any other alert known in the art can be used.

[0629] For example, when a surgeon is operating on a patient to replace the patient's left knee, one or more implant components or attached retainers or packaging labels or sterile packaging can include an optical indicia of a two-dimensional code. The two-dimensional code can indicate laterality, such as a left femoral component versus a right femoral component. If a grinding technician accidentally delivers a surgeon's right femoral component for implantation into the patient's left knee, the image and / or video capture system integrated with or attached to the surgeon's optical head-mounted display can read the two-dimensional code when the surgeon takes the femoral component, and when the femoral component with the attached optical indicia and two-dimensional code enters the surgeon's field of view or enters the vicinity of the surgical field. The image and / or video capture system and associated system software can read the two-dimensional code to identify that the implant component is for the right knee; then, the system software can compare the information to the patient's virtual surgical plan or templates and / or size information indicating a left knee plan, and then trigger an alert that the incorrect femoral component has entered the field of view or has entered the vicinity of the surgical field (such as distinguished by another optical indicia). The alert can assist the surgeon in correcting the error by switching to the correct lateral component.

[0630] In another example, when a surgeon is operating on a patient to replace the patient's left knee, one or more implant components or attached retainers or packaging labels or sterile packaging can include an optical indicia of a two-dimensional code. The two-dimensional code can indicate the size of the implant component, such as a size 5 or 6 or other femoral component, or a size 5 or 6 or other tibial component, or a size 2 or 3 or other patellar component. If a grinding technician accidentally delivers a femoral component sized 4 (templated as a femoral component sized 6) for implantation into the patient's left knee to the surgeon, the image and / or video capture system integrated with or attached to the surgeon's optical head-mounted display can read the two-dimensional code when the surgeon takes the femoral component, and when the femoral component with the attached optical indicia and two-dimensional code enters the surgeon's field of view or enters the vicinity of the surgical field. The image and / or video capture system and associated system software can read the two-dimensional code to identify that the implant component is sized 4; then, the system software can compare the information to the patient's virtual surgical plan or templates and / or size information, which can indicate that a femoral component sized 6 has been planned, and then trigger an alert that the incorrect femoral component has entered the surgeon's view or has entered the vicinity of the surgical field (such as distinguished by another optical indicia). The alert can assist the surgeon in correcting the error by switching to the correct size component.

[0631] An image and / or video capture system and / or barcode and / or QR code reader integrated into, attached to, or separate from the optical head-mounted display can also be used to read embedded information about the virtual surgical instruments and / or implant assembly for inventory management and billing and invoicing purposes. For example, the image and / or video capture system and / or barcode and / or QR code reader can detect which instruments are used, monitor their frequency of use, and when a certain recommended frequency of use is reached, the system can trigger an alert to send the instruments for servicing. In some embodiments, the image and / or video capture system and / or barcode and / or QR code reader can detect which instruments are used and trigger an alert to send the instruments for servicing. In some embodiments, the image and / or video capture system and / or barcode and / or QR code reader can detect which disposable instruments are used and trigger a system alert to replenish the supply, sending new, additional disposable instruments to replace the used instruments. In some embodiments, the image and / or video capture system and / or barcode and / or QR code reader can detect which implant assemblies and other chargeable components are used and trigger an alert in the system to replenish the supply, and send new, additional implants to replace the used implants; the alert can also trigger a command to generate an invoice for the hospital and / or surgical center and monitor payment.

[0632] Any of the foregoing embodiments can be applied during any type of surgery to any surgical step and any surgical instrument or any implant assembly, such as knee joint replacement, hip joint replacement, shoulder joint replacement, ligament repair, including anterior cruciate ligament repair, spinal surgery, spinal fusion (such as anterior and posterior), vertebroplasty, and / or kyphoplasty.

[0633] In some embodiments of the application, pins or other implantable or attached markers or calibration or registration phantoms or devices including optical markers can be initially placed in, for example, bone or osteophytes or bone spurs or other bone anatomy or deformities. Registration of virtual image data can be performed, such as using anatomical landmarks or locations or osteophytes or bone spurs or other bone anatomy or deformities where the pins have been physically placed and optionally marked on the electronic image, and patient real-time data is processed. The pins can then be selectively removed, such as if they would interfere with steps of the surgical procedure. After steps of the surgical procedure are performed (such as bone cuts), the pins are optionally reinserted into the pinholes left in the residual bone under the bone cuts, and the pins can be used to register the virtual data. The patient real-time data can be obtained even if the surgical site and anatomy have been changed by the surgical procedure.

[0634] In some embodiments of the application, the techniques described herein can be used to register patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or registered with surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue features in the patient virtual data. Matching, superimposing and / or registering patient real-time data and patient virtual data after surgical tissue alteration can use the same techniques described in the foregoing or any other registration techniques described in this specification or any other registration techniques known in the art.

[0635] Registering patient virtual data and patient real-time data using patient specific markers or templates

[0636] Various techniques have been described for registering patient virtual data to patient real-time data using patient specific markers or templates, including those described in WO9325157A1, which is expressly incorporated herein by reference.

[0637] In some embodiments of the application, pre-operative imaging is performed to acquire three-dimensional data of the patient. For example, pre-operative imaging can entail ultrasound, computed tomography or nuclear magnetic resonance, as previously described, optionally with injection of contrast agent.

[0638] Pre-operative imaging can include a single region or site, such as a lumbar vertebra or a portion of a lumbar vertebra or one or more spinal segments, or a single joint (such as a knee joint, hip joint, ankle joint, shoulder joint, elbow joint or wrist joint). Alternatively, pre-operative imaging can include scanning a portion or all of one or more adjacent joints. This approach can be beneficial when information is needed regarding limb length or axial or rotational alignment. For example, when planning a hip replacement surgery, image information acquired through the distal femur and optionally the knee joint and / or ankle joint is helpful in determining leg length, etc. When planning a knee replacement surgery, it is beneficial to obtain image information through the hip joint and ankle joint. For example, in this manner, the hip center and ankle joint are determined. This information can be used to determine the mechanical axial alignment of the patient, and optionally, to plan a mechanical axial correction.

[0639] Preoperative imaging can also be performed in one or more positions, such as prone, supine, standing, flexed, extended, laterally bent, etc. Data obtained from scanning the patient in different positions can be optionally combined or fused. For example, standing weight-bearing portions or full leg X-rays can be used to determine mechanical axis alignment of the leg. For example, three-dimensional data of the knee from computed tomography or magnetic resonance can be used to obtain detailed anatomic information about the joint, such as to obtain surface shape information and design patient-specific markers or templates. Information from standing scans can be used to align patient-specific markers or templates or orientation relative to the mechanical axis. Information from three-dimensional knee scans can be used to derive one or more patient-specific surfaces that are adapted to the unique shape of the patient.

[0640] In patients with spinal symptoms, three-dimensional data of the spine can be obtained by computed tomography or magnetic resonance scans or rotational fluoroscopy or C-arm scans. For example, standing imaging in flexion and extension can be used to determine the presence and extent of spinal instability, such as prior to an intended spinal fusion procedure with pedicle screws and / or fusion cages. The extent of instability or slippage can be determined and used to determine the extent of intended correction, if any, or the extent of laminotomy required, both of which can be optionally predetermined from the three-dimensional data. Optionally lateral bending views are used to determine the extent and angle of partial vertebral body resection and the desired location and / or height of intervertebral pedicles. Thus, data from standing imaging can be combined or optionally fused with data from supine or prone imaging. Data from two-dimensional imaging can be combined or fused with data from three-dimensional imaging. Three-dimensional data can be used to derive one or more patient-specific surfaces that are adapted to the unique shape of the patient, such as the unique shape of one or more of the patient's spinous processes, one or more of the patient's transverse processes, one or more of the patient's lamina, one or more of the patient's articular processes, one or more of the patient's vertebral bodies.

[0641] Patient-specific markers or templates can include one or more surfaces that are designed and manufactured to fit the corresponding surface of the patient, typically similar to or substantially negative. Optionally the surfaces can be smoothed. Alternatively, the surfaces can be intentionally "roughened" to include more surface features than the three-dimensional surface of the section of the patient's target anatomy. Such surface features can include spikes or needle-like structures to allow the patient-specific markers or templates to enhance fixation on the surface of the patient's tissue.

[0642] The patient-specific marker or template can be developed from computed tomography, nuclear magnetic resonance or ultrasound scans as well as X-ray imaging. In principle, any multiplanar two- or three-dimensional imaging modality is suitable, in particular when it provides information about the surface shape or provides information to derive an estimate of the surface shape of an anatomical region. The patient-specific marker or template can comprise one or more surfaces which are designed or manufactured to fit the corresponding

[0643] - the patient's cartilage surface

[0644] - the patient's subchondral bone surface

[0645] - the patient's cortical bone surface

[0646] - the patient's bone disease or spur

[0647] - the patient's bone defect

[0648] - the patient's exostosis

[0649] - the patient's subchondral cyst

[0650] - the shape of soft tissue, such as the shape of the thigh or calf or lower back, or chest region, or neck region, or foot or ankle region, or shoulder region, etc.

[0651] - the shape of soft tissue in different body positions or postures, such as in prone or supine or lateral decubitus

[0652] - the patient's ligament

[0653] - the patient's upper lip

[0654] - the patient's meniscus

[0655] - the shape of an organ of the patient

[0656] - the organ margin or patient margin, such as the liver margin or the spleen margin

[0657] For a given tissue, different imaging modalities are particularly suitable. For example, if the patient-specific marker or template is designed to fit the patient's cartilage shape, then nuclear magnetic resonance and ultrasound or computed tomography arthrography are most suitable to provide surface information. If the patient-specific marker or template is intended to fit the subchondral bone shape or the cortical bone shape, then computed tomography can be used, although nuclear magnetic resonance and ultrasound can also provide information about the bone shape.

[0658] The patient-specific markers or templates can be manufactured using different materials, such as ABS or nylon or different types of plastics or metals. They can be machined from a blank, where a CAD / CAM machining system transfers the patient-specific shape information into a milling machine. They can also be produced using stereolithography or three-dimensional printing techniques known in the art. If three-dimensional printing is used, air cleaning operations and / or water baths can be used to remove residual powder. Three-dimensional printing can be performed using powder-based or liquid resin-based methods, including but not limited to continuous liquid interface production.

[0659] The patient-specific markers or templates can include or incorporate optical markers, such as optical markers having different geometric shapes or patterns, two-dimensional codes, bar codes, alphanumeric codes. The geometric shapes or patterns, two-dimensional codes, bar codes, alphanumeric codes can optionally be printed, such as when three-dimensional printing is used to manufacture the patient-specific markers or templates. The three-dimensional printing can be performed with software, such as Materialise Magics (Materialise, Leuven, Belgium) and hardware known in the art, such as 3D printers by 3D Systems, Rock Hill, SC or Concept Laser, Lichtenfels, Germany.

[0660] The patient-specific markers or templates can be made with different material properties. For example, they can be non-elastic, semi-elastic or elastic. They can be very hard. They can be solid or include hollow spaces or openings. They can be opaque. The patient-specific markers or templates can be translucent. The patient-specific markers can be transparent. In some embodiments, the patient-specific markers or templates can be semi-opaque or translucent. However, when the patient-specific markers or templates are in contact with the patient and the patient-specific surface of the marker or template is well-matched to the corresponding surface of the patient, the patient-specific markers or templates are transparent due to the presence of tissue moisture on the corresponding surface of the patient.

[0661] One or more patient-specific markers or templates can be used on a first surface of a joint. One or more patient-specific markers can be used on a second surface of the joint. The first and second surfaces can be located on the same load-bearing side of the joint. The first and second surfaces can be located on opposite sides of the joint. The one or more patient-specific markers or templates on the first surface of the joint can not be connected to the one or more patient-specific markers or templates on the second surface of the joint. In some embodiments, the one or more patient-specific markers or templates on the first surface of the joint are optionally connected or connected to the second surface of the joint. Thus, the one or more patient-specific markers or templates are optionally referenced to each other.

[0662] Patient-specific markers or templates can be designed for any joint, any portion of the spinal column, and any tissue of the human body. Patient-specific markers or templates generally include one or more surfaces or shapes that are designed to fit a corresponding surface or shape of the patient. Representative, non-limiting examples of patient surfaces for which patient-specific markers or templates can be designed and / or installed include:

[0663] Spine:

[0664] - a portion or the entire spinous process

[0665] - a portion or the entire lamina

[0666] - a portion or the entire spinal facet

[0667] - a portion or the entire facet joint

[0668] - a portion or the entire transverse process

[0669] - a portion or the entire pedicle

[0670] - a portion or the entire vertebral body

[0671] - a portion or the entire intervertebral disc

[0672] - a portion or the entire spinal osteophyte

[0673] - a portion or the entire spinal spur

[0674] - a portion or the entire spinal fracture

[0675] - a portion or the entire vertebral body fracture

[0676] - any combination of the foregoing

[0677] Hip:

[0678] - a portion or the entire acetabulum

[0679] - a portion or the entire rim of the acetabulum

[0680] - multiple portions of the rim of the acetabulum

[0681] - a portion of the iliac wall

[0682] - a portion of the pubic bone

[0683] - a portion of the ischium

[0684] - a portion or the entire greater trochanter

[0685] - a portion or the entire lesser trochanter

[0686] - a portion or the entire femoral shaft

[0687] - a portion or all of the femoral neck

[0688] - a portion or all of the femoral head

[0689] - the femoral head concavity

[0690] - the transverse acetabular ligament

[0691] - the posterior tubercle of the thalamus

[0692] - a ligament

[0693] - the upper lip

[0694] - one or more osteophytes, femoral and / or acetabular

[0695] - any combination of the foregoing

[0696] KNEE:

[0697] - a portion or all of the medial condyle

[0698] - a portion or all of the lateral condyle of the femur

[0699] - a portion or all of the femoral notch

[0700] - a portion or all of the trochlea

[0701] - a portion of the anterior cortex of the femur

[0702] - a portion of the anterior cortex of the femur and an adjacent portion of the trochlea

[0703] - a portion of the anterior cortex of the femur with adjacent portions of the trochlea and osteophytes, when present

[0704] - one or more osteophytes femoral and / or tibial

[0705] - one or more spurs femoral and / or tibial

[0706] - the supracondylar ridge

[0707] - a portion or all of the central tibial plateau

[0708] - a portion or all of the lateral tibial plateau

[0709] - a portion or all of the central tibial spine

[0710] - a portion or all of the lateral tibial spine

[0711] - a portion of the anterior cortex of the tibia

[0712] - a portion of the anterior cortex of the tibia and a portion of the tibial plateau, medial or lateral or both

[0713] - a portion of the anterior cortex of the tibia and a portion of the tibial plateau, medial or lateral or both and osteophytes, when present

[0714] - a portion or the entire patella

[0715] - the medial edge of the patella

[0716] - the lateral edge of the patella

[0717] - the superior pole of the patella

[0718] - the inferior pole of the patella

[0719] - a patellar osteophyte

[0720] - the anterior cruciate ligament

[0721] - the posterior cruciate ligament

[0722] - the medial collateral ligament

[0723] - the lateral collateral ligament

[0724] - a portion or the entire medial meniscus

[0725] - a portion or the entire lateral meniscus

[0726] - any combination of the foregoing

[0727] shoulder:

[0728] - a portion or the entire glenoid

[0729] - a portion or the entire coracoid process

[0730] - a portion or the entire acromion

[0731] - a portion of the clavicle

[0732] - a portion or the entire humeral head

[0733] - a portion or the entire humeral neck

[0734] - a portion of the humeral shaft

[0735] - one or more humeral osteophytes

[0736] - one or more glenoid osteophytes

[0737] - a portion or the entire glenoid labrum

[0738] - a portion or the entire shoulder ligament, such as the coracoclavicular ligament, superior, middle, or inferior glenohumeral ligament

[0739] - a portion of the scapula

[0740] - any combination of the foregoing

[0741] skull and brain:

[0742] - a portion of the skull

[0743] - a portion of the occipital bone

[0744] - a portion of the temporal bone

[0745] - a portion of the occipital bone

[0746] - a portion of the parietal bone

[0747] - a portion of the frontal bone

[0748] - a portion of the facial skeleton

[0749] - a portion or the entire bony structure inside the skull

[0750] - a portion or all of a selected gyrus

[0751] - a portion or all of a selected sulcus

[0752] - a portion of a sinus

[0753] - a portion of a venous sinus

[0754] - a portion of a vessel

[0755] Organ:

[0756] - a portion of an organ, such as the upper or lower pole of a kidney

[0757] - a margin or edge of the liver, spleen, lung

[0758] - a portion of a lobe of the liver

[0759] - a portion of a vessel

[0760] - a portion of a fissure, such as in the liver or spleen

[0761] - a portion of the uterus

[0762] The patient-specific marker or template can be designed or adapted to any of the foregoing tissues, if applicable to the particular anatomical region, such as cartilage, subchondral bone, cortical bone, osteophytes, etc. The patient-specific marker or template can be designed or adapted to normal tissue only. The patient-specific marker or template can be designed or adapted to abnormal or diseased tissue only. The patient-specific marker or template can be designed or adapted to a combination of normal and abnormal or diseased tissue. For example, a patient-specific marker can be designed to normal cartilage, or to diseased cartilage, or to a combination of normal and diseased cartilage, such as on the same or opposite articular surface. The patient-specific marker can be used to register one or more normal or pathological tissues or structures in a common coordinate system, such as with one or more optical headsets of the patient and virtual data. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0763] Patient-specific markers or templates can be designed using patient virtual data, such as from preoperative imaging studies such as computed tomography scans, nuclear magnetic resonance scans, or ultrasound scans. The patient-specific markers or templates include one or more surfaces designed and / or manufactured to achieve a close fit with a corresponding surface of the patient.

[0764] In some embodiments of the present application, the surgeon or operator can apply the patient-specific marker or template to the corresponding tissue of the patient. Once a satisfactory fit is achieved and the two corresponding surfaces are in substantial contact, the patient-specific marker or template can be used to register the virtual surgical plan with the patient virtual data and optionally the patient real-time data. By applying the patient-specific marker or template to the corresponding surface of the patient, the surgeon can effectively identify the corresponding structure or surface in the virtual data and the patient real-time data.

[0765] The position, location, and / or orientation of the patient-specific marker or template relative to the optical head-mounted display can then be determined. Any of the embodiments described herein can be applied to determine the position, location, and / or orientation of the patient-specific marker or template relative to the optical head-mounted display. For example, the side of the patient-specific marker or template opposite the patient-specific surface can include certain standardized geometric features, such as a rectangle, triangle, circle, etc., to facilitate identification by the image and / or video capture system integrated with or coupled to the optical head-mounted display. In alternative embodiments, the patient-specific marker or template can include one or more inertial measurement units, including accelerometers, magnetometers, and gyroscopes, similar to the optical head-mounted display. In some embodiments, the patient-specific marker or template can include one or more radio frequency tags or markers or retro-reflective markers, and its position, location, and / or orientation can be captured by the surgical navigation system. The radio frequency tags can be active or passive. The optical head-mounted display can also optionally include one or more radio frequency tags or markers or retro-reflective markers, and its position, location, and / or orientation can also be captured by the surgical navigation system and referenced to the patient-specific marker or template. The patient-specific marker or template can also include a light source, such as a laser or light-emitting diode (LED). For example, a laser can be projected onto a wall or ceiling, and the optical head-mounted display can be referenced thereto. Light-emitting diodes attached or integrated into the patient-specific marker or template can be identified, such as by the image and / or video capture system integrated with or coupled to the optical head-mounted display.

[0766] In further embodiments, one or more surgical instruments used during surgery and / or one or more implanted devices can include certain standardized geometric features, such as rectangles, triangles, circles, etc., that are easily identifiable by an image and / or video capture system integrated into or connected to or coupled to the optical head-mounted display. In alternative embodiments, one or more surgical instruments used during surgery and / or one or more implanted devices can include one or more inertial measurement units, including accelerometers, magnetometers, and gyroscopes, similar to the optical head-mounted display. In some embodiments, one or more surgical instruments used during surgery and / or one or more implanted devices can include one or more radio frequency tags or markers or retro-reflective markers, and their position, location, and / or orientation can be captured by the surgical navigation system. The optical head-mounted display can also optionally include one or more radio frequency tags or markers or retro-reflective markers, whose position, location, and / or orientation can also be captured by the surgical navigation system and referenced to the patient-specific markers or templates and / or one or more surgical instruments used during surgery and / or one or more implanted devices. One or more surgical instruments used during surgery and / or one or more implanted devices can also include a light source, such as a laser or light-emitting diode. For example, a laser can be projected on a wall or ceiling, and the optical head-mounted display and patient can be referenced relative to the laser. For example, an image and / or video capture system integrated into or connected to or coupled to the optical head-mounted display can recognize a light-emitting diode attached to or integrated into one or more surgical instruments and / or one or more implanted devices. Optionally, multiple light-emitting diodes are used. Optionally, two or more of the multiple light-emitting diodes emit light of different wavelengths or colors. The two or more light-emitting diodes can be positioned in a spatially defined position and orientation, such as positioned at a pre-defined or fixed distance and at one or more pre-defined or fixed angles. In this way, when the image and / or video capture system is close to the surgeon's eye, the two or more light-emitting diodes can be positioned by the image and / or video capture system integrated into the optical head-mounted display, attached to the optical head-mounted display, or separate from the optical head-mounted display, and by the measured distance and / or angles seen by the image and / or video capture system. For example, this can be used to determine the distance and / or orientation of the surgeon to a target anatomical structure. By using light-emitting diodes of different wavelengths or colors, the image and / or video capture system can distinguish between the different light-emitting diodes; when the light-emitting diodes are arranged in a known spatial orientation, this information can help improve the accuracy of registration and / or be used to obtain precise distance, angle, orientation, and / or velocity measurements. In all embodiments where light-emitting diode use or adaptation is included, two or more light-emitting diodes of different wavelengths and colors, measurements, or registrations are used throughout the specification.

[0767] Optionally, the selective patient-specific markers or templates used during surgery and one or more surgical instruments and / or implanted devices can also include color markers, color markers of different geometrical shapes or color markers at different, known locations and different, known angles, such as, can be recognized by image and / or video capture systems integrated to or attached to or coupled to the optical head-mounted display and, such as, estimate distances and angles, such as, distances and angles from the surgical site to the optical head-mounted display or between two markers, two surgical instruments or medical device components.

[0768] Optionally, the patient-specific markers or templates and one or more surgical instruments and / or one or more implanted devices used during surgery can also include scales of various dimensions, such as, metric distances, inches, or angular scales. Such scales or angles can be recognized by image and / or video capture systems integrated to or attached to or coupled to the optical head-mounted display and, such as, estimate distances and angles, such as, distances and angles from the surgical site to the optical head-mounted display or between two surgical instruments or medical device components.

[0769] In some embodiments of the present application, the patient-specific markers or templates can be attached to the respective surface of the patient or to the adjacent surface of the patient, such as, using tissue glue (e.g., fibrin glue or needles or pins).

[0770] In some embodiments, the patient-specific markers or templates can include openings or guides, such as, for accepting surgical instruments or tools, such as, drill bits, saws, reamers, pins, screws and any other instruments or tools known in the art.

[0771] By using the patient-specific markers or templates and one or more selective surgical instruments and / or one or more implanted devices used during surgery and the optical head-mounted display to refer to each other the patient virtual data and the patient real-time data, it is now possible to obtain and use during surgery the coordinate information, distance information, axis information, functional information contained in the patient virtual data.

[0772] In some embodiments of the present application, the techniques described herein can be used to register patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or registered with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the patient virtual data. The matching, superimposing and / or registering of the patient real-time data and the patient virtual data after the surgical tissue alteration can use the same techniques described in the foregoing or any other registration techniques described in this specification or any other registration techniques known in the art.

[0773] Registering patient virtual data and patient real-time data using intraoperative imaging

[0774] In some embodiments of the present application, intraoperative imaging can be performed, such as using X-ray imaging or computed tomography imaging and / or ultrasound imaging. The patient virtual data obtained using intraoperative imaging can be used to register the patient virtual data obtained preoperatively, such as using preoperative X-ray, ultrasound, computed tomography or nuclear magnetic resonance imaging. The preoperative and intraoperative virtual data of the patient and the patient real-time data can be registered in a common coordinate system with one or more optical head-mounted displays by identifying and selectively marking corresponding anatomical landmarks, surfaces, object shapes, such as, for example. In the preoperative virtual data of the patient, the intraoperative virtual data of the patient, such as, for example, the electronic two-dimensional or three-dimensional images of one or more of the above, and the patient real-time data, can be the surgical site or target tissue. The virtual preoperative, virtual intraoperative and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0775] This embodiment can be advantageous when the amount of information obtained by intraoperative imaging is more anatomically or otherwise limited than the amount of information obtainable by preoperative imaging, and vice versa.

[0776] For example, intraoperative imaging can be performed using X-ray imaging, which is typically only two-dimensional. X-ray imaging can be enhanced by image acquisition in more than one plane, such as orthogonal planes or one or more planes separated by a defined angle. Intraoperative X-ray images can be used to identify certain landmarks or shapes, which can then be registered to preoperative imaging and / or real-time data of the patient during surgery. Preoperative imaging can optionally include three-dimensional image data, such as obtained by computed tomography or nuclear magnetic resonance. Acquiring intraoperative images in multiple planes can aid in more accurately defining the location of certain anatomical landmarks, contours or shapes for registration of preoperative virtual data, intraoperative virtual data and real-time data of the patient. Intraoperative virtual data of the patient can be two- or three-dimensional patient intraoperative images for clarification.

[0777] For example, in spinal surgery such as vertebroplasty, kyphoplasty, pedicle screw placement or placement of an anterior spinal device including an artificial intervertebral disc or a fusion cage, intraoperative X-ray imaging can be used to identify the spinal segment of the surgery, certain landmarks or contours in the anteroposterior projection, such as the tip of the spinous process, the facet joints, the superior or inferior tips of the facet joints, the cortical margins of the lamina, the superior or inferior endplates or osteophytes or spurs or other bony anatomy or deformities. Alternatively, the distance between the X-ray tube and the patient resulting in X-ray magnification can be taken into account in any registration in order to improve the registration accuracy of the patient's virtual preoperative data, the patient's virtual intraoperative data or real-time patient data. The intraoperative X-ray images can then be registered and optionally superimposed into the patient's preoperative data or the patient's real-time data through the optical head-mounted display. The patient's intraoperative virtual data, such as the tip of the spinous process, the facet joints, the superior or inferior tips of the facet joints, the cortical margins of the lamina, the superior or inferior endplates, can be registered to the real-time data, such as by pointing the tip of the device through a contact arm or a needle or pin inserted through the skin to the corresponding anatomical landmark and by referencing the tip of the device to the real-time data. In this manner, any of the patient's preoperative virtual data, the patient's intraoperative virtual data and the patient's real-time data and combinations thereof can be co-registered. The two or three data sets, the patient's preoperative virtual data, the patient's intraoperative virtual data and the patient's real-time data, can be optionally seen in the optical head-mounted display. However, in many embodiments, intraoperative imaging can be used only to enhance the registration accuracy of the patient's preoperative virtual data and the patient's real-time data, and the patient's preoperative virtual data, such as of the patient and / or the medical device intended to be placed at the surgical site, and the patient's or the surgical site's real-time data, will be displayed together by the optical head-mounted display.

[0778] In some embodiments of the present application, the registration of the patient virtual data and the patient real-time data, and optionally the intraoperative imaging, can be repeated after one or more surgical steps have been performed using the techniques described herein. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue feature in the intraoperative repeat image data of the patient or patient in situ can be matched, superimposed and / or registered to the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue feature in the patient's virtual data, such as in a virtual surgical plan prepared for the patient. The matching, superimposing and / or registering of the patient real-time data and the patient virtual data after the surgical tissue alteration can use the same techniques described in the foregoing or any other registration techniques described in this specification or any other registration techniques known in the art.

[0779] Registration of patient virtual data and patient real-time data using skin markers or soft tissue markers

[0780] In some embodiments of the application, skin markers and soft tissue markers, calibration or registration phantoms or devices can be used to register preoperative virtual data, optionally register intraoperative virtual data, such as data obtained from intraoperative x-ray imaging, and real-time data viewed through optical head-mounted displays in a common coordinate system having one or more optical head-mounted displays. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system. For example, initial registration between preoperative virtual data and patient real-time data can occur at the beginning of the procedure. For example, initial registration can be performed using corresponding anatomical landmarks, surfaces or shapes, or using intraoperative imaging to produce intraoperative virtual data or other embodiments described in the application. For example, registration can be used to place virtual data and real-time data and optical head-mounted displays into a common coordinate system. Skin markers, calibration or registration phantoms or devices can then be applied. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system. Optionally, or in addition, soft tissue markers, calibration or registration phantoms or devices can be applied. Typically, more than one, such as two, three, four or more, skin markers and soft tissue markers, calibration or registration phantoms or devices are applied. For clarity, the term implantable markers, attached markers, skin markers, soft tissue markers, calibration or registration phantoms or devices as used herein can include optical markers, such as optical markers with different geometric shapes or patterns, with two-dimensional codes, with bar codes, with alphanumeric codes. Skin markers and soft tissue markers, calibration or registration phantoms or devices can be applied to the skin or soft tissue in the form of using tissue-compatible adhesives, including fibrin glue and the like. In some embodiments, one, two, three, four or more skin markers and soft tissue markers, calibration or registration phantoms or devices can be included in a surgical drape or dressing transparent film applied to the skin prior to surgery. Skin markers and soft tissue markers, calibration or registration phantoms or devices can then be registered in real-time data and referenced to virtual data. Skin markers and soft tissue markers, calibration or registration phantoms or devices can be used subsequently, such as when the surgical site is changed and anatomical landmarks, surfaces or shapes used for initial registration of virtual and real-time data are changed or removed, cannot be used or cannot be reliably used to maintain registration between virtual data and real-time data. Virtual preoperative, virtual intraoperative and real-time data can include bone spurs or osteophytes or other skeletal anatomy or deformities.

[0781] In some embodiments of the present application, the techniques described herein can be used to register patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue feature in the live patient can be matched, superimposed and / or registered with the surgically altered tissue or tissue surface or tissue profile or tissue perimeter or tissue volume or other tissue feature in the patient virtual data. The matching, superimposing and / or registering of the patient real-time data and the patient virtual data after the surgical tissue alteration can use the same techniques described in the foregoing or any other registration techniques described in this specification or any other registration techniques known in the art.

[0782] If the markers or phantoms are still present, the same skin markers or soft tissue markers or calibration phantoms or registration phantoms can be used after the performance of one or more surgical steps. Alternatively, a re-registration of the patient real-time data and the patient virtual data can be performed after one or more surgical steps or surgical alterations. After the re-registration, one or more new skin markers or soft tissue markers or calibration phantoms or registration phantoms can be applied and the re-registered real-time and virtual data can be cross-referenced after the surgical steps or alterations. The skin markers or soft tissue markers or calibration phantoms or registration phantoms can then be used for subsequent matching, superimposing, moving and registering of the patient real-time data and the patient virtual data.

[0783] Using calibration or registration phantoms of defined size or shape to register the patient virtual data and the patient real-time data.

[0784] In some embodiments of the present application, calibration or registration phantoms of defined size or shape can be used to perform the registration of the patient virtual data and the patient real-time data. The calibration or registration phantoms can have primarily two-dimensional or three-dimensional characteristics. For example, the calibration or registration phantoms can be primarily arranged or positioned in a single plane. Other calibration phantoms can be located in multiple planes, thereby creating opportunities for registration using multiple planes. For the purposes of clarity, the term calibration or registration phantoms, implantable markers, attachable markers, skin markers, soft tissue markers, calibration or registration phantoms or devices used in this application can include optical markers, such as optical markers with different geometric shapes or patterns, with two-dimensional codes, with bar codes, with alphanumeric codes.

[0785] Such calibration or registration phantom can be attached to the patient's skin. The calibration or registration phantom can be integrated or attached to the surgical drape. The calibration or registration phantom can be attached to the patient's tissue. The calibration or registration phantom can be part of the medical device or a component thereof. The components of the medical device usually have known dimensions. By using the calibration or registration phantom and other markers, the patient real-time data and the patient virtual data can be registered in a common coordinate system, such as, for example, with one or more optical head-mounted displays. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0786] In some embodiments, the calibration or registration phantom comprises known dimensions, angles or geometric two or three dimensional shapes. For example, the calibration or registration phantom can comprise the following structures:

[0787] - circle, oval, ellipse, square, rectangle, complex two-dimensional geometry, two-dimensional geometry with one or more defined distances, two-dimensional geometry with one or more defined angles

[0788] - sphere, egg-shaped structure, cylinder, cube, cuboid, complex three-dimensional geometric shape or shape, three-dimensional geometry with one or more defined distances, three-dimensional geometry with one or more defined angles, three-dimensional geometry with one or more defined surfaces

[0789] Optionally, if preoperative or intraoperative imaging is performed using imaging modalities with ionizing radiation, the calibration or registration phantom can be radiopaque, such as, for example, x-ray imaging, two or three-dimensional fluoroscopy, computed tomography, cone beam computed tomography, etc.

[0790] In some embodiments, the calibration or registration phantom can be visible to nuclear magnetic resonance or to nuclide imaging or single photon emission tomography or positron emission tomography, such as, for example, by including parts or containers in the phantom containing gadolinium-DTPA or radionuclide or positron emission tomography (PET) isotope emitting water. Any contrast agent or nuclear magnetic resonance or nuclide imaging or single photon emission tomography or positron emission tomography visible agent known in the art can be used in this way.

[0791] In some embodiments, the calibration or registration phantom comprises retroreflective markers or features that facilitate detection by image and / or video capture systems. The calibration or registration phantom can also stand out from the patient's tissue, including blood and surgical drapes, by choosing colors, such as, for example, bright green, bright blue, bright yellow, bright pink, etc. Color combinations are possible. Any color or color combination known in the art can be used.

[0792] The calibration or registration phantom can optionally comprise light emitting diodes, optionally powered by a battery. Multiple light emitting diodes can be used. The light emitting diodes can emit light of a known color, hue and intensity, preferably chosen to be easily recognizable by the image and / or video capturing system, as well as by segmentation techniques or algorithms for detecting the position, location and / or orientation of the light emitting diodes.

[0793] The light emitting diodes can be arranged in a spatially defined manner, wherein two or more light emitting diodes are arranged in a defined distance and angle in substantially the same plane or in different planes. If the light emitting diodes are arranged in different planes, the spatial orientation of the planes is known and defined.

[0794] When using two or more light emitting diodes, the two or more light emitting diodes can emit light with different wavelengths, colors, intensities and also optionally blinking frequencies. In this way, an image and / or video capturing system integrated into, attached to or separate from the optical head mounted display can identify each different light emitting diode based on one or more of their different wavelengths, colors, intensities and / or blinking frequencies. When the light emitting diodes are arranged in a spatially defined and known manner, such as using a known distance or angle within the same plane or within different planes, the identification of each individual light emitting diode and the change in distance and angle measured by the image and / or video capturing system can be used to determine the position, location and / or orientation of the optical head mounted display and / or the surgeon's head (such as if the image and / or video capturing system is integrated into or connected to the optical head mounted display), or in certain applications, the movement of the calibration or registration phantom and the patient or body part to which the light emitting diodes are attached.

[0795] The light emitting diodes used throughout the specification can be reusable. The light emitting diodes used throughout the specification can also be disposable, optionally with an integrated disposable battery cell / battery. The light emitting diodes can be operated with wires, such as connected to a power source and / or connected to a wired user interface or control unit. The light emitting diodes can be wireless, such as without a connection to a power source (such as battery powered) and / or connected to a wireless (such as WiFi, Bluetooth) control unit.

[0796] Light emitting diodes can be connected and / or organized in LIF networks. For example, data or information can be transmitted or received from one or more optical headsets to a control unit or computer using one or more LIF networks, which can optionally have a user interface. In this example, light emitting diodes participating or connected in one or more LIF networks can be integrated into or attached to optical headsets. Light emitting diodes participating or connected in one or more LIF networks can be connected to or integrated into, as applicable, a surgeon, operating room staff, a patient, a surgical site, one or more optical headsets, one or more navigation systems, one or more navigation markers, such as retro-reflective markers, infrared markers, radio frequency markers; one or more optical markers, calibration or registration phantoms, etc.

[0797] LIF networks can also be used to transmit or receive data or information about the spatial position, orientation, direction of movement, speed of movement, etc. of individual light emitting diodes. For example, using image and / or video capture systems to measure the position, orientation, direction of movement, speed of movement of the same light emitting diode relative to a surgeon or patient or surgical site, the same light emitting diode can be used to transmit or receive information in a LIF network, which can optionally use different wavelengths, colors, frequencies, blinking patterns, depending on the type of data being transmitted. The information can be about the position, orientation, direction of movement, speed of movement, etc. of individual light emitting diodes. This information can also be data that the optical headset is transmitting or receiving. This information can be information or data that the optical headset is displaying. The information can be information generated or received by navigation markers, radio frequency markers. This information can be information captured by one or more image and / or video capture systems or cameras.

[0798] 1, 2, 3, 4, or more light emitting diodes can be connected or attached to a patient, a target anatomical structure, a surgical site, a surgical site (e.g., using a virtual surgical plan), an optical headset, a second, third, and / or additional optical headset, such as worn by a second surgeon, a scrub nurse, other operating room personnel, a hand, a forearm, an upper arm, and / or other body part of a surgeon / surgical team, after a first, second, or more surgical change.

[0799] For example, when one or more light emitting diodes emit light using different wavelengths, colors, intensities, and optionally blinking frequencies, the relative position, orientation, movement, direction of movement, speed of movement of each light emitting diode can be determined (e.g., using one or more image and / or video capture systems integrated into, connected to, or separate from one or more optical headsets).

[0800] The calibration or registration phantom may optionally include one or more lasers, which may be battery powered. Multiple lasers may be used. The lasers may emit light of known color, hue, and intensity, such as light that can be easily identified by an image and / or video capture system, and any segmentation technique or algorithm may be used to detect the laser's location, position, and / or orientation.

[0801] Lasers can be arranged in a spatially defined manner, where two or more lasers are arranged at one or more defined distances in substantially the same plane or at defined angles or multiple angles in different planes. If the lasers are arranged in different planes, the spatial orientation of the planes can be known and defined.

[0802] The calibration or registration phantom may optionally include an radio frequency (RF) transmitter, which may be battery powered. Multiple RF transmitters may be used. Each RF transmitter may emit one or more signals, selected for easy identification by an RF receiver system used to detect the location, position, and / or orientation of the RF transmitter. One or more RF transmitters may emit signals with different frequencies and intensities, allowing the RF receiver system to distinguish between different RF transmitters.

[0803] Radio frequency (RF) transmitters can be arranged in a spatially defined manner, wherein two or more RF transmitters are arranged at one or more defined distances in substantially the same plane or in different planes at defined angles or angular distances. If the RF transmitters are arranged in different planes, the spatial orientation of the planes can be known and defined.

[0804] The calibration or registration phantom may optionally include an ultrasound (US) transmitter, which may be battery powered. Multiple US transmitters may be used. The US transmitter may transmit signals or signals selected for easy identification by the US receiver or transducer system for detecting the location, position, and / or orie...

Claims

1. A system for preparing a physical joint in a patient's body, comprising: At least one computer processor is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual tool or virtual instrument; and A see-through optical head-mounted display is configured to display the virtual tool or the virtual instrument at a predetermined position, predetermined orientation, or a combination thereof on the surface of the physical joint, so as to overlay the virtual tool or the virtual instrument on the surface of the physical joint. The virtual tool or virtual device is configured to allow superposition and alignment between the virtual tool or virtual device and a physical tool or physical device at the predetermined position, the predetermined orientation, or a combination thereof, and The physical tool or the physical instrument is configured to place a physical implant assembly at a desired angle, version, tilt, resection level, varus or valgus angle, flexion, inclination, rotation, or a combination thereof, or is configured to modify the surface of the physical joint to place the physical implant assembly at a desired angle, version, tilt, resection level, varus or valgus angle, flexion, inclination, rotation, or a combination thereof.

2. The system of claim 1, wherein the physical tool or the physical instrument comprises a saw blade, saw, drill bit, nail, drill, milling cutter, awl, reamer, impactor, broach, cutting block, guide rod, drill guide plate, tibial template, tibial drill tower or tibial keel punch, or wherein the physical tool or the physical instrument is configured to perform one or more bone cuts or bone removals.

3. The system of claim 1, wherein the virtual tool or the virtual device is a positioning indicator of the physical tool or the physical device, wherein the positioning indicator is a three-dimensional digital representation corresponding to at least a portion or axis of the physical tool or the physical device, or wherein the virtual tool or the virtual device is a surface rendering of the physical tool or the physical device.

4. The system of claim 1, wherein the system is used to prepare a physical hip joint for hip replacement, and wherein the predetermined position, predetermined orientation, or combination thereof of the virtual tool or the virtual device determines the cup angle, offset, medial position, lateral position, anteversion angle, tilt, or combination thereof of the acetabular implant assembly or the position, orientation, offset, version, leg length, or combination thereof of the femoral implant assembly, or wherein the desired angle, version, tilt, resection level, varus or valgus angle, flexion, oblique angle, rotation, or combination thereof includes the cup angle, offset, medial position, lateral position, anteversion angle, tilt, or combination thereof of the acetabular implant assembly or the position, orientation, offset, version, leg length, or combination thereof of the femoral implant assembly, or The system is used to prepare a physical knee for knee replacement, and the virtual tool or the virtual instrument determines the position, orientation, varus or valgus angle, flexion, rotation, or a combination thereof of the femoral implant assembly or the tibial implant assembly in the predetermined position, the predetermined orientation, or a combination thereof, or the desired angle, version, tilt, resection level, varus or valgus angle, flexion, tilt, rotation, or a combination thereof, including the position, orientation, varus or valgus angle, flexion, rotation, or a combination thereof of the femoral implant assembly or the tibial implant assembly.

5. The system of claim 1, wherein the predetermined position, the predetermined orientation, or a combination thereof is based on information from X-ray, computed tomography, magnetic resonance imaging, or a combination thereof from the physical joint.

6. The system of claim 1, wherein the system further comprises a robot for guiding the physical tool or the physical instrument, wherein the robot comprises a robotic arm.

7. The system according to claim 1, wherein the virtual surgical guide includes a virtual nail, a virtual drill bit, a virtual axis, a virtual predetermined path, a virtual predetermined trajectory, or a combination thereof.

8. The system of claim 1, wherein the predetermined position, the predetermined orientation, or a combination thereof is tangent to or intersects the surface of the physical joint, and wherein the surface of the physical joint includes an articular surface, a cartilage surface, a subchondral bone surface, a cortical bone surface, opposing articular surfaces, a cut bone surface, a reamed bone surface, a modified bone surface, skin, subcutaneous tissue, ligaments, ligament remnants, menisci, lip structures, intra-articular structures, or a combination thereof.

9. The system of claim 1, wherein the computer processor is configured to generate two or more virtual axes displayed by the see-through optical head-mounted display, which is configured to allow superposition and alignment between the two or more virtual axes and two or more physical pins, physical drill bits or combinations thereof, wherein the two or more physical pins, physical drill bits or combinations thereof are configured to attach physical frames, physical cutting blocks, physical guide rods or physical drilling guides to the physical joints or physical bones.

10. The system of claim 9, wherein the computer processor is configured to generate a virtual frame displayed by the see-through optical head-mounted display, wherein the computer processor is configured to allow at least partial overlay and alignment between the virtual frame and the physical frame.

11. The system of claim 1, wherein the at least one computer processor is configured to: maintain the display of the virtual tool or the virtual instrument with respect to one or more anatomical structures at the predetermined position, the predetermined orientation, or a combination thereof when the physical tool or the physical instrument moves in the user's field of vision or when the patient's physical joint moves.

12. The system of claim 11, wherein the one or more anatomical structures include articular surfaces, cartilage surfaces, subchondral bone surfaces, cortical bone surfaces, opposing articular surfaces, cut bone surfaces, enlarged bone surfaces, altered bone surfaces, anatomical landmarks, biomechanical axes, mechanical axes, or combinations thereof.

13. The system of claim 1, wherein the predetermined position, the predetermined orientation, or combination thereof includes at least one predetermined start position, predetermined start point, predetermined intermediate position, predetermined intermediate point, predetermined end position, predetermined end point, or combination thereof, or wherein the predetermined position, the predetermined orientation, or combination thereof includes at least one predetermined angle, predetermined start orientation, predetermined start alignment, predetermined intermediate orientation, predetermined intermediate alignment, predetermined end orientation, predetermined end alignment, or combination thereof.

14. The system of claim 1, wherein the system comprises at least one infrared marker, radio frequency marker, active marker, passive marker, IMU, LED, optical marker, geometric pattern or combination thereof, and wherein the at least one computer processor is configured to use the at least one infrared marker, radio frequency marker, active marker, passive marker, IMU, LED, optical marker, geometric pattern or combination thereof to track one or more anatomical structures, the physical tool or the physical instrument, the physical joint, the fluoroscopic optical head-mounted display or combination thereof.

15. The system of claim 1, wherein the system further comprises a camera, a video system, an image capture system, a laser scanner, a 3D scanner, or a combination thereof, wherein the camera, video system, image capture system, laser scanner, 3D scanner, or combination thereof is separate from the see-through optical head-mounted display, or the system further comprises a camera, video system, image capture system, laser scanner, 3D scanner, or combination thereof, wherein the camera, video system, image capture system, laser scanner, 3D scanner, or combination thereof is integrated with or connected to the see-through optical head-mounted display.

16. The system of claim 15, wherein the system is configured to detect an optical marker attached to the patient, wherein the optical marker comprises at least one of a geometric pattern, a barcode, a QR code, or a combination thereof.

17. The system of claim 1, wherein the physical implant component is a femoral implant component or tibial implant component in knee replacement, an acetabular implant component or femoral implant component in hip replacement, a glenoid implant component or humeral implant component in shoulder replacement, or a tibial implant component or talus implant component in ankle replacement.

18. The system of claim 1, wherein the at least one computer processor is configured to display a three-dimensional surface rendering of the patient's bone structure via the perspective optical head-mounted display, wherein the three-dimensional surface rendering is generated based on computed tomography, magnetic resonance imaging, or a combination thereof.

Images