Augmented reality guidance for spine and joint surgery

Abstract

An apparatus and method for performing a surgical procedure or procedure with visual guidance using an optical head-mounted display are disclosed.

Description

[0001] Description of the case

[0002] This application is a divisional application of the Chinese invention patent application with application date of March 10, 2017, application number 201780029093.2, and invention name “Device and method for surgery”.

[0003] Related applications

[0004] This patent application claims the benefit of U.S. Provisional Application No. 62 / 307,476, filed March 12, 2016; U.S. Provisional Application No. 62 / 318,157, filed April 4, 2016; U.S. Provisional Application No. 62 / 323,716, filed April 17, 2016; U.S. Provisional Application No. 62 / 331,995, filed May 5, 2016; U.S. Provisional Application No. 62 / 354,780, filed June 26, 2016; and U.S. Provisional Application No. 62 / 371,995, filed August 23, 2016. 8,242; U.S. Provisional Application No. 62 / 393,054, filed September 11, 2016; U.S. Provisional Application No. 62 / 406,379, filed October 10, 2016; U.S. Provisional Application No. 62 / 425,019, filed November 21, 2016; U.S. Provisional Application No. 62 / 445,691, filed January 12, 2017; and U.S. Provisional Application No. 62 / 453,484, filed February 1, 2017, the entire contents of which are incorporated herein by reference in their entirety. Technical Field

[0005] Various aspects of the present invention relate to apparatus and methods for performing a surgical procedure or surgical treatment using an optical head-mounted display for visual guidance. Background Art

[0006] With computer-assisted surgery (e.g., surgical navigation or robotics), preoperative imaging of the patient is available. Imaging can be displayed on an external computer monitor in the operating room (OR), and the patient's anatomy (e.g., measurement points) can be registered with the information displayed on the monitor. Because the surgical site is located in a different location and the surgeon's view coordinate system differs significantly from that of the external computer monitor, hand-eye coordination can be challenging for the surgeon. Summary of the Invention

[0007] Furthermore, aspects of the present invention are directed to enabling simultaneous visualization of real-time patient data (e.g., a patient's spine or joints), and a digital representation of virtual data (e.g., a virtual cut on an optical head mounted display (OHMD) and / or a virtual surgical guide with a cutting block or drill guide). In some embodiments, the surgical site comprising the real-time patient data, the optical head mounted display, and the virtual data are all registered in a common coordinate system. In some embodiments, the virtual data is superimposed on and aligned with the real-time patient data. Unlike virtual reality head mounted systems that mix real-time data, the optical head mounted display allows the surgeon to view real-time patient data (e.g., a surgical field) while simultaneously viewing virtual data of the patient and / or virtual surgical instruments or implants with predetermined positions and / or orientations using the display of the optical head mounted display unit.

[0008] Various aspects of the present invention describe novel apparatus for performing a surgical procedure or surgical treatment using an optical head-mounted display, for example, by displaying a virtual representation of: one or more virtual surgical tools, a virtual surgical instrument including a virtual surgical guide or cutting block, a virtual trial implant, a virtual implant assembly, a virtual implant or virtual device, a predetermined starting point, a predetermined starting position, a predetermined starting orientation or alignment, a predetermined intermediate point, a predetermined intermediate position, a predetermined intermediate orientation or alignment, a predetermined endpoint, a predetermined ending position, a predetermined ending orientation or alignment, a predetermined path, a predetermined plane, a predetermined cutting plane, a predetermined contour or profile or cross-section or surface feature or shape or projection, a predetermined depth marker or depth gauge, a predetermined angle or orientation or rotation marker, a predetermined axis (e.g., a rotation axis, a bending axis, an extension axis, a predetermined axis of a virtual surgical tool, a virtual surgical instrument including a virtual surgical guide or cutting block, a virtual trial implant, a virtual implant assembly, an implant or device, etc.), a non-visualized portion for one or more devices or implants or implant assemblies or surgical instruments or surgical tools, and / or one or more predetermined tissue changes or alterations, etc.

[0009] Various aspects of the present invention relate to an apparatus comprising at least one optical head-mounted display configured to generate a virtual surgical guide. In some embodiments, the virtual surgical guide is a three-dimensional representation in a digital format that corresponds to at least a portion of a physical surgical guide, a position indicator of the physical surgical guide, or a combination thereof. In some embodiments, the at least one optical head-mounted display is configured to display the virtual surgical guide superimposed on a physical joint based at least in part on predetermined position coordinates of the virtual surgical guide, and the virtual surgical guide is configured to align a physical surgical guide or a physical saw blade with the virtual surgical guide to guide bone cutting in the joint.

[0010] In some embodiments, the device includes one, two, three or more optical head-mounted displays.

[0011] In some embodiments, the virtual surgical guide is configured to guide bone cutting in a knee replacement, hip replacement, shoulder replacement, or ankle replacement.

[0012] In some embodiments, the virtual surgical guide includes a virtual slot for a virtual or physical saw blade.

[0013] In some embodiments, the virtual surgical guide includes a planar area for aligning a virtual or physical saw blade.

[0014] In some embodiments, the virtual surgical guide includes two or more virtual guide holes or paths for aligning two or more physical drill bits or pegs.

[0015] In some embodiments, the predetermined position of the virtual surgical guide includes anatomical information and / or alignment information of the joint. For example, the anatomical and / or alignment information of the joint can be based on at least one of joint coordinates, anatomical axes of the joint, biomechanical axes of the joint, mechanical axes, or a combination thereof.

[0016] In some embodiments, at least one optical head mounted display is configured to align a virtual surgical guide based on a predetermined limb alignment, such as a normal mechanical axis alignment of a leg.

[0017] In some embodiments, the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined femoral or tibial component rotation. In some embodiments, the at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined flexion of the femoral component or a predetermined inclination of the tibial component.

[0018] In some embodiments, the virtual surgical guide is configured to guide the proximal femoral cut based on a predetermined leg length.

[0019] In some embodiments, the virtual surgical guide is configured to guide bone cutting of the distal tibia or talus in an ankle replacement, and at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined ankle alignment, wherein the predetermined ankle alignment includes coronal plane implant component alignment, sagittal plane implant component alignment, axial plane component alignment, implant component rotation, or a combination thereof.

[0020] In some embodiments, the virtual surgical guide is configured to guide bone cutting of the proximal humerus in a shoulder replacement, and at least one optical head-mounted display is configured to align the virtual surgical guide based on a predetermined humeral implant component alignment, wherein the humeral implant component alignment includes coronal plane implant component alignment, sagittal plane implant component alignment, axial plane component alignment, implant component, or a combination thereof.

[0021] In some embodiments, the predetermined position of the surgical guide is based on preoperative or intraoperative imaging, one or more intraoperative measurements, intraoperative data, or a combination thereof.

[0022] Aspects of the present invention relate to a device comprising two or more optical head-mounted displays for two or more users, wherein the device is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a three-dimensional representation in a digital format, the three-dimensional representation corresponding to at least one of a portion of a physical surgical guide, a position indicator of the physical surgical guide, or a combination thereof, wherein the optical head-mounted display is configured to display the virtual surgical guide superimposed on a physical joint based at least in part on coordinates of a predetermined position of the virtual surgical guide, and wherein the virtual surgical guide is configured to be used to align a physical surgical guide or a saw blade to guide bone cutting of the joint.

[0023] Aspects of the present invention relate to an apparatus comprising at least one optical head-mounted display and a virtual bone cutting plane, wherein the virtual bone cutting plane is configured to guide bone cutting of a joint, wherein the virtual bone cutting plane corresponds to at least a portion of the bone cutting plane, and wherein the optical head-mounted display is configured to display the virtual bone cutting plane superimposed on a physical joint based at least in part on the coordinates of a predetermined position of the virtual bone cutting plane. In some embodiments, the virtual bone cutting plane is configured to guide bone cutting in a predetermined varus or valgus orientation or a predetermined tibial slope (predetermined tibialslope) or a predetermined femoral flexion or predetermined leg length of an implant assembly.

[0024] Various aspects of the present invention relate to methods for preparing a joint for a prosthesis in a patient. In some embodiments, the method includes registering one or more optical head-mounted displays worn by a surgeon or a surgical assistant in a coordinate system, obtaining one or more intraoperative measurements from the patient's physical joint to determine one or more intraoperative coordinates, registering the one or more intraoperative coordinates from the patient's physical joint in the coordinate system, generating a virtual surgical guide, determining a predetermined position and / or orientation of the virtual surgical guide based on the one or more intraoperative measurements, displaying and overlaying the virtual surgical guide on the physical joint using the one or more optical head-mounted displays based at least in part on the coordinates of the predetermined position of the virtual surgical guide, and aligning a physical surgical guide or a physical saw blade with the virtual surgical guide for guiding bone cutting of the joint.

[0025] In some embodiments, one or more optical head-mounted displays are registered in a common coordinate system. In some embodiments, the common coordinate system is a shared coordinate system.

[0026] In some embodiments, a virtual surgical guide is used to guide bone cutting in a knee replacement, hip replacement, shoulder replacement, or ankle replacement.

[0027] In some embodiments, the predetermined position of virtual surgical guide is determined to be used for implanting the tibial slope of one or more tibial implant assemblies in knee replacement.In some embodiments, the predetermined position of virtual surgical guide is determined to be used for implanting the tibial slope of one or more tibial implant assemblies in knee replacement.In some embodiments, the predetermined position of virtual surgical guide is determined to be varus or valgus correction angle of femur and / or tibial assembly in knee replacement.

[0028] In some embodiments, the virtual surgical guide corresponds to a physical distal femoral guide or cutting block, and the predetermined position of the virtual surgical guide determines femoral component flexion.

[0029] In some embodiments, the virtual surgical guide corresponds to a physical anterior or posterior femoral surgical guide or cutting block, and the predetermined position of the virtual surgical guide determines the femoral component rotation.

[0030] In some embodiments, the virtual surgical guide corresponds to a physical chamfer femoral guide or cutting block.

[0031] In some embodiments, the virtual surgical guide corresponds to a physical multi-cut femoral guide or cutting block, and the predetermined position of the virtual surgical guide determines one or more of anterior cuts, posterior cuts, chamfer cuts, and femoral component rotations.

[0032] In some embodiments, the virtual surgical guide is used for hip replacement, and the predetermined position of the virtual surgical guide determines the leg length after implantation.

[0033] In some embodiments, the virtual surgical guide is a virtual plane used to align a physical saw blade to guide joint bone cutting.

[0034] In some embodiments, the one or more intraoperative measurements include detecting one or more optical markers attached to a joint of the patient, the operating table, a fixed structure in the operating room, or a combination thereof. In some embodiments, one or more cameras or an image capture or video capture system in the optical head mounted display detects the one or more optical markers including their coordinates (x, y, z) and at least one or more of the position and orientation, alignment, direction of movement, or speed of movement of the one or more optical markers.

[0035] In some embodiments, spatial mapping techniques may be used to perform registration of one or more optical head-mounted displays, surgical sites, joints, spines, surgical instruments, or implant components.

[0036] In some embodiments, a depth sensor may be used to perform registration of one or more optical head-mounted displays, surgical sites, joints, spines, surgical instruments, or implant components.

[0037] In some embodiments, a virtual surgical guide is used to guide bone cutting of the distal tibia or talus in an ankle replacement, and one or more optical head-mounted displays are used to align the virtual surgical guide based on a predetermined tibial or talar implant component alignment, wherein the predetermined tibial or talar implant component alignment includes coronal plane implant component alignment, sagittal plane implant component alignment, axial plane component alignment, implant component rotation, or a combination thereof.

[0038] In some embodiments, a virtual surgical guide is used to guide bone cutting of the proximal humerus in shoulder replacement, and one or more optical head-mounted displays are used to align the virtual surgical guide based on a predetermined humeral implant component alignment, wherein the humeral implant component alignment includes coronal plane implant component alignment, sagittal plane implant component alignment, axial plane component alignment, humeral implant component rotation, or a combination thereof.

[0039] Aspects of the present invention relate to a system comprising at least one optical head-mounted display and a virtual library of implants, wherein the virtual library of implants comprises at least one virtual implant component, wherein the virtual implant component has at least one dimension, the at least one dimension corresponding to an implant component dimension or having a dimension substantially the same as a dimension of the implant component, wherein the at least one optical head-mounted display is configured to display the virtual implant component in substantial alignment with tissue in which the implant component is intended to be placed, wherein placement of the virtual implant component is intended to achieve a predetermined implant component position and / or orientation. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Embodiments of the illustrative, non-limiting examples will be more clearly understood from the following detailed description of the accompanying drawings.

[0041] Figure 1 Multiple optical head mounted displays are described for use with multiple observers, such as a primary surgeon, a secondary surgeon, a surgical assistant, and / or a nurse, according to certain embodiments of the present invention.

[0042] Figure 2 The workflow of segmentation and selection of subsequent steps according to certain embodiments of the present invention is described.

[0043] Figure 3 Certain embodiments according to the present invention are described for registering digital holograms for an initial surgical step, performing the surgical step, and re-registering one or more digital holograms for subsequent surgical steps.

[0044] Figure 4A , B and C are arbitrary virtual plane examples in the hip and femoral neck cutting planes according to certain embodiments of the present invention.

[0045] Figure 5 is an example illustrating an arbitrary imaginary plane in the knee extending through the medial and lateral joint spaces of the knee in accordance with certain embodiments of the present invention.

[0046] Figure 6 is an illustrative flow chart of various methods for resolving discrepancies between variations caused by surgical procedures and expected, predicted, or predetermined variations in patient virtual data, according to certain embodiments of the present invention.

[0047] Figure 7A -H is an example illustrating femoral neck cutting and corrective femoral neck cutting techniques according to certain embodiments of the present invention.

[0048] Figure 7A -H is an example illustrating femoral neck cutting and corrective femoral neck cutting techniques according to certain embodiments of the present invention.

[0049] Figure 8A-H is an example illustrating distal femoral cutting and corrective distal femoral cutting techniques according to certain embodiments of the present invention.

[0050] Figure 9A -G is an example illustrating distal femoral cutting and corrective distal femoral cutting techniques according to certain embodiments of the present invention.

[0051] Figure 10A -G is an example illustrating distal femoral cutting and proximal tibial cutting and correction cutting techniques according to certain embodiments of the present invention.

[0052] Figure 11 The present invention provides an example of how to generate a virtual surgical plan using intra-operative data, such as intra-operative measurements, obtained using one or more cameras and an image capture system or a video capture system that is integrated into, attached to, or separate from an optical head-mounted display, according to certain embodiments of the present invention.

[0053] Figure 12 is an exemplary workflow for generating a virtual surgical plan according to certain embodiments of the present invention.

[0054] Figure 13 The examples illustrate how a virtual surgical plan may be modified using intraoperative data, such as intraoperative measurements, according to certain embodiments of the present invention.

[0055] Figure 14 The examples illustrate how, according to certain embodiments of the present invention, multiple optical head-mounted displays can be used during a surgical procedure, for example by a first surgeon, a second surgeon, a surgical assistant, and / or one or more nurses, and how a surgical plan can be modified and displayed during a procedure using multiple optical head-mounted displays while maintaining the correct perspective of the virtual data and corresponding real-time data for each operator.

[0056] Figure 15 The examples illustrate how to use or apply 2D to 3D deformation data.

[0057] Figure 16A and B are flow charts outlining model generation, registration, and view projection for one or more optical head-mounted displays, such as for a primary surgeon, a second surgeon, a surgical assistant nurse, or others, according to certain embodiments of the present invention.

[0058] Figure 17A -D is an illustrative flow chart for selecting protocols and methods when performing spinal surgery in a mixed reality environment, according to certain embodiments of the present invention.

[0059] Figure 18A -F's example illustrates how, according to certain embodiments of the present invention, one or more optical head-mounted displays may be used to display a virtual acetabular reaming axis and align a physical acetabular reamer with the virtual reaming axis for placement of an acetabular cup with a predetermined cup angle, offset, medial or lateral position, and / or anteversion.

[0060] Figure 19A -D is a non-limiting illustrative example of how a virtual surgical guide (e.g., a distal femoral cutting block displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical distal femoral cutting block for a knee replacement) may be used in accordance with certain embodiments of the present invention.

[0061] Figure 20A -C is a non-limiting illustrative example of how a virtual surgical guide (e.g., an anteroposterior (AP) femoral cutting block displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical AP cutblock for a knee replacement) may be used in accordance with certain embodiments of the present invention.

[0062] Figure 21A -F's non-limiting illustrative examples illustrate how to use a virtual surgical guide (e.g., a virtual proximal tibial cutting guide displayed by an optical head-mounted display) and a physical surgical guide (e.g., a physical proximal tibial cutting guide according to certain embodiments of the present invention).

[0063] Figure 22A The anteroposterior and lateral views of A and B illustrate an exemplary normal anterior cruciate ligament (ACL) comprising anteromedial and posterolateral fibers.

[0064] Figure 22C The anteroposterior and lateral views of A and D illustrate exemplary anterior cruciate ligament tunnels (straight solid lines) on the femoral and tibial sides.

[0065] Figure 22E The frontal and lateral views of FIG and F illustrate exemplary virtual anterior cruciate ligament tunnels on the femoral and tibial sides (straight dashed lines), according to certain embodiments of the present invention.

[0066] Figure 22G The frontal and lateral views of H and H illustrate an exemplary virtual anterior cruciate ligament implant extending through the interarticular space between the femur and tibia (straight solid line) on the femoral and tibial sides, according to certain embodiments of the present invention.

[0067] Figure 23 The illustrative, non-limiting flow charts illustrate different methods for planning the location, position, orientation, alignment and / or direction of one or more femoral or tibial tunnels (e.g., for single-bundle or double-bundle techniques) or for placement of an anterior cruciate ligament graft, according to certain embodiments of the present invention. DETAILED DESCRIPTION

[0068] Various example embodiments will be described more fully below with reference to the accompanying drawings, some of which are illustrated in the accompanying drawings. However, the present invention can be implemented in many different forms and should not be construed as limited to the example embodiments set forth herein. Rather, these example embodiments are provided to make this disclosure thorough and complete and to fully convey the scope of the present invention to those skilled in the art. In the accompanying drawings, the sizes and relative sizes of layers and regions may be exaggerated for clarity. Like reference numerals represent like elements throughout.

[0069] As used herein, the term "real-time patient data" includes the patient's surgical site, anatomy, anatomical structures or tissues, and / or pathology, pathological structures or tissues as seen by the surgeon's or observer's eyes, without information from virtual data, virtual data, or stereoscopic views of imaging studies. The term "real-time patient data" does not include internal or subsurface tissues or structures or hidden tissues or structures that can only be viewed with the aid of a computer monitor or optical head-mounted display.

[0070] The terms "real surgical instruments," "actual surgical instruments," "physical surgical instruments," and "surgical instruments" are used interchangeably throughout this application; the terms "real surgical instruments," "actual surgical instruments," "physical surgical instruments," and "surgical instruments" do not include virtual surgical instruments. For example, a physical surgical instrument can be a surgical instrument provided by a manufacturer or supplier for spinal surgery, pedicle screw fixation, anterior spinal fusion, knee replacement, hip replacement, ankle replacement, and / or shoulder replacement; for example, a physical surgical instrument can be a cutting block, a guide rod, an awl, a reamer, an impactor, or a medullary nail. A physical surgical instrument can be reusable or disposable, or a combination thereof. A physical surgical instrument can be patient-specific. The term "virtual surgical instrument" does not include real surgical instruments, actual surgical instruments, physical surgical instruments, and surgical instruments.

[0071] The terms "real surgical tools," "actual surgical tools," "physical surgical tools," and "surgical tools" are used interchangeably throughout this application; the terms "real surgical tools," "actual surgical tools," "physical surgical tools," and "surgical tools" do not include virtual surgical tools. A physical surgical tool is a surgical tool provided by a manufacturer or supplier. For example, a physical surgical tool may be a staple, drill, saw blade, retractor, a frame for separating tissue, and other tools used in orthopedic, neurosurgery, urology, or cardiovascular surgery. The term "virtual surgical tool" does not include real surgical tools, "actual surgical tools," "physical surgical tools," and "surgical tools."

[0072] The terms real implant or implant component, actual implant or implant component, physical implant or implant component, and implant or implant component are used interchangeably throughout this application; the terms real implant or implant component, actual implant or implant component, physical implant or implant component, and implant or implant component do not include virtual implants or implant components. A physical implant or implant component can be an implant or implant component provided by a manufacturer or supplier. For example, a physical surgical implant can be a pedicle screw, a spinal fixation rod, a spinal fusion cage, a femoral or tibial component in a knee replacement, an acetabular cup or femoral stem and head in a hip replacement. The term "virtual implant" or "implant component" does not include real implants or implant components, actual implants or implant components, physical implants or implant components, and implants or implant components.

[0073] Using surgical navigation, a first virtual instrument can be displayed on a computer monitor that is an image of a physical instrument tracked with a navigation marker, such as an infrared or radio frequency marker, and the position and / or orientation of the first virtual instrument can be compared with the position and / or orientation of a corresponding second virtual instrument generated in the virtual surgical plan. Thus, through surgical navigation, the positions and / or orientations of the first and second virtual instruments are compared.

[0074] Various aspects of the present invention relate to apparatus, systems, and methods for locating a virtual path, virtual plane, virtual tool, virtual surgical instrument, or virtual implant component in a mixed reality environment using a head-mounted display device, optionally coupled to one or more processing units.

[0075] While being guided in a mixed reality environment, a virtual surgical guide, tool, instrument, or implant can be superimposed onto a physical joint, spine, or surgical site. Furthermore, the physical guide, tool, instrument, or implant can be aligned with the virtual surgical guide, tool, instrument, or implant displayed or projected on the optical head-mounted display. Thus, guidance in a mixed reality environment eliminates the need to use multiple virtual images of the guide, tool, instrument, or implant, and eliminates the need to compare the position and / or orientation of multiple virtual images of the virtual guide, tool, instrument, or implant.

[0076] In various embodiments, the optical head-mounted display can display one or more virtual surgical tools, virtual surgical instruments including a virtual surgical guide or a virtual cutting block, a virtual trial implant, a virtual implant assembly, a virtual implant or a virtual device, a predetermined starting point, a predetermined starting position, a predetermined starting orientation or alignment, a predetermined intermediate point, a predetermined intermediate position, a predetermined intermediate orientation or alignment, a predetermined endpoint, a predetermined ending position, a predetermined ending orientation or alignment, a predetermined path, a predetermined plane, a predetermined cutting plane, a predetermined profile or contour or cross-section or surface feature or shape or projection, a predetermined depth mark or depth gauge, a predetermined angle or orientation or rotation mark, a predetermined axis, such as a rotation axis, a bending axis, an extension axis, a virtual surgical tool, a virtual surgical instrument including a virtual surgical guide or a cutting block, a virtual trial implant, a virtual implant assembly, a predetermined axis of an implant or device, one or more more devices or implants or implant assemblies or surgical instruments or surgical tools, and / or an estimated or predetermined non-visualized portion of one or more predetermined tissue changes or alterations.

[0077] The position, location, orientation, alignment, direction, speed of movement, applied force, either virtual or physical, of a surgical instrument or tool may be predetermined using, for example, preoperative imaging, preoperative data, preoperative measurements, intraoperative imaging, intraoperative data and / or intraoperative measurements.

[0078] Any position, location, orientation, alignment, sagittal plane alignment, coronal plane alignment, axial plane alignment, rotation, implant pitch, implant angle, flexion, offset, anteversion, retroversion of an implant component; position, location, orientation, alignment relative to one or more anatomical landmarks; position, location, orientation, alignment relative to one or more anatomical planes; position, location, orientation, alignment relative to one or more anatomical axes; position, location, orientation, alignment relative to one or more biomechanical axes; position, location, orientation, alignment relative to a mechanical axis, virtual and / or physical, of a trial implant, implant component, or implant can be predetermined using, for example, preoperative imaging, surgical data, preoperative measurements, intraoperative imaging, intraoperative data, and / or intraoperative measurements. Intraoperative measurements may include measurements used for registration purposes of, for example, a joint, spine, surgical site, bone, cartilage, an optical head-mounted display, a surgical tool or instrument, a trial implant, an implant component, or an implant.

[0079] In some embodiments, multiple coordinate systems can be used instead of a common or shared coordinate system. In this case, coordinates can be transformed from one coordinate system to another, for example, to register an optical head-mounted display, real-time patient data including surgical sites, virtual instruments and / or virtual implants, and physical instruments and physical implants.

[0080] Optical head-mounted display

[0081] In some embodiments of the present invention, a pair of glasses is used. The glasses may include an optical head-mounted display. An optical head-mounted display (OHMD) may be a wearable display that has the ability to reflect projected images and allow the user to see through. Various types of optical head-mounted displays may be used to implement the present invention. These include curved mirror or curved combiner optical head-mounted displays and waveguide optical head-mounted displays or light-guide optical head-mounted displays. The optical head-mounted display may optionally use diffractive optics, holographic optics, polarization optics, and reflective optics.

[0082] Conventional input devices that can be used with optical head-mounted displays include, but are not limited to, touchpads or buttons, smartphone controllers, voice recognition, and gesture recognition. Advanced interfaces, such as brain-computer interfaces, are possible.

[0083] Optionally, the computer, server, or workstation can transmit data to the optical head-mounted display. Data transmission can be performed via cables, Bluetooth, WiFi, optical signals, or any other method or mode of data transmission known in the art. The optical head-mounted display can display virtual data, such as patient virtual data, in uncompressed or compressed form. The patient virtual data can optionally be reduced in resolution when transmitted to or displayed by the optical head-mounted display.

[0084] When the virtual data are transmitted to the optical head-mounted display, they may exist in a compressed form during transmission. The optical head-mounted display may then choose to decompress them so that the optical head-mounted display displays the uncompressed virtual data.

[0085] Alternatively, when the virtual data are sent to the optical head-mounted display, they can be reduced in resolution during transmission, for example, by increasing the slice thickness of the image data before transmission. The optical head-mounted display can then selectively increase the resolution, for example, by re-interpolating to the original slice thickness of the image data or even a thinner slice thickness, so that the resolution of the virtual data is equal to or greater than the original virtual data or at least higher than the resolution of the transmitted data displayed by the optical head-mounted display.

[0086] In some embodiments, the optical head mounted display can send data back to the computer, server, or workstation. Such data may include, but is not limited to:

[0087] - Information about the position, location or orientation of the optical head-mounted display or a person or surgeon wearing the optical head-mounted display

[0088] - Changes in the position, location, or orientation of the optical head-mounted display

[0089] -Data generated by one or more inertial measurement units (IMUs)

[0090] - Data from markers (RF, optical, light, other) connected to, integrated into, or coupled to optical head-mounted displays

[0091] -Data generated by a surgical navigation system connected to, integrated with, or coupled to an optical head-mounted display

[0092] - Data generated by an image and / or video capture system connected to, integrated with, or coupled to the optical head-mounted display

[0093] - Parallax data, e.g., using two or more image and / or video capture systems connected to, integrated with, or coupled to the optical head-mounted display, e.g., a first image and / or video capture system positioned above, below, or near the left eye and a second positioned above, below, or near the right eye

[0094] - Distance data, such as disparity data generated by two or more image and / or video capture systems, to assess changes in the distance between the optical head-mounted display and the surgical field or subject

[0095] -Motion parallax data

[0096] - Data related to the calibration or registration phantom (see other sections of this manual)

[0097] - Any type of real-time data of the patient captured by an optical head-mounted display, including image and / or video capture systems connected, integrated, or coupled to the optical head-mounted display

[0098] ○For example, changing the surgical site

[0099] o For example, using certain surgical instruments detected by image and / or video capture systems

[0100] ○ For example, the use of certain medical devices or experimental implants detected by image and / or video capture systems

[0101] - Any type of modification to the surgical plan

[0102] ○Partial or multi-faceted on-site surgical planning

[0103] ○ Partial or multi-faceted virtual surgical planning

[0104] The radio frequency tags used throughout the embodiments may be of active or passive type, with or without batteries.

[0105] Exemplary optical head-mounted displays include the ODG R-7, R-8, and R-8 Smart Glasses from ODG (Osterhout Group, San Francisco, CA), the NVIDIA 9423-D Vision Wireless Glasses (NVIDIA, Santa Clara, CA), and the Microsoft HoloLens (Microsoft, Redmond, WI).

[0106] Microsoft HoloLens is manufactured by Microsoft. It is a pair of augmented reality smart glasses. It runs on the Windows 10 operating system. The front of the HoloLens includes sensors, related hardware, several cameras, and a processor. The visor includes a pair of transparent, beam-combining lenses for displaying the projected image. HoloLens can be adjusted for interpupillary distance (IPD) using an integrated program that recognizes gestures. A pair of speakers are also integrated. The speakers do not reject external sounds and allow users to hear virtual sounds. A USB 2.0 micro-B socket and a 3.5mm headphone jack are integrated.

[0107] The Microsoft HoloLens has an inertial measurement unit (IMU) with an accelerometer, gyroscope, and a magnetometer, four environment mapping sensors / cameras (two on each side), a depth camera with a 120°×120° field of view, a 2.4-megapixel camera, a four-microphone array, and an ambient light sensor.

[0108] The Microsoft Holographic Glasses have an Intel Cherry Trail SoC that includes a CPU and GPU. The Microsoft Holographic Glasses also include a custom Microsoft Holographic Processing Unit (HPU). The SoC and HPU each have 1GB of LPDDR3 and share 8MB of SRAM. The SoC also controls 64GB of eMMC and runs the Windows 10 operating system. The HPU processes and integrates data from sensors, as well as handles tasks such as spatial mapping, gesture recognition, and voice and speech recognition. The Microsoft Holographic Glasses include IEEE 802.11ac Wi-Fi and Bluetooth 4.1 Low Energy (LE) wireless connectivity. The headset uses Bluetooth LE and can be connected to a Clicker, which is a finger-operated input device that can be used to select menus and functions.

[0109] Many apps are available for Microsoft HoloLens, such as the Hologram Catalog, HoloStudio, a 3D modeling app with Microsoft 3D Print capabilities, Autodesk Maya 3D creation app, FreeForm, which connects Microsoft HoloLens to Autodesk Fusion 360 cloud-based 3D development app, and others.

[0110] Microsoft HoloLens, powered by the HPU, can use sensory and natural interface commands—voice, gestures, and hand gestures. Gaze commands, like head tracking, allow users to direct applications to focus on whatever they perceive. Air tapping can be used to select any virtual application or button, similar to clicking a virtual computer mouse. Holding a tap simulates dragging, moving the display. Voice commands are also available.

[0111] The Microsoft HoloLens shell incorporates many components and concepts from the Windows desktop environment. The bloom gesture, used to open the main menu, is performed by opening one hand with the palm facing up and fingers extended. Windows can be dragged to a specific location, locked, and / or resized. Virtual windows or menus can be pinned to multiple locations or physical objects. Virtual windows or menus can move with the user or be pinned to the user. Alternatively, they can follow the user as they move.

[0112] Developers can use the Microsoft HoloLens app for Windows 10 PCs and Windows 10 Mobile devices to run applications and view live streams from the Microsoft HoloLens user's perspective, as well as capture augmented reality photos and videos.

[0113] Almost all Universal Windows Platform apps can run on Microsoft HoloLens. These apps can project content in 2D. Microsoft HoloLens currently supports select Windows 10 APIs. Microsoft HoloLens apps can also be developed on Windows 10 PCs. Holographic apps can use Windows Holographic APIs. Unity and Vuforia are examples of supported apps. Apps can also be developed using DirectX and Windows APIs.

[0114] Computer image observation approach

[0115] In some embodiments of the present invention, the optical head-mounted display uses a computer graphics viewing approach that includes the following steps to display three-dimensional objects or two-dimensional objects or other computer-generated objects and models located in three-dimensional space.

[0116] Figure 16B :

[0117] 1. Registration

[0118] 2. View the projection

[0119] Registration:

[0120] The various objects to be displayed in an optical head-mounted display computer graphics system (e.g., virtual anatomical models, virtual instrument models, geometric and surgical references, and guides) are initially defined in their own independent model coordinate systems. During the registration process, the spatial relationships between these various objects are defined, and each object is transformed from its own model coordinate system into a common global coordinate system. Various techniques, as described below, can be applied to this registration process.

[0121] For augmented reality optical head-mounted displays that overlay computer-generated objects with a real-time view of the physical environment, the overall coordinate system is defined by the environment. A process called spatial mapping, described below, creates a computer image of the environment that can be merged and registered with the computer-generated objects, thereby defining the spatial relationship between the computer-generated objects and the physical environment.

[0122] View projection:

[0123] Once all objects to be displayed have been registered and transformed into the same global coordinate system, they are prepared for viewing on the display by transforming the coordinates of the displayed objects from the global coordinate system into the view coordinate system and then projecting them onto the display plane. This view projection step uses the viewpoint and view direction to define the transformation applied in this step. For stereoscopic displays such as optical head-mounted displays, two different view projections can be used, one for the left eye and the other for the right eye. For augmented reality optical head-mounted displays, the position of the viewpoint and view direction relative to the physical environment can be known so that computer-generated objects can be correctly superimposed with the physical environment. When the viewpoint and view direction change, for example due to head movement, the view projection is updated so that the computer-generated display follows the new view changes.

[0124] Eye tracking system

[0125] The present invention provides methods for using the human eye, including eye movements and eyelid movements and movements caused by the muscles around the eye sockets, to execute computer commands. The present invention also provides methods for executing computer commands through facial movements and head movements.

[0126] The execution of commands caused by eye and eyelid movements, as well as movements caused by the muscles around the eye sockets, facial movements, and head movements, is most advantageous in environments where the operator is unable to use their hands to type on a keyboard or execute commands on a touchpad or other handheld computer interface. These situations include, but are not limited to, industrial applications, specifically automotive and aircraft manufacturing, chip manufacturing, medical or surgical procedures, and many other potential applications.

[0127] In some embodiments, the optical head-mounted display may include an eye-tracking system. Different types of eye-tracking systems may be used. The embodiments provided below are in no way to be considered as limiting the present invention. Any eye-tracking system currently known in the art may be used.

[0128] Eye movements can be divided into fixations and saccades—that is, when the eyes temporarily fixate on a certain location and when they move to another location, respectively. The resulting series of fixations and saccades can be defined as a scanpath. The center of the visual field, within one or two degrees, provides the majority of visual information; input from the periphery is less abundant. Therefore, the location of fixations along the scanpath describes the location of information processed during an eye-tracking session, for example, during surgery.

[0129] Eye trackers can measure eye rotation or movement in a variety of ways, such as by measuring the movement of an object attached to the eye (e.g., in the form of a contact lens), optical tracking without direct contact with the eye, and measuring the electrical potential of electrodes placed around the eye.

[0130] If an eye attachment is used, it can be a special contact lens with an embedded mirror or magnetic field sensor. The attachment's movement can be measured, assuming it does not significantly slip when the eye rotates. Measuring tightly fitting contact lenses can provide very accurate eye movement measurements. Alternatively, a magnetic search coil can be used to measure horizontal, vertical, and torsional eye movement.

[0131] Alternatively, non-contact optical methods of measuring eye movements can be used. The eye can reflect light, optionally infrared; with this technology, this can be sensed by an optical sensor or a camera. Information is then measured to infer rotation and / or movement of the eye from changes in the reflection. An optical sensor or video-based eye tracker can use the corneal reflection (the so-called first Purkinje image) and the pupil center as tracking features, optionally changing over time. A more sensitive eye tracker, the dual Purkinje eye tracker, uses reflections from the front of the cornea (the first Purkinje image) and the back of the lens (the fourth Purkinje image) as tracking features. A more sensitive tracking method is to image visual features (such as retinal blood vessels) from inside the eye and track these features as the eye rotates and / or moves. Optical methods, in particular methods based on optical sensors or video registration, can be used for gaze tracking.

[0132] In some embodiments, an optical or video-based eye tracker can be used. A camera focuses on one or both eyes and tracks their movement while the observer performs a function, such as surgery. The eye tracker can use the center of the pupil for tracking. Infrared or near-infrared non-collimated light can be used to generate the corneal reflection. The vector between the pupil center and the corneal reflection can be used to calculate the point of focus on the surface or gaze direction. A calibration procedure can optionally be performed at the beginning of eye tracking.

[0133] Both bright pupil and dark pupil tracking are possible. The difference lies in the position of the illumination source relative to the optics. If the illumination is coaxial with the optical path, the eye behaves retroreflectively, as light reflects off the retina, creating a bright pupil effect similar to red eye. If the illumination source is off-path, the pupil appears dark, as the retroreflection from the retina is directed away from the optical sensor or camera.

[0134] Bright pupil tracking allows for greater iris / pupil contrast, making eye tracking more reliable using iris pigmentation. It also reduces interference caused by eyelashes. Tracking is possible in both dark and very bright lighting conditions.

[0135] As described above, optical tracking methods can include tracking eye movements including the pupil. Optical tracking methods can also include tracking movements of the eyelids and periorbital and facial muscles.

[0136] In some embodiments, the eye tracking device is integrated into the optical head mounted display.In some embodiments, head movements can be tracked simultaneously, for example, using an inertial measurement unit formed by a combination of an accelerometer and a gyroscope (see below).

[0137] In some embodiments, electrical potential can be measured using electrodes placed around the eye. The eye generates an electrical potential field that can be detected even if the eye is closed. This field can be modeled as being generated by a dipole, with a positive pole at the cornea and a negative pole at the retina. This can be measured by placing two electrodes on the skin around the eye. The electrical potential measured in this manner is called an electrooculogram.

[0138] If the eye moves from a central position toward the periphery, the retina approaches one electrode, while the cornea approaches the opposite electrode. The change in dipole orientation and the resulting change in the electric potential field result in changes in the measured electrooculogram (EOG) signal. These changes can be analyzed to assess eye movements. Two separate directions of movement can be identified: horizontal and vertical. If a posterior skull electrode is used, a radial EOG component can be measured. This is typically referenced to the average of the EOG channels of the posterior skull electrode. The radial EOG channel measures the saccadic spikes originating from the extraocular muscles at the onset of a saccade.

[0139] The use of EOG can be limited to measuring slow eye movements and detecting gaze direction. However, EOG is well suited to measuring rapid or saccadic eye movements associated with gaze shifts and detecting blinks. Unlike optical or video-based eye trackers, EOG can register eye movements even with the eyes closed. The main disadvantage of EOG is its relatively poor gaze direction accuracy compared to optical or video trackers. Optionally, both methods, optical or video tracking and EOG, can be used in combination in selected embodiments of the present invention.

[0140] Sampling rates of 15, 20, 25, 30, 50, 60, 100, 120, 240, 250, 500, 1000 Hz or higher can be used. Any sampling frequency is possible. In many embodiments, a sampling rate greater than 30 Hz is preferred.

[0141] Measure position, orientation, acceleration

[0142] The position, orientation, and acceleration of the human head, various human parts (such as hands, arms, legs, or feet), various parts of the patient's body (such as the patient's head or limbs, including hips, knees, ankles, feet, shoulders, elbows, hands, or wrists), and any other body parts can be measured by combining gyroscopes and accelerometers. In selected applications, magnetometers can also be used. Such a measurement system using any of these components can be defined as an inertial measurement unit (IMU).

[0143] As used herein, the term inertial measurement unit (IMU) relates to electronic devices that use a combination of accelerometers and gyroscopes, and optionally a magnetometer, to measure and transmit information about body-specific forces, angular rates, and, optionally, the magnetic field surrounding the body. The IMU or its components can be coupled or registered with a navigation system or robot, for example, by registering the body or body parts within a shared coordinate system. Optionally, the IMU can be wireless, for example, using a WiFi network or a Bluetooth network.

[0144] Pairs of accelerometers spread over a region of space can be used to detect differences (gradients) in the appropriate accelerations of the reference frames associated with those points.

[0145] Single-axis and multi-axis accelerometer models can be used to detect the magnitude and direction of acceleration as a vector quantity and can be used to detect orientation (due to weight changes), coordinate acceleration (as long as it produces gravity or changes in gravity), vibration, shock. In some embodiments, micromachined accelerometers can be used to detect the position of the device or the operator's head.

[0146] Piezoelectric, piezoresistive, and capacitive devices can be used to convert mechanical motion into electrical signals. Piezoelectric accelerometers rely on piezoelectric ceramics, or single-crystal piezoresistive accelerometers can also be used. Capacitive accelerometers typically use silicon micromachined sensing elements.

[0147] In some embodiments, the accelerometer used may include a miniature micro-electromechanical system (MEMS), including, for example, a cantilever beam with a proof mass.

[0148] Optionally, an accelerometer may be integrated into the optical head-mounted device, and output from both the eye tracking system and the accelerometer may be used for command execution.

[0149] The following exemplary information about the operator and patient and various body parts can be captured by the inertial measurement unit: rate, velocity, acceleration, spatial location, position change, alignment, direction and / or direction of movement (e.g., by continuous measurement)

[0150] Parts of the operator's and / or patient's body that utilize an inertial measurement unit to transmit such information include, but are not limited to, the head, chest, torso, shoulders, elbows, wrists, hands, fingers, arms, hips, knees, ankles, feet, toes, legs, and internal organs such as the brain, heart, lungs, liver, spleen, intestines, and bladder.

[0151] Any number of inertial measurement units can be placed on the optical head-mounted display, the operator, and / or the patient, and optionally, these inertial measurement units can be referenced to each other in a single or multiple coordinate systems, or, optionally, they can be referenced to each other in relation to the optical head-mounted display, a second and third or more optical head-mounted displays, a navigation system or robot, and one or more coordinate systems used by such navigation system and / or robot. The navigation system can be used in conjunction with the optical head-mounted display without the use of inertial measurement units. For example, navigation markers including infrared markers, retroreflective markers, and radiofrequency markers can be attached to the optical head-mounted display and, optionally, to various parts or segments of the patient or the patient's anatomy. In this manner, the optical head-mounted display and the patient or the patient's anatomy can be referenced to each other or registered in one or more coordinate systems used by the navigation system, and the movement of the optical head-mounted display or the operator wearing the optical head-mounted display can be registered with respect to the patient in one or more coordinate systems. Once the virtual data and real-time data of the patient and the optical head-mounted display are aligned in the same coordinate system, for example, using an inertial measurement unit, optical markers, navigation markers, including infrared markers, retroreflective markers, radio frequency markers, and any other alignment method described in the specification or known in the art, any change in the position of the optical head-mounted display associated with the patient measured in this manner can be used to change the relationship between the patient's virtual data and the patient's real-time data, so that the visual image of the patient's virtual data and the patient's real-time data seen through the optical head-mounted display are always aligned regardless of how the optical head-mounted display and / or the operator's head and / or the operator wearing the optical head-mounted display moves. Similarly, when multiple optical head-mounted displays are used, such as one for the primary surgeon and several, such as two, three, four, or more, optical head-mounted displays worn by other surgeons, assistants, residents, fellows, nurses, and / or visitors, or other staff other than the primary surgeon, the optical head-mounted displays can also display a virtual image of the patient's virtual data that is consistent with the corresponding real-time data of the patient as viewed through the optical head-mounted displays, wherein the perspective of the virtual data of the patient and / or surgical site is used for the location, position, and / or orientation of the observer using the optical head-mounted displays and the respective observer's eyes. The foregoing embodiments are possible because the inertial measurement unit, optical markers, radio frequency markers, infrared markers, and / or navigation markers placed on the surgeon and / or patient, as well as any spatially located markers, can be registered in the same coordinate system of the primary optical head-mounted display and any other optical head-mounted displays.The position, orientation, alignment, and changes in position, orientation, and alignment of the patient and / or surgical site of the remaining optical head-mounted displays can be individually monitored to align and / or overlay corresponding structures of the patient's real-time data and the patient's virtual data of the additional optical head-mounted displays, regardless of the position, orientation, and / or alignment of the patient and / or surgical site.

[0152] refer to Figure 1 The system 10 is configured for multiple observers using multiple optical head-mounted displays 11, 12, 13, 14, such as a primary surgeon, a secondary surgeon, a surgical assistant, and / or a nurse. The multiple optical head-mounted displays can be registered in a common coordinate system 15 using anatomical structures, anatomical landmarks, calibration phantoms, reference phantoms, optical markers, navigational markers, and / or spatial anchors, such as those used by Microsoft HoloLens. Preoperative patient data 16 can also be registered in the common coordinate system 15. For example, real-time patient data 18, such as from the surgical site, the spine (optionally with minimally invasive access), a hip arthrotomy site, a knee arthrotomy site, a bone cut, or an altered surface, can be measured using one or more inertial measurement units, optical markers, navigational markers, image or video capture systems, and / or spatial anchors. The real-time patient data 18 can be registered in the common coordinate system 15. Intraoperative imaging findings 20 can also be registered in the common coordinate system 15. The operating room table or room fixtures may be registered in a common coordinate system 15 using, for example, an optical marker inertial measurement unit, navigation markers, or a spatial map 22. Preoperative data 16 or real-time data 18 including intraoperative measurements, or a combination thereof, may be used to generate, create, or modify a virtual surgical plan 24. The virtual surgical plan 24 may be registered in the common coordinate system 15. The optical head mounted display 11, 12, 13, 14 may project virtual data or a digital hologram of virtual data into a left eye view using the view position and orientation of the left eye 26, and may project virtual data or a digital hologram of virtual data into a right eye view using the view position and orientation of the user's right eye 28, thereby generating a shared digital holographic experience 30. Using a virtual or other interface, a surgeon wearing the optical head-mounted display 111 can execute a command 32, such as to display the next scheduled bone cut, such as from a virtual surgical plan or imaging examination or intraoperative measurement, which can trigger the optical head-mounted display 11, 12, 13, 14 to project a digital hologram of the next surgical step 34 superimposed on the surgical site and aligned with the surgical site to a predetermined position and / or orientation.

[0153] By registering the real-time patient data, such as the surgical field, the patient virtual data, and each optical head-mounted display in a shared coordinate system, the patient virtual data can be projected and superimposed on the real-time patient data of each observer using each optical head-mounted display from their respective perspectives. Thus, the patient virtual data, including aspects of the virtual surgical plan, can remain superimposed and / or aligned with the real-time patient data regardless of the observer's perspective, and can remain aligned and / or superimposed as the observer moves their head or body.

[0154] Novel user interface

[0155] A subject of the present invention is to provide a novel user interface in which human eye, including eyeball movements, and eyelid movements, including movements caused by the orbit and peri-orbital and selected cranial muscles, are detected by an eye tracking system and processed to execute predefined, actionable computer commands.

[0156] An exemplary list of eye movements and eyelid movements that can be detected by the system is provided in Table 1.

[0157] Table 1: Example list of eye movements and eyelid movements detected by eye tracking software

[0158] ○1 blink

[0159] ○ Blink twice

[0160] ○3 blinks

[0161] ○ Blinking quickly, for example, less than 0.5 seconds

[0162] Slow blinking, for example, longer than 1.0 second

[0163] ○ Two or more blinks spaced quickly apart, for example, less than 1 second

[0164] ○ Two or more blinks separated by a long time interval, e.g., more than 2 seconds (usually chosen to be less than the natural time interval between blinks)

[0165] Blink only the left eye

[0166] Blink only the right eye

[0167] ○ Blink your left and right eyes simultaneously

[0168] ○ Blink the left eye first, then within a short interval (e.g., less than 1 second), blink the right eye

[0169] ○ Blink your right eye first, then within a short interval (e.g., less than 1 second), blink your left eye

[0170] ○ Blink the left eye first, then after a long interval (e.g., more than 2 seconds), blink the right eye

[0171] Blink the right eye first, then after a long interval (e.g., more than 2 seconds), blink the left eye

[0172] ○ Rapid eye movement to the left

[0173] ○ Rapid eye movement to the right

[0174] ○ Rapid upward eye movement

[0175] Rapid downward eye movement

[0176] ○ Keep your eyes open for short intervals, for example, less than 1 second

[0177] ○ Keep your eyes open for a long time, for example, more than 2 seconds

[0178] ○ Close your eyes for 1 second

[0179] ○ Close your eyes for 2 seconds or more

[0180] Close your eyes, hold them, then open them and blink rapidly

[0181] ○ Close your left eye for only 1 or 2 seconds

[0182] ○ Close your right eye for only 1 or 2 seconds

[0183] ○ Close your left eye, then your right eye

[0184] ○ Close your right eye, then your left eye

[0185] ○ Blink left eye, then right eye

[0186] ○ Blink right eye, then left eye

[0187] ○ Stare at the live, virtual button for 1, 2, 3 or more seconds; activate a function, such as zooming in or out

[0188] Any combination of blinks, eye movements, sequences, and time intervals can be used to encode various types of commands—for example, computer commands to guide or manipulate surgical instruments or robots.

[0189] The present invention also provides a method for executing commands through facial movements and head movements.

[0190] An exemplary list of facial and head movements that may be detected by the system is provided in Table 2. (This list is merely an example and is by no means exhaustive; any number or combination of movements is possible).

[0191] Table 2: Example list of detected facial and head movements:

[0192] ○Move your head quickly to the right and hold

[0193] ○Move your head quickly to the left and hold

[0194] ○Move your head quickly downward and hold

[0195] ○Move your head quickly downward and hold

[0196] ○ Move your head quickly to the right and back

[0197] ○ Move your head quickly to the left and back

[0198] ○ Move your head quickly downward and backward

[0199] ○ Move your head quickly downward and backward

[0200] ○Tilt your head to the left and hold

[0201] ○Tilt your head to the right and hold

[0202] Tilt your head to the left and back

[0203] Tilt your head to the right and back

[0204] ○ Open your mouth and hold

[0205] ○ Open and close your mouth

[0206] ○Pinch your nose once

[0207] ○Twist your nose twice

[0208] Exemplary commands executed using eye movements, eyelid movements, facial movements, and head movements are listed in Table 3.

[0209] Table 3: List of exemplary command lines that may be executed by tracking eye movements, eyelid movements, facial movements, and head movements (this list is merely an example and is by no means intended to be exhaustive; any number or combination of commands is possible; special purpose commands may also be executed in this manner).

[0210] ○ Click

[0211] ○point

[0212] ○Move pointer

[0213] ■Slow

[0214] ■ Fast

[0215] Scroll, for example, through images

[0216] ■ Fast scrolling

[0217] ■Slow scrolling

[0218] Scroll up

[0219] Scroll down

[0220] ○Scroll left

[0221] Scroll right

[0222] ○Drag

[0223] ○ Rotation

[0224] ○Registration

[0225] ○2D and 3D switching

[0226] ○Switch imaging examination

[0227] ○Superimposed images

[0228] ○ Fusion image

[0229] ○Register images

[0230] ○Cut

[0231] ○Paste

[0232] ○Copy

[0233] ○Cancel

[0234] Rework

[0235] ○Delete

[0236] ○Purchasing

[0237] ○Provide credit card information

[0238] ○ Authorization

[0239] ○Transfer to shopping card

[0240] ○ Optical head-mounted display opens

[0241] Optical head-mounted display off

[0242] Eye tracking is on

[0243] Eye tracking off

[0244] ○Execute eye command to open

[0245] ○Execute eye command to close

[0246] ○Execute facial commands to turn on

[0247] ○Execute facial command to close

[0248] ○ Turn on surgical instruments (e.g., oscillating saw, laser, etc.)

[0249] ○ Close surgical instruments

[0250] ○ Increase the strength, speed, and energy of surgical instruments

[0251] ○ Reduce the strength, speed, and energy of surgical instruments

[0252] ○Change the direction of surgical instruments

[0253] ○Change the orientation of surgical instruments

[0254] ○Change any type of setting of surgical instruments

[0255] In some embodiments of the present invention, eye movement, eyelid movement, facial movement, head movement, alone or in combination, can be used as a signal digital code or digital sequence or machine operation sequence. For example, such a digital sequence can be used to execute certain machine operation sequences.

[0256] Head movements control the movement of surgical instruments

[0257] In some embodiments of the present invention, head movements can be used to control surgical instruments. For example, in robotic-assisted procedures with robotic tactile feedback, a surgeon can manually control the direction of a surgical instrument. The surgeon can move their head forward. This forward motion is captured by an inertial measurement unit (IMU) and translated into forward movement of a robotic arm, which steers the surgical instrument in its direction. Backward movement of the head can also be captured by the IMU and translated into backward movement of the robotic arm, which steers the surgical instrument in its direction.

[0258] In some embodiments of the present invention, eye movements, eyelid movements, facial movements, head movements, or a combination thereof can be used to send Morse code signals. International Morse code encodes the Latin alphabet using a small set of punctuation marks and program signals as standardized sequences of short and long signals called dots and dashes. Each character (letter or number) is represented by a single sequence of dots and dashes. The duration of a dash is three times the duration of a dot. Each dot or dash is followed by a brief silence equal to the duration of the dot. Letters of a word are separated by a space equal to three dots (a dash), and words are separated by a space equal to seven dots.

[0259] An example of executing Morse code using eye commands is as follows; this is in no way meant to be limiting. Many different implementations are possible. For example, a dot can be executed using a quick blink of both eyes (typically less than 1 second), while a line can be executed by closing only the right eye (for example, 1 second). The letter A in Morse code is a dot followed by a line. Using this encoding of Morse code, the letter A can be executed with a quick blink of both eyes (the dot) and then closing the right eye for just one second (the line). The letter B (the line, three dots) can be executed by closing the right eye for just one second (the line) and then blinking three times quickly (the three dots), etc. For example, the letters can be separated by maintaining a gap of two seconds or more between eye commands. Or, in another example, the letters can be separated by closing only the left eye for about one second.

[0260] Binary codes can also be implemented using eye commands. For example, rapidly blinking both eyes could represent the number 0, while closing only the right eye for approximately one second could represent the number 1. Alternatively, closing only the right eye for approximately one second could represent the number 0, while closing only the left eye for approximately one second could represent the number 1. Many different types of encoding are possible. Other codes can also be implemented using some of the eye, eyelid, facial, and / or head movements shown in Tables 1 and 2.

[0261] Many different languages ​​can be executed in this manner. Also optionally computer languages ​​such as Fortran, Pascal, C, C++, C--, Basic and many other computer languages ​​known in the art are also possible.

[0262] In some embodiments of the present invention, eye, eyelid, facial, and head movement commands can be paired or used in conjunction with voice commands, hand commands, gesture commands, keyboard commands, trackpad commands, mouse commands, graphical user interface commands, and any other command input device known in the art. The optical head-mounted display may also optionally include one or more touch-sensitive sensors.

[0263] Another benefit of eye command in select environments is the ability to maintain privacy or confidentiality while navigating screens or executing commands. For example, in a hospital setting, eye command can be used to review patient records or order lab or other diagnostic tests with other patients or visitors nearby, without bystanders being aware that these registrations are being reviewed or tests are being ordered.

[0264] During a meeting, the wearer of the optical head-mounted display can use eye commands to turn on the video or audio registration function or transmit to a remote site or remote conference room without revealing that the registration function has been activated. This is in contrast to manually initiating the registration function, such as when the user presses a button or touch-sensitive sensor on the optical head-mounted display to activate the registration function.

[0265] In some embodiments, a user can use eye movements, facial movements, or head movements to direct a digital camera to take a photo or video. Commands may include, but are not limited to, zooming in, zooming out, moving a target area left, right, up, or down, taking a photo, taking a series of photos, turning the flash on / off, starting video recording, stopping video recording, changing resolution, increasing resolution, or decreasing resolution.

[0266] Any other camera command known in the art may be executed in this manner using eye movement-based, facial movement-based, or head movement-based commands. By utilizing one or more commands of this type, a user may maintain privacy while obtaining image information about their surroundings.

[0267] Eye commands are very useful for surgeons or operating room personnel to execute commands without using their hands, thereby maintaining sterile conditions.

[0268] Fusing the physical world with patient images and other data

[0269] In some embodiments of the present invention, an operator such as a surgeon can observe the patient's physical data or information, such as the surgical site or changes caused at the surgical site, through an optical head-mounted display, and at the same time, pre-existing patient data is superimposed on the physical visual image of the on-site patient.

[0270] The patient's pre-existing data may be imaging or picture data or other types of data including metabolic or functional information.

[0271] Pre-existing patient data can be obtained outside of the time of surgery, including one or more imaging studies, or other types of data including metabolic or functional information. For example, pre-existing patient data can be obtained one, two, three, or more days or weeks prior to surgery.

[0272] Pre-existing patient data, including one or more imaging studies, or other types of data including metabolic or functional information, are typically obtained from a patient or surgical site located at a different location or in a different object coordinate system than the pre-existing data when compared to a location or object coordinate system of a live patient or surgical site in a live patient. Thus, the pre-existing data of the patient or surgical site are typically located in a first object coordinate system, and the real-time data of the patient or surgical site are typically located in a second object coordinate system; the first and second object coordinate systems are typically different from each other. The first object coordinate system with the pre-existing data needs to be registered with the second object coordinate system for the real-time patient data, e.g., the live surgical site.

[0273] Scanning technology

[0274] The following is an exemplary list of scanning and imaging technologies that can be used or applied to various aspects of the present invention; this list is not exhaustive and is exemplary only. Those skilled in the art will recognize other scanning or imaging technologies that can be used to implement the present invention.

[0275] For a detailed description of these various scanning and imaging techniques, see, for example, Bushberg et al., Basic Physics of Medical Imaging, 3rd ed., Wolters, Kluwer, Lippincott, 2012.

[0276] - X-ray imaging, 2D, 3D, supine, upright or other body positions and postures, including analog and digital X-ray imaging

[0277] -Digital Tomosynthesis

[0278] - Cone-beam computed tomography (CT)

[0279] -Ultrasound

[0280] - Doppler ultrasound

[0281] - Elastography, for example, using ultrasound or MRI

[0282] -Computed tomography

[0283] -Magnetic resonance imaging (MRI)

[0284] ○ Includes, for example, functional MRI, diffusion imaging, stroke imaging, MRI with contrast agents

[0285] - Functional MRI (fMRI), for example, brain imaging and functional brain mapping

[0286] -Magnetic resonance spectroscopy

[0287] -Positron emission tomography (PET)

[0288] -Single Photon Emission Tomography-Computed Tomography (SPECT-CT)

[0289] -Positron emission tomography-computed tomography (PET-CT)

[0290] -Positron emission tomography-magnetic resonance imaging (PET-MRI)

[0291] - Scanning vertically, optionally in multiple planes or in three dimensions, using any of the aforementioned modalities (including X-ray imaging, ultrasound, etc.).

[0292] -Contrast agents

[0293] o For example, iodinated contrast agents used in X-ray and computed tomography scans, or MRI contrast agents.

[0294] ○ Contrast agents may include antigens or antibodies for specific targeting of cells or tissues

[0295] ○ Other targeted technologies, such as the use of liposomes, can also be applied

[0296] ○Molecular imaging

[0297] ■ To highlight metabolic abnormalities in the brain and direct surgical instruments to areas of metabolic abnormalities

[0298] o Any contrast agent known in the art may be used in conjunction with the present invention.

[0299] Multidimensional imaging, reconstruction and visualization

[0300] Various embodiments of the present invention can be implemented in one, two, three, or more dimensions. The following is an exemplary list of potential dimensions, views, projections, angles, or reconstructions that may be applied; this list is not exhaustive and is merely exemplary. Those skilled in the art may identify additional dimensions, views, projections, angles, or reconstructions useful in implementing the present invention. Exemplary dimensions are listed in Table 4.

[0301] Table 4: Example list of potential sizes, views, projections, angles, or reconstructions of virtual images that can be displayed using an optical head-mounted display, optionally stereoscopic

[0302] The first dimension: up and down, for example, the patient's physical data

[0303] Second dimension: medial and lateral, e.g., the patient's physical data

[0304] The third dimension: anteroposterior, for example, the patient's physical data

[0305] 4th-6th Dimensions: 1, 2, or 3-D head movement (and eyewear movement / optical head-mounted displays)

[0306] Dimensions 7-9: 1-, 2-, or 3-dimensional motion of the instrument, e.g., relative to the surgical field, organs, or head, including head motion

[0307] 10th-13th dimensions: 1, 2, or 3-dimensional arm or hand movements, e.g., related to the surgical field, organs, or head, including head movements

[0308] 14th-16th dimensions: Virtual 3D data of the patient, e.g., from scans or intraoperative measurements

[0309] Dimensions 17-19: Vascular flow; in 1, 2, or 3 dimensions, e.g., related to the surgical field, organs, or head, including head motion

[0310] 20th-22nd Dimensions: Temperature maps (including changes caused by low or high temperatures), thermal imaging, in 1, 2 or 3 dimensions, e.g., related to the surgical field

[0311] Dimensions 25-28: Metabolic mapping (e.g., using MRS, positron emission tomography-computed tomography, single photon emission tomography-computed tomography), in 1, 2, or 3 dimensions, e.g., related to the surgical field

[0312] Dimensions 29-32: Functional mapping (e.g., using functional magnetic resonance imaging, positron emission tomography-computed tomography, single photon emission tomography-computed tomography, positron emission tomography, motion imaging), in 1, 2, or 3 dimensions, e.g., related to the surgical area or patient

[0313] Any slope is possible. Any perspective projection is possible. Any tilt angle is possible. Any curved surface is possible. Any curved perspective projection is possible. Any combination of one-dimensional, two-dimensional, and three-dimensional data is possible between different types of data.

[0314] Registering real-time and virtual data via an optical head-mounted display

[0315] In some embodiments, patient virtual data can be superimposed on the real-time data viewed through the optical head-mounted display. The virtual data can be raw data in an unprocessed form, such as pre-operative images of the patient, or it can be processed data, such as filtered data or segmented data.

[0316] Data segmentation

[0317] When the patient image is superimposed on the real-time data viewed through the optical head-mounted display, image segmentation may be required in many embodiments. Any algorithm known in the art can be used for this purpose, such as value processing, seed point techniques, lines of fire, deformable models, statistical models, active shape models, level set methods, marching cubes algorithms, artificial neural networks, deep learning techniques, or combinations thereof. Many of these algorithms are part of open source or commercial libraries, such as the Insight Segmentation and Registration Toolkit (ITK), the open source computer vision library OpenCV, G'MIC (GREYC's Magic for Image Computing), Caffe, or MATLAB (MathWorks, Natick, Mass.). Figure 2 A representative workflow for segmentation and subsequent procedures is provided in . Optional preoperative imaging 40 may be obtained. Optional intraoperative imaging 41 may be obtained. The preoperative 40 or intraoperative 41 imaging may be segmented 42 to extract, for example, surfaces, volumes, or key features. An optional three-dimensional reconstruction or three-dimensional rendering 43 may be generated. The preoperative 40 or intraoperative 41 imaging and any three-dimensional reconstruction or three-dimensional rendering 43 may be registered in a common coordinate system 44. The preoperative 40 or intraoperative 41 imaging and any three-dimensional reconstruction or three-dimensional rendering 43 may be used to generate a virtual surgical plan 45. The virtual surgical plan 45 may be registered in a common coordinate system 44. The surgical site 46 may be registered in a common coordinate system 44. Intraoperative measurements 47 may be obtained and used to generate the virtual surgical plan 45. An optical head mounted display 48 may project or display virtual data or a digital hologram of the virtual data 49 superimposed on and aligned with the surgical site. The optical head-mounted display 48 is configured to use a built-in camera or image capture or video capture system 50 to optionally detect and / or measure the position and / or orientation and / or alignment of one or more optical markers 51, which can be used for coordinate measurement 52 and can be part of intraoperative measurements 47.

[0318] Registration software and algorithms

[0319] Various techniques known in the art can be used to perform the registration of real-time data and virtual data. These include, but are not limited to, surface registration algorithms (e.g., iterative closest point algorithms), statistical models, active shape models (ASMs), mutual information-based or other volume registration algorithms, object recognition, pattern recognition or computer vision techniques, deep learning or other artificial intelligence methods. The processed data, for example, can consist of mesh data, parametric surface data, point cloud data, volume data or a combination thereof. These methods are known in the art and have been implemented in public and / or commercially available code libraries and application programming interfaces (APIs), such as the Insight Segmentation and Registration Toolkit (ITK), the open source computer vision library OpenCV, Elastix, Plastimatch or the Medical Image Registration Toolkit (MIRTK).

[0320] Optical head-mounted display superimposes virtual data and real-time data

[0321] In some embodiments, the segmented data or raw data can be superimposed on the patient's real-time data viewed through the optical head-mounted display. This superposition can occur in an unregistered form, that is, the patient's virtual data may not be aligned with the real-time data viewed through the optical head-mounted display. In this case, the operator wearing the optical head-mounted display can move his head in the direction of the corresponding features of the virtual data and the patient's real-time data that can be superimposed. The surgeon or operator can also use other means to move and redirect the virtual data, such as a trackball or virtual display interface displayed in the optical head-mounted display, which is independent of the surgeon's / operator's head movement. The operator can adjust the magnification of the real-time data so that the size, shape, length, thickness of certain features of the virtual data match the size, shape, length, thickness of the real data within a given distance of the object / patient.

[0322] For example, during brain surgery, a surgeon can observe the exposed gyri and sulci of a patient's brain while visually displaying real-time data. The optical head-mounted display can display a virtual three-dimensional model of the patient's gyri and sulci. The surgeon can selectively adjust the magnification of the three-dimensional model so that the model matches the size, width, or length of the corresponding gyri and sulci in the real-time data. The surgeon can selectively adjust the transparency or opacity of the virtual data displayed in the optical head-mounted display. The ratio of virtual data to real-time data transmitted through the optical head-mounted display can be 1:10, 1:9, 1:8, 1:5, 1:2, 1:1, 2:1, 3:1, 5:1, 8:1, 10:1, and fractions or multiples thereof. Any combination of transparency or opacity of virtual data and real-time data is possible. The direction of movement of the surgeon's head can superimpose virtual features, such as the patient's gyri and sulci, and the patient's real-time data.

[0323] Once the data is superimposed, the surgeon can choose to register the virtual data with the real-time data. Registration is as simple as described above, for example, the surgeon visually confirms that the virtual and real-time data substantially match or substantially overlap. At this point, the surgeon can optionally reference the real-time data and / or the real-time data's coordinate system to the virtual data and / or the virtual data's coordinate system (in two, three, or more dimensions). Once the data is registered, the surgeon can move their head to any desired position or orientation, for example, to observe the patient's brain or lesion and adjacent structures from different perspectives, such as sensitivity. The inertial measurement unit of the optical head-mounted display will register the head movement, the direction of the head movement, the latest head position, and the head orientation. The changes in the surgeon's head position and orientation can be applied simultaneously or, if desired, non-simultaneously to the virtual data, which can now be superimposed with the resulting new position and orientation of the real-time data. Additionally, as the surgeon moves their head or body further away from the target anatomical structure, the inertial measurement unit can measure the change in position and the increased distance from the target anatomical structure. Depending on the distance from the inertial measurement unit, the virtual data can be scaled up or down by a factor such that, in some embodiments, the size, shape, and dimensions of the virtual data will approach or match the size, shape, and dimensions of the real-time data regardless of the distance, position, and orientation of the surgeon's head.

[0324] For the purpose of registering the virtual data with the real-time data, the optical head-mounted display is optionally placed in a fixed position, such as mounted on a stand or tripod. When the optical head-mounted display is placed in the fixed position, the surgeon can view the real-time data and can choose to register it with a camera and / or display it on a monitor. The virtual data can then be overlaid, and matching and registration of the virtual data and the real-time data can be performed. At this point, the surgeon or operator can remove the optical head-mounted display from the fixed position, and the surgeon can wear the optical head-mounted display during the surgical procedure.

[0325] The virtual data can optionally be displayed using different colors, such as red, green, or yellow. Optionally, only the outlines of selected features of the virtual data can be displayed. For example, these features could be the sulci of the patient's brain (e.g., black lines or lines in black or another color), without visualizing the gyri at the edges of these sulci. Alternatively, for example, only one lesion can be displayed, such as a tumor, such as a glioblastoma, in the case of the brain. Alternatively, the virtual data of normal and pathological tissues can be displayed in combination.

[0326] The virtual data can be registered with the real-time data viewed through the optical head-mounted display. Registration can be performed using any method known in the art to register or cross-reference virtual data and real-time data in 2, 3 or higher dimensions.

[0327] In some embodiments, the registration of virtual data and real-time data is maintained throughout the surgical procedure. In some embodiments, the registration of virtual data and real-time data is maintained during the surgical procedure or selection of a surgical plan, which can be or can include a virtual surgical plan, such as a surgical plan generated preoperatively.

[0328] In some embodiments of the present invention, the optical head-mounted display superimposes virtual data and real-time data simultaneously. In some embodiments, the optical head-mounted display superimposes virtual data and real-time data non-simultaneously. For example, the virtual data may be superimposed intermittently.

[0329] The virtual data can be transparent, semi-transparent, or opaque. If the virtual data are opaque, they can be displayed intermittently so that the operator or surgeon can see how they are projected in relation to the patient's real-time data.

[0330] If a combination of virtual data is displayed simultaneously with real-time data, different types of virtual data may be displayed in different colors. Representative combinations of virtual and real-time data are provided below. The following is merely illustrative and in no way limiting of the present invention:

[0331] Real-time data: Patient's brain; gyri and sulci exposed during surgery.

[0332] Live data: Surgical instruments, such as biopsy needles or cutting tools

[0333] Virtual data: patient's brain with gyri and sulci, derived from an imaging modality and optionally segmented from a display modality, e.g., a CT scan or an MRI scan

[0334] Virtual Data: Brain Tumors Deep in the Brain

[0335] Virtual data: The same surgical instrument currently used by the surgeon, with the virtual data indicating the desired direction, position or orientation of the surgical instrument in a virtual image of the instrument.

[0336] Any of the aforementioned virtual data may be displayed in two or three dimensions. Multi-dimensional displays as outlined in other parts of the description are also possible.

[0337] For example, normal tissue of a patient, such as normal brain tissue, can be displayed in two dimensions, for example, using a grayscale image, while abnormal tissue of a patient, such as a stroke, hemorrhage, or tumor, can be displayed in three dimensions. The optical head-mounted display can display any combination of two-dimensional, three-dimensional, and multi-dimensional images; and the optical head-mounted display can overlay any combination of two-dimensional, three-dimensional, and multi-dimensional images on real-time patient data.

[0338] Virtual 2D, 3D and multi-dimensional data can be generated or acquired through different data acquisition technologies, such as different imaging examinations, etc.

[0339] Lock or move virtual data

[0340] In some embodiments of the present invention, the virtual data can be associated with the surgeon or operator, or with the patient or a target anatomical structure within the patient. This means that even if the surgeon moves their head or body or part of the patient's anatomy moves, the virtual data will not move in the OHMD display. For example, once registered, the optical head-mounted display can display a virtual image of the target tissue or adjacent tissue. The virtual image of the target tissue or adjacent tissue can be, for example, an image through a tumor or other type of pathological tissue. When the surgeon or operator moves their head or body during the surgical procedure, the virtual data will not move, but will be displayed in the same position.

[0341] In some embodiments of the present invention, the virtual data can move in conjunction with the surgeon or operator, or in conjunction with the patient or a target anatomical structure within the patient's body. This means that if the surgeon moves his or her head or is moving the patient's body or part of the anatomical structure, the virtual data will move in the OHMD display. For example, once registered, the optical head-mounted display can display a virtual image of the target tissue or adjacent tissue. The virtual image of the target tissue or adjacent tissue can be, for example, an image through a tumor or other type of pathological tissue. As the surgeon or operator moves his or her head or body during the surgical procedure, the virtual data will move and change its position and orientation in the same way that the surgeon moves his or her head or body, generally reflecting the change in perspective or angle of view created by the surgeon's movement of his or her head or body.

[0342] Optionally, the movement of the virtual data is a larger virtual distance or a larger angle, or a smaller virtual distance or a smaller angle than the movement of the surgeon's head or body.

[0343] Improve accuracy when moving or redirecting virtual data

[0344] Once registration is achieved between the virtual data and the physical data, movement or reorientation of the virtual data can follow, for example, the surgeon's head movements or body movements or movements of an operating arm or hand, or movements of the patient or certain body parts, for example, by monitoring the movement and changes in position and / or orientation of the surgeon's head using an inertial measurement unit of the optical head-mounted display.

[0345] In some embodiments, an optical or radio frequency tracker (RF tracker) or other tracking device known in the art can be applied to the optical head mounted display and / or the patient, including a selected body part or target tissue of the patient, such as the patient's knee. Using standard surgical navigation techniques known in the art, the spatial position of the optical or radio frequency tracker can be registered, such as for a starting posture or position or positioning. Movement of the tracker, such as caused by movement of the surgeon's head or body or by movement of at least a portion of the patient, can be tracked using the navigation system. Information about changes in position, orientation, or direction of movement of the surgeon's head or the patient or both is then used to update virtual data accordingly, or to update virtual data displayed by the optical head mounted display accordingly, or both. In this way, virtual data and real-time data can be superimposed by the optical head mounted display, typically in a precise manner.

[0346] Optionally, the position, orientation, direction, and other data generated by the inertial measurement unit can be combined with such data generated by the surgical navigation system. The combination of data can be beneficial in more accurately measuring changes in the position or orientation of the surgeon's head, body, operating arm, hand, or patient.

[0347] Using dummy data in 2 or more dimensions

[0348] In some embodiments of the present invention, an optical head-mounted display can display a two-dimensional virtual image of a patient. The image can be a transmission-type image, such as an X-ray or a CT scout scan. The image can be a cross-sectional image of selected anatomical structures of the patient. The image can be an original image or a reconstructed, segmented, or partially segmented image of the patient.

[0349] In some embodiments of the present invention, the surgeon will view the patient's real-time data through an optical head-mounted display, such as the exposed brain surface and the patient's gyri and sulci. The surgeon can align the patient's virtual data with the patient's real-time data, such as performing an MRI scan of the patient's brain. The alignment can be performed in 2, 3 or more dimensions. The alignment of real-time data and virtual data can include aligning different types of virtual data, such as different types of normal or diseased tissue, using different imaging modes, different sizes for different types of normal or diseased tissue, etc. Multiple two-dimensional scan planes can be displayed simultaneously. These two-dimensional scan planes can be parallel or non-parallel planes with variable angles, orthogonal or non-orthogonal planes.

[0350] Scroll and move virtual data overlaid on real-time data

[0351] In some embodiments of the present invention, a surgeon or operator can optionally scroll through a set of continuous or non-continuous virtual two-dimensional image data and three-dimensional image data being superimposed on the patient's real-time data, typically the real-time data is from the same anatomical site, such as the brain, spine, hip, knee, etc. The scrolling can be guided by any type of user interface known in the art. For example, the surgeon can use a virtual interface projected by an optical head-mounted display, in which he or she can move a virtual arrow up or down or left or right to scroll the image backward or forward, or, for example, rotate the image, or display the image at different multi-planar angles or change the viewing angle or projection angle.

[0352] The surgeon may optionally scroll or move the virtual image data by moving the head forward or backward, e.g., to scroll backward or forward in a virtual image volume. For example, the surgeon may move the head left or right to rotate the image, display the image at different multi-planar angles, or change the viewing angle or projection angle of the three-dimensional image.

[0353] The surgeon can optionally scroll the virtual image data by moving their hand, finger, or any other body part forward and backward, for example, to scroll backward or forward in the virtual image volume. For example, the surgeon can move their hand, finger, or any other body part left and right to rotate the image or display the image at different multi-planar angles or change the viewing angle or projection angle. The surgeon can move their hand or finger in a rotational or turning motion to rotate or turn the virtual data. Any combination of head, hand, eye, and other body signals can be used to change the display of the virtual data.

[0354] These on-screen changes of the virtual data may optionally be performed in the optical head-mounted display using the same variables associated with the location, position, orientation, angle, direction, and motion of the surgeon's body part that are used to trigger the on-screen changes. Alternatively, any of the related changes in the location, position, orientation, angle, direction, and motion of the virtual data may be performed using a magnification factor or reduction factor associated with the change in the location, position, orientation, angle, direction, and motion of the surgeon's body part. These magnification factors or reduction factors may be linear or non-linear, such as exponential or logarithmic. In some embodiments, the further the surgeon's body part that is controlled to move the virtual data in the OHMD display is from its original position, the greater the change caused by the movement of the virtual data in the optical head-mounted display. In some embodiments, the further the surgeon's body part that is controlled to move the virtual data in the OHMD display is from its original position, the smaller the change caused by the movement of the virtual data in the optical head-mounted display.

[0355] Using dummy data in 3 or more dimensions

[0356] In some embodiments of the present invention, an optical head-mounted display can display a three-dimensional virtual image of a patient. The three-dimensional image of the patient can include three-dimensional images of different types of anatomical structures, such as in the area of ​​a planned surgery or surgical site.

[0357] Three-dimensional reconstructions of image data or other patient data can be generated before, during, and / or after surgery. The virtual three-dimensional image can include an entire anatomical region or selected tissues or selected tissues within an anatomical region. For example, an optical head-mounted display can virtually display different tissues in three dimensions using different colors. This allows for the display of both normal and pathological tissues.

[0358] For example, normal tissues may include brain tissue, heart tissue, lung tissue, liver tissue, vascular structures, bone, cartilage, spinal tissue, intervertebral discs, and nerve roots. The optical head-mounted display can virtually visualize any tissue.

[0359] For example, registering patient virtual data with real-time data at the surgical site

[0360] In some embodiments of the present invention, virtual patient data displayed by the optical head-mounted display and real-time patient data viewed through the optical head-mounted display are spatially registered with respect to one another, e.g., in a common coordinate system, with one or more optical head-mounted displays in the same coordinate system. Virtual and physical surgical instruments and implant components may also be registered in a common coordinate system. Spatial co-registration has the advantage that the simultaneous display of virtual and real-time patient data is not affected or is less affected when the surgeon moves his or her head or body, when the optical head-mounted display moves, or when the patient moves. Thus, the perspective of the real-time patient data viewed by the surgeon's eyes through the optical head-mounted display, e.g., the real-time surgical field, may remain the same as the perspective of the virtual patient data viewed by the surgeon's eyes through the display of the optical head-mounted display device, e.g., the virtual surgical field, virtual surgical plane, virtual path, virtual cutting path or plane projected into the surgeon's eyes, even if the surgeon moves his or her head or body. In this manner, the surgeon does not need to rethink or adjust his or her hand-eye coordination because the real-time patient data seen through the surgeon's eyes and the virtual patient data seen through the OHMD display are superimposed, which is fundamentally different from other approaches, such as surgical navigation methods that utilize a separate computer monitor in the operating room, where the surgeon's perspective is different from his or her perspective of the real-time data of the patient and surgical field. Furthermore, with surgical navigation, a first virtual instrument can be displayed on a computer monitor that is an image of a physical instrument tracked with a navigation marker, such as an infrared or radio frequency marker, and the position and / or orientation of the first virtual instrument can be compared with the position and / or orientation of a corresponding second virtual instrument generated in the virtual surgical plan. Thus, with surgical navigation, the positions and / or orientations of the first and second virtual instruments are compared.

[0361] When providing guidance in a mixed reality environment, for example, using a stereoscopic display similar to an electronic holographic environment, a virtual surgical guide, tool, instrument, or implant can be superimposed on the joint, spine, or surgical site. Furthermore, the physical guide, tool, instrument, or implant can be aligned with the two-dimensional or three-dimensional image of the virtual surgical guide, tool, instrument, or implant. Thus, guidance in the mixed reality environment does not require the use of multiple virtual images of the guide, tool, instrument, or implant, and does not require comparison of the position and / or orientation of multiple virtual images of the virtual guide, tool, instrument, or implant.

[0362] In some embodiments, the virtual data can move in conjunction with the surgeon or operator, or with the patient or a target anatomical structure within the patient's body. This means that if the surgeon moves their head or is moving the patient's body or part of the patient's anatomy, the virtual data will move in the OHMD display. For example, once the optical head-mounted display, the patient's virtual data, and the patient's real-time data are registered in a common coordinate system, the optical head-mounted display can display a virtual image of the target tissue or adjacent tissue. The virtual image of the target tissue or adjacent tissue can be, for example, a tumor or other type of pathological tissue, or an image of the spine or spinal pedicles. As the surgeon or operator moves their head or body during the surgical procedure, the virtual data will move and change its position and orientation in the same manner as the surgeon moves their head or body, generally reflecting the change in perspective or angle of view created by the surgeon's movement of their head or body. The virtual data can include a three-dimensional image of a surgical tool or instrument, such as a needle used in kyphoplasty or vertebroplasty, where the virtual image of the needle shows its intended position, orientation, or path relative to the spine and / or pedicles. The virtual data may also include a medical device, such as a pedicle screw, wherein the virtual data of the pedicle screw indicates its expected position, orientation or path relative to the spine, and / or pedicle, and / or vertebral body.

[0363] In some embodiments, the registration is performed using at least three or more points that can be superimposed or fused into a common object coordinate system for the virtual data and the real-time data. The registration can also be performed using the surface or three-dimensional shape of the anatomical structure present in the patient's virtual data and the real-time data. In this case, the virtual surface can be moved until it substantially matches the patient's live surface, or the virtual shape can be moved until it substantially matches the patient's live shape.

[0364] Different means can be used to achieve the registration of the patient virtual data and the patient real-time data. The following is in no way intended to limit the present invention, but is merely exemplary.

[0365] Registering virtual patient data with live patient data using directly or indirectly linked object coordinate systems

[0366] If virtual data (e.g., patient image data) is acquired with the patient positioned in a first target coordinate system, and real-time data (e.g., during surgery) is observed or acquired with the patient positioned in a second target coordinate system, where the first and second target coordinate systems can be directly connected (e.g., physically) or indirectly connected (e.g., non-physically), then registration of the patient's virtual data and real-time data can be performed. The direct connection between the first and second target coordinate systems can be, for example, a physical connection between the first and second target coordinate systems. For example, the patient can be moved along the length of a tape measure from the first target coordinate system to the second target coordinate system. Alternatively, the patient can be scanned using a scanner, such as a computed tomography scanner or an magnetic resonance imaging scanner, and the scanner table can then be moved out of the scanner to perform a surgical procedure on the patient while the procedure is still on the scanner table. In this case, the scanner table can be the physical connection between the first and second target coordinate systems, and the length of the table movement between the scanning position and the position outside the scanner (for real-time data, such as surgery) can define the coordinate transformation from the first target coordinate system to the second target coordinate system.

[0367] An indirect connection between a first (virtual data) object and a second (real-time data) object can be established, provided that the patient moves between acquiring the virtual data (e.g., using imaging studies) and the real-time data (e.g., when performing a surgical procedure along a defined path, where the direction and angle of the path are known, such that the first and second object coordinate systems can be referenced to each other, and using the known information about the patient's defined path and the virtual data, an object coordinate transfer can be applied, and the patient's and the optical head-mounted display's real-time data can be registered in a common coordinate system). Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0368] Registration of virtual patient data with real-time patient data can also be performed without a direct or indirect connection to an object coordinate system using other means and methods explained in the following paragraphs and columns, for example, when the patient performs one or more movements of unknown direction, length, or magnitude. Combinations of all the different registration methods described in this specification are possible, for example, by switching between registration methods during operation or by using multiple registration methods simultaneously to improve registration accuracy.

[0369] Registration using spatial mapping

[0370] A spatial mapping process can be used to acquire or register real-time patient data, such as the position and / or orientation of a physical instrument, the position and / or orientation of an implant component, and the position and / or orientation of one or more optical head-mounted displays. The process uses, for example, but not limited to, a depth sensor, a laser scanner, a structured light sensor, a time-of-flight sensor, an infrared sensor, or a tracking probe to create a three-dimensional mesh that describes the surface of one or more objects or environmental structures. These devices can generate three-dimensional surface data by collecting three-dimensional coordinate information or distance information from sensors of one or more surface points on one or more objects or environmental structures. The three-dimensional surface points can then be connected to the three-dimensional surface mesh to produce a three-dimensional surface image of the real-time data. The surface mesh can then be merged with the virtual data using any of the registration techniques described in the specification.

[0371] Real-time data can be static or, more preferably, continuously updated with additional information to incorporate changes in the position or surface of one or more objects or environmental structures. The additional information can be acquired through depth sensors, laser scanners, structured light sensors, time-of-flight sensors, infrared sensors, or tracking probes.

[0372] For initial spatial mapping and updating of mapping data, commonly available software code libraries can be used. For example, this functionality can be provided by the Microsoft HoloToolkit or Google Project Tango platforms. Various techniques for spatial mapping and tracking have been described, including those described in US Pat. No. 9,582,717, which is expressly incorporated herein by reference.

[0373] Registering virtual patient data with live patient data using visual anatomical features

[0374] a) The surgeon or operator can visually register the patient's virtual data with the patient's real-time data

[0375] In some embodiments, the surgeon or operator can visually align or match the patient virtual data with the patient real-time data. This visual alignment or matching of the patient virtual data and the patient real-time data can be performed by moving the optical head-mounted display, for example, by moving the head of the operator wearing the optical head-mounted display. In this example, the patient virtual data is displayed in a fixed manner, rather than changing the perspective when the operator moves the optical head-mounted display. The operator will move the optical head-mounted display until the patient real-time data is aligned or superimposed on the fixed projection of the patient virtual data. Once satisfactory alignment, matching or superposition of the patient data with the patient virtual data is achieved, the surgeon can execute a registration command, for example, through a voice command or a keyboard command. The patient virtual data and the patient real-time data are now registered. At this point, after registration is completed, the patient virtual data will move as the optical head-mounted display moves, for example, by measuring movement using an integrated inertial measurement unit, image and field of view tracking; for example, using an image and / or video capture system, and / or using an additional navigation system with optical or radio frequency or other trackers that can be attached to the patient, the surgical site, the bone or any other tissue of the patient, the surgeon, the surgeon's arm, the surgeon's head or the optical head-mounted display worn by the surgeon, and using anchor points in the image or field of view for measurements.

[0376] Thus, once satisfactory alignment or matching is achieved, the surgeon can execute commands for successful registration. Registration can include a change in at least one of the position, orientation, and magnification of the virtual data and the real-time data in order to achieve alignment or matching. The magnification applied to the virtual data can indicate the distance from the optical head-mounted display or the surgeon's head to the matching tissue. As a means of maximizing registration accuracy, the estimated distance between the optical head-mounted display and the target tissue or skin surface or other reference tissue can be confirmed by optional physical measurements of the distance, particularly if the optical head-mounted display is in a fixed position, such as on a stand or tripod, which can be optionally used during initial registration. After successful alignment or matching, the surgeon's commands can be registered in the same common coordinate system, such as the patient virtual data and the patient real-time data or the optical head-mounted display. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0377] In some embodiments of the invention, the visual anatomical data may be gyri or osteophytes or bone spurs or pathological bone deformations or tumor nodules or nodules, for example, on the surface of the liver or brain.

[0378] In some embodiments of the present invention, the methods described herein may be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surfaces or tissue contours or shapes, such as the shape of a bone after milling or reaming, or tissue perimeters, such as the perimeters of a bone cut or tissue volume or other tissue features of a live patient, may be matched, superimposed and / or aligned with surgically altered tissue or tissue surfaces or tissue contours or tissue perimeters or tissue volumes or other tissue features in the patient virtual data. For example, in a virtual surgical plan developed for a patient, the surgeon's eyes see the patient virtual data through an optical head-mounted display device display, and the surgeon's eyes see the patient real-time data through an optical head-mounted display device at substantially the same viewing angle. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration may use the same methods described above or any other registration methods described in this specification or any other registration methods known in the art. Reference Figure 3 , Figure 3The example illustrates registering a digital hologram or virtual data for an initial surgical step, performing the surgical step, and re-registering one or more holograms for subsequent surgical steps. An optical head-mounted display can project or display virtual data or a digital hologram of the virtual data of a patient 55. The digital hologram is optionally fixed to the optical head-mounted display so that it moves with the movement of the optical head-mounted display 56. The surgeon can move the optical head-mounted display until the digital hologram of the virtual data or the patient's virtual data is superimposed on the patient's real-time data and aligned with the patient's real-time data (e.g., surgical site 57). The virtual data or the digital hologram of the virtual data can then be registered 58 using the same or similar coordinates as the real-time data on which the digital hologram was superimposed. The surgeon can then perform one or more predetermined surgical steps, such as bone cutting 59. After a surgical change is made using the real-time data 60, the virtual data or the digital hologram of the virtual data can be optionally registered or re-registered. The virtual data or the digital hologram of the virtual data after the surgical change can optionally be displayed by the optical head-mounted display 61. The virtual data after the surgical change or the digital hologram of the virtual data is optionally fixed relative to the optical head-mounted display so that it moves with the movement of the optical head-mounted display 62. The surgeon can move the optical head-mounted display until the virtual data after the surgical change or the digital hologram of the virtual data is superimposed and aligned with the real-time data after the surgical change 63. The virtual data or the digital hologram of the virtual data can then be registered using the same or similar coordinates as the real-time data after the surgical change to which the digital hologram is superimposed 64. The surgeon can then perform one or more predetermined subsequent surgical steps, such as bone cutting, milling, or drilling 65. The aforementioned steps are optionally repeated until the surgical procedure is completed 66. A virtual surgical plan can be used 67. Optionally, the optical head-mounted display 68 can display the patient's native anatomy, including after the first surgical change. The optical head-mounted display optionally displays digital holograms of subsequent surgical steps 69.

[0379] b) Automatically or semi-automatically registering patient real-time data with patient virtual data using image processing and / or pattern recognition and matching techniques

[0380] c) In some embodiments of the present invention, image processing techniques, pattern recognition techniques, or techniques based on deep learning / artificial neural networks can be used to match virtual patient data with real-time patient data. Alternatively, image processing and / or pattern recognition algorithms can be used to identify certain features, such as gyri or sulci on the brain surface of the virtual patient data. Ears with unique shapes can also be used to match virtual patient data with real-time patient data.

[0381] For example, during brain surgery, the patient can be placed on an operating table. A cleansing or disinfecting solution, such as povidone-iodine, can be applied to the shaved skull. The optical head-mounted display can be placed on the patient, either on a tripod or worn by the surgeon, with the patient's head tilted sideways to cover the patient's ears and skull. The optical head-mounted display is then placed over the area of ​​the patient where the patient's virtual data is to be displayed.

[0382] Virtual patient data can be displayed in an optical head-mounted display. The virtual patient data can include visualizations of the patient's skin, ears, or nose, or other data, such as from preoperative MRI data. Virtual data of the patient's skin or other structures, such as the patient's ears or nose, can be displayed simultaneously with the real-time patient data. The virtual patient data can then be moved, reoriented, realigned, and optionally scaled up or down until the alignment, match, or superposition is satisfactory. The optical head-mounted display can also optionally be moved during this process to achieve a satisfactory size match between the virtual patient data and the real-time patient data, optionally without scaling up or down the virtual patient data.

[0383] Once satisfactory alignment, matching, or superposition is achieved between the patient virtual data and the real-time patient data, the operator can execute a command indicating successful registration. The position, orientation, or change in orientation of the optical head-mounted display measured by an integrated inertial measurement unit, image and field of view tracking (e.g., utilizing anchor points in an image or field of view using an image and / or video capture system), and / or a navigation system coupled to the optical head-mounted display can be used to move the patient virtual data through the view of the real-time patient data obtained by the optical head-mounted display using substantially the same object coordinates as the patient virtual data and the real-time patient data, thereby maintaining registration during surgery regardless of movement of the optical head-mounted display, such as head movement of a surgeon wearing the optical head-mounted display, and ensuring that the patient virtual data is correctly superimposed with the real-time patient data when projected into the surgeon's field of view.

[0384] After successfully registering the patient virtual data to the patient's skin or other structure, such as an ear or nose, the surgeon or assistant may apply a marker or calibration or registration phantom or device to the patient, for example, near the intended site of craniotomy. The marker or calibration or registration phantom or device cannot be covered by any drape or surgical cover subsequently placed. The patient virtual data and the patient's real-time data are then re-registered to each other by registering the patient virtual data to the patient's real-time data, for example, the real-time data of the patient's skin or other structure, such as an ear or nose, using the real-time marker or calibration or registration phantom or device placed on the patient and by cross-referencing. This can be achieved by registering the patient's skin or other structure, such as an ear or nose, in the same coordinate system as the marker or calibration or registration phantom or device placed on the patient, for example, by co-registering the patient virtual data of the patient's skin or other structure, such as an ear or nose, or osteophytes or bone spurs or other bony anatomy or deformity, with the real-time data of the marker or calibration or registration phantom or device. The distance, offset, angular offset, or overall difference in coordinates between the patient's skin or other structures, such as the ear or nose, or osteophytes or spurs, or other bony anatomy or deformity, and a marker or calibration or registration phantom or device attached to the patient can be measured and used to switch the registration of the patient's virtual data to the patient's real-time data of the skin or other structure, such as the ear or nose, and the real-time data of the marker or calibration or registration phantom or device. The system can choose to maintain both the real-time data of the patient's skin or other structure, such as the ear or nose, and the real-time data of the marker or calibration or registration phantom or device for registration. Optionally, the system can assess whether registration to the real-time data of the patient's skin or other structure, such as the ear or nose, or the real-time data of the marker or calibration or registration phantom or device is more accurate, and the system can switch back and forth between the two. For example, if the distance from the optical head-mounted display to the patient's skin or other structure, such as the ear or nose, increases or decreases by more than a certain amount, such as an optionally predefined threshold, or is partially covered by a drape, the system can switch the registration to the real-time data of the marker or calibration or registration phantom or device. The reverse is also possible. Alternatively, if the angle between the patient's skin or other structures such as the ear or nose, or osteophytes or bone spurs or other bony anatomy or deformity, and the optical head-mounted display increases or decreases beyond a certain level, such as an optionally predefined threshold, the system can switch registration to real-time data from a marker or calibration or registration phantom or device. The reverse is also possible.

[0385] The surgeon or assistant can then place a sterile drape or surgical cover over the site, but preferably without covering the markers or calibration or registration phantom or device. Registration can be maintained by real-time data from the markers or calibration or registration phantom or device attached to the patient, for example, near or within the craniotomy site.

[0386] Image processing and / or pattern recognition can then be performed on the real-time patient data via the optical head-mounted display, for example, using a built-in image capture device to capture real-time data from the patient or an image and / or video capture system attached to, integrated with, or coupled to the optical head-mounted display.

[0387] Then, features or patterns of the virtual and real-time data can be matched. To successfully register the patient's real-time data and the superposition of the two, the matching can include movement and / or reorientation and / or enlargement and / or reduction of the virtual data. The virtual and real-time data can include osteophytes or bone spurs or other skeletal anatomy or deformities.

[0388] A combination of (a) and (b) (e.g., an automatic registration function with a manual adjustment option), such as by moving the virtual image data against the real-time image data after the image processing software and / or pattern recognition software and / or matching software identifies a potential match or performs an initial match, and then manual / surgeon-based adjustments can be made. Alternatively, manual / surgeon-based matching and registration can be performed first, and then fine-tuned by matching and registration-based software or algorithms (image processing, pattern recognition, etc.). The virtual data and the real-time data may include osteophytes or bone spurs or other bony anatomy or deformities.

[0389] In some embodiments of the present invention, the methods described herein can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan developed for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same methods described above or any other alignment methods described in this specification or any other alignment methods known in the art.

[0390] Registering virtual patient data with live patient data using anatomical landmarks

[0391] In some embodiments, the surgeon can identify selected anatomical landmarks in the patient's virtual data, such as the patient's electronic preoperative plan, as well as anatomical landmarks in the patient's real-time data. For example, the surgeon can identify anatomical landmarks by placing a cursor or marker on the electronic image of the patient's virtual data and, once the cursor or marker is at the desired location, clicking on the anatomical landmark to identify it. In the spine, such anatomical landmarks can be, for example, the posterior tip of the spinous process, the vertebral lamina, the inferior articular facet on the patient's left side, the superior articular facet on the patient's left side, the inferior articular facet on the patient's right side, the superior articular facet on the patient's right side, the tips of the facet joints, bone spurs, osteophytes, etc. In the hip, such landmarks can be the most anterior point of the acetabulum, such as osteophytes at the acetabular rim, within the acetabulum, adjacent to the acetabulum, on the femoral head, at the femoral neck or neck-shaft junction, at the center of the femoral head in two-dimensional or three-dimensional images, at the most anterior point of the femoral head, the anterior superior iliac spine, the anterior inferior iliac spine, the pubic symphysis, the greater trochanter, the lesser trochanter, etc. In the knee, such a landmark can be the femoral condyle, the femoral recess, the condylar space, the medial or lateral epicondyle, the femoral axis, the epicondylar axis, the trochlear axis, the mechanical axis, the trochlear groove, the femoral osteophyte, the marginal femoral osteophyte, the central femoral osteophyte, the patellar dome, the upper edge of the patella or femur or the femoral articular surface, the inner edge, the outer edge, the lower edge, the patellar osteophyte, the anterior tibia, the tibial spine, the inner edge, the outer edge, the anterior edge, the posterior edge of the tibia, the tibial osteophyte, the marginal tibial osteophyte, the central tibial osteophyte. The surgeon can then identify the same landmark in the patient. For example, when the surgeon is observing through an optical head-mounted display, the surgeon can point to the corresponding anatomical landmark in the real-time data with a finger or with an indicating device. The tip of the pointer or the tip of the finger optionally includes a tracker that positions the tip of the pointer or the finger in space. Such positioning can also be accomplished intuitively using image capture, for example, to more accurately determine the distance and position of a pointer or finger from the optical head-mounted display in stereo. An image and / or video capture system can also be connected to, integrated with, or coupled to the optical head-mounted display. The virtual data and real-time data can include osteophytes or bone spurs or other bony anatomy or deformities.

[0392] Representative anatomical landmarks that can be used to register a patient's virtual and real-time data may include, but are not limited to:

[0393] In the spine:

[0394] - Part or all of the spinous process

[0395] - Partial or complete spinal plate

[0396] - Part or all of the spinal articular process

[0397] - Part or all of a facet joint

[0398] - Part or all of the transverse process

[0399] - Part or all of the pedicle

[0400] - Part or all of a vertebral body

[0401] - Part or all of the intervertebral disc

[0402] - Osteophytes on part or all of the spine

[0403] - Bone spurs on part or all of the spine

[0404] - Fracture of part or all of the spine

[0405] - Fracture of part or all of the vertebral body

[0406] - Any combination of the above

[0407] Hip:

[0408] - Part or all of the acetabulum

[0409] - Part or the entire rim of the acetabulum

[0410] -Multiple segments of the acetabular rim

[0411] -Part of the iliac wall

[0412] -Part of the pubic bone

[0413] -Part of the ischium

[0414] -Anterior superior iliac spine

[0415] -Anterior inferior iliac spine

[0416] - Pubic symphysis

[0417] -Part of the greater rotor or the entire greater rotor

[0418] -Part of or the entire lesser trochanter

[0419] - Part of or the entire femoral shaft

[0420] - Part or all of the femoral neck

[0421] - Part or all of the femoral head

[0422] -Bone concave

[0423] - Transverse acetabular ligament

[0424] - Posterior thalamic tubercle

[0425] -ligament

[0426] -upper lip

[0427] - One or more osteophytes, femoral and / or acetabular

[0428] - Any combination of the above

[0429] knee:

[0430] - Part or all of the medial condyle

[0431] - Part or all of the lateral femoral condyle

[0432] - Partial or complete femoral notch

[0433] -Part or all of the pulley

[0434] -Part of the anterior femoral cortex

[0435] - A portion of the anterior femoral cortex and the adjacent portion of the trochlea

[0436] - A portion of the anterior femoral cortex, with adjacent portions of the trochlea and osteophytes when present

[0437] - One or more osteophytes on the femur and / or tibia

[0438] - One or more bone spurs in the femur and / or tibia

[0439] - Epicondylar eminence

[0440] - Part or all of the mid-tibial plateau

[0441] - Part or all of the lateral tibial plateau

[0442] - Part or all of the middle tibial spine

[0443] - Partial or entire lateral tibial spine

[0444] -Part of the anterior tibial cortex

[0445] - Part of the anterior tibial cortex and part of the tibial plateau, medial or lateral or both

[0446] - Part of the anterior tibial cortex and part of the tibial plateau, medial or lateral or both and the presence of osteophytes

[0447] - Part or all of the patella

[0448] - Medial edge of the patella

[0449] - Lateral edge of the patella

[0450] - Upper pole of the patella

[0451] -Inferior pole of the patella

[0452] - Patellar osteophytes

[0453] -Anterior cruciate ligament

[0454] -Posterior cruciate ligament

[0455] -Medial collateral ligament

[0456] - Lateral collateral ligament

[0457] - Part or all of the medial meniscus

[0458] - Part or all of the lateral meniscus

[0459] - Any combination of the above

[0460] Shoulders:

[0461] - Partial or complete glenoid

[0462] - Part or all of the coracoid process

[0463] - Part or all of the acromion

[0464] -Part of the clavicle

[0465] - Part or all of the humeral head

[0466] - Part or all of the humeral neck

[0467] -Part of the humeral shaft

[0468] - One or more humeral osteophytes

[0469] - One or more glenoid osteophytes

[0470] - Part or all of the glenoid labrum

[0471] - Part or all of the shoulder ligaments, such as the coracoacromial ligament, superior, middle, or inferior glenohumeral ligament

[0472] -Part of the shoulder blade

[0473] - Any combination of the above

[0474] Skull and Brain:

[0475] -Part of the skull

[0476] -Part of the occipital bone

[0477] -Part of the temporal bone

[0478] -Part of the occipital bone

[0479] -Part of the parietal bone

[0480] -Part of the frontal bone

[0481] -Part of the facial bones

[0482] - Part of facial structure

[0483] - Part or all of the bone structure inside the skull

[0484] - Select some or all of the gyri

[0485] - Select part or all of the sulci

[0486] -Part of the sinus

[0487] -Part of the venous sinus

[0488] -Part of the blood vessel

[0489] -Part of the ear

[0490] -Part of the external auditory canal

[0491] organ:

[0492] - Part of an organ, such as the upper or lower pole of a kidney

[0493] -The edges or margins of the liver, spleen, or lungs

[0494] -Part of the liver lobe

[0495] -Part of the blood vessel

[0496] - A part of a crack, for example, in the liver or spleen

[0497] -Part of the uterus

[0498] One skilled in the art can identify other anatomical landmarks of hard tissue, soft tissue, and / or organs, including the brain, that can be used to register virtual data (including an optional virtual surgical plan) with real-time data from the patient and the optical head-mounted display in a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system.

[0499] In some embodiments of the present invention, an optical head-mounted display can display an arbitrary virtual plane over the surgical field. The arbitrary virtual plane can be moved using a virtual interface or other interface. For example, the arbitrary virtual plane can include a "contact area" where gesture recognition software, such as the software provided by Microsoft with Microsoft HoloLens, which has an integrated virtual "drag function" for holograms, can be used to move the arbitrary virtual plane. For example, one or more cameras integrated into or attached to the optical head-mounted display can capture the movement of the surgeon's finger relative to the contact area; using gesture tracking software, the virtual plane can then be moved by pushing the finger toward the contact area in the desired direction.

[0500] The optical head-mounted display can initially display any virtual plane at any position, for example, projected onto or outside the surgical field, such as a hip, knee, shoulder, ankle, or spine. The optical head-mounted display can optionally display any virtual plane at a limited angle, for example, orthogonal or parallel relative to a fixed structure in the operating room. Any virtual plane can be identified using one or more cameras integrated into the optical head-mounted display, an image capture or video capture system, and spatial recognition software such as Microsoft HoloLens provided by Microsoft, or can be identified using one or more additional optical markers or navigation markers (including infrared or radio frequency markers). For example, one or more optical markers can be attached to an extension of the operating table. The optical head-mounted display can detect these one or more optical markers and determine their coordinates, and thereby determine the horizontal plane of the operating table. The virtual plane can then be displayed perpendicularly or at another angle relative to the operating room table.

[0501] For example, during a hip replacement, an optical head-mounted display can display a virtual plane over the surgical site. The virtual plane can be perpendicular to the operating table. Using a virtual interface, such as contact areas and gesture tracking on the virtual surgical plane, the optical head-mounted display can detect how the surgeon moves the virtual plane. Optionally, the virtual plane can maintain its perpendicular (or other desired angle) orientation relative to the operating table as the surgeon moves and / or reorients the plane; a perpendicular orientation may be desired when the surgeon intends to make a perpendicular cut to the femoral neck. A different angle may be desired when the surgeon intends to cut the femoral neck in another orientation.

[0502] Then, using the contact areas or other virtual interfaces, the surgeon can move the virtual plane to a desired position, orientation, and / or alignment. Movement of the virtual plane can include translation and rotation in any desired direction using any desired angle or vector, or a combination thereof. The surgeon can move the virtual plane to intersect a selected anatomical landmark or to intersect a selected anatomical or biomechanical axis. The surgeon can move the virtual plane to be tangential to a selected anatomical landmark or a selected anatomical or biomechanical axis.

[0503] For example, during a hip replacement, the surgeon can move the virtual plane to be tangent to the uppermost portion of the greater trochanter and the uppermost portion of the lesser trochanter. Figure 4A The example of illustrates a virtual plane 70 which the primary surgeon has moved and aligned to be tangential to the uppermost portion of the greater trochanter 71 and the uppermost portion of the lesser trochanter 72 . Figure 4BThe example illustrates the same virtual plane 70 where the primary surgeon has moved and aligned to be tangential to the uppermost portion of the greater trochanter 71 and the uppermost portion of the lesser trochanter 72, with the view now being from the optical head mounted display of a second surgeon or surgical assistant, e.g., on the other side of the OR table.

[0504] Optionally, for example, using a pointer with additional optical markers or additional navigational markers, or using an image or video capture system integrated into an optical head-mounted display, and gesture recognition software such as provided by Microsoft in Microsoft HoloLens to detect his finger, or using a finger with additional optical markers or navigational markers, the surgeon can point to and identify the sulcus point, for example, the lowest point between the greater trochanter and the femoral neck, as an additional reference. The line connecting the uppermost aspect of the greater trochanter and the uppermost aspect of the lesser trochanter can then be determined on a preoperative or intraoperative AP radiograph of the hip joint; optionally, the sulcus point can also be detected on an AP radiograph. The AP radiograph can include a template that the surgeon uses to select and determine the size of, for example, the femoral and acetabular component components, as well as the liner and / or femoral head. The radiographic template can include a femoral neck cutting indication. The angle between the line connecting the uppermost aspect of the greater trochanter and the uppermost aspect of the lesser trochanter and the femoral neck cutting indication can be determined. Figure 4C The illustrated second virtual plane 73 (virtual femoral neck cutting plane 73) can then be projected or displayed by the optical head-mounted display. This plane is also perpendicular to the operating table, like virtual plane 70, and is tangent to the uppermost portion of the greater trochanter 71 and the uppermost portion of the lesser trochanter 72. The angle and / or distance between femoral neck cutting plane 73 and the arbitrary virtual plane is the same as the angle and distance between the line connecting the uppermost portions of the greater trochanter and the lesser trochanter and the femoral neck cutting indication on the radiograph. In this manner, the femoral neck cutting plane can be defined using a second virtual plane that is specified or predetermined based on a virtual plane placed during surgery, which is moved by the surgeon to be tangent to the uppermost portions of the greater trochanter and the lesser trochanter. The virtual femoral neck cutting plane specified and projected or displayed in this manner can also be a virtual guide, such as a virtual cutting block that projects a virtual slot for guiding a physical saw. The virtual guide or cutting block can have one or more dimensions identical to a physical guide or cutting block, such that the physical guide or cutting block can be aligned with the virtual guide or cutting block. The virtual guide or cutting block can be an outline of the physical guide or cutting block, a two-dimensional or three-dimensional representation, a partial outline or a complete outline, having one or more identical dimensions, so that the surgeon can align the physical guide or cutting block with the virtual guide or cutting block. The virtual guide or cutting block can include placement markings for the physical guide or cutting block.

[0505] If radiographic magnification is of interest in defining a second virtual plane, such as a virtual cutting plane, based on a first virtual plane, e.g., a plane tangent to or intersecting one or more anatomical landmarks or one or more anatomical or biomechanical axes at an angle associated with or derived from a preoperative radiograph, a combination of measuring distances and applying a correction for magnification can be optionally employed. For example, the distance between one or more anatomical landmarks that are tangent to or intersecting the virtual plane can be measured in the live patient data and can be measured on the radiograph. If the radiographic distance is greater or less than the live patient distance, a magnification correction can be applied, e.g., the distance between a first virtual plane, such as a plane tangent to or intersecting one or more anatomical landmarks or one or more anatomical or biomechanical axes, and a second virtual plane, such as a virtual cutting plane, can be corrected based on a radiographic magnification factor.

[0506] In another example, an arbitrary virtual plane can be projected or displayed outside or on the surgical field during knee replacement. Optionally, the arbitrary virtual plane is at least initially perpendicular to the operating table or at a certain angle to the operating table. If the mechanical axis of the leg has been determined in a previous step using intraoperative measurements, such as an optical marker attached to the thigh and one or more optical markers attached to the ankle joint, for determining the center of rotation of the hip joint and the center of the ankle joint using an image capture or video capture system integrated into, connected to, or separated from an optical head-mounted display, the virtual plane can be configured to be perpendicular to the mechanical axis of the leg. Using a virtual interface such as a contact area, and an image or video capture system integrated into or attached to the optical head-mounted display and optional gesture tracking software, the surgeon can move and / or realign the virtual plane, such as to intersect the medial and lateral joint spaces of the exposed knee joint, such as to increase flexion or to 5, 10, 15, 20, 30, 45 degrees of flexion or more. Figure 5 An example of is an arbitrary imaginary plane 74 in the knee that intersects the extended medial 76 and lateral 75 joint spaces.

[0507] Then, one or more additional virtual planes are optionally projected, for example, perpendicular to the operating table or at another angle, or using a desired femoral component flexion angle or a desired tibial slope. The surgeon can optionally move these one or more virtual planes to coincide with one or more anatomical axes, for example, the anatomical femoral shaft axis or the anatomical tibial shaft axis in the live patient. The surgeon can also move the virtual plane to be placed and oriented in the center of the femoral notch, with the virtual plane parallel to the notch wall and extending centrally between the medial and lateral femoral shaft cortices, as a means of estimating the anatomical femoral shaft axis.

[0508] Once the anatomical femoral and / or tibial axes are determined or estimated, a virtual surgical plan is designed that includes femoral and tibial resections to achieve the desired femoral mechanical axis correction, e.g., from the patient's mechanical axis alignment correction, e.g., 5, 10, 15 degrees of varus or valgus, to normal mechanical axis alignment or any desired residual, e.g., congenital varus or valgus may be formed or produced. Implant size and required polyethylene thickness may be factored into the virtual surgical plan. An optical head-mounted display may then project a virtual surgical cutting plane based on the virtual surgical plan and / or intraoperative measurements, the desired varus and / or valgus correction, the desired slope and / or the desired implant rotation. The surgeon may then align the physical saw blade with the projected or displayed virtual saw blade. Alternatively, the optical head-mounted display can display a virtual guide or virtual cutting block having at least one or more dimensions that are the same as the physical guide or physical cutting block, and the surgeon can align the physical cutting guide or cutting block with the virtual guide or cutting block, insert the saw blade into the physical guide or cutting block and perform one or more blocks.

[0509] The aforementioned concepts of projecting arbitrary virtual planes and aligning them with one or more anatomical landmarks, anatomical axes, or biomechanical or mechanical axes can be applied to any joint as well as the spine. Similarly, these concepts can be applied to brain surgery, where one or more virtual planes can be projected or displayed and moved to be tangential to or intersecting one or more anatomical landmarks, such as the gyrus, pons, cerebellum, etc. Similarly, these concepts can be applied to organ surgery, where one or more virtual planes can be projected or displayed and moved to be tangential to or intersecting one or more anatomical landmarks, such as the hilum and anterior liver margin, one or more heart valves, etc.

[0510] Any other two-dimensional and / or three-dimensional virtual shape or contour or surface, such as a cube, cuboid, prism, cone, cylinder, sphere, ellipsoid-derived three-dimensional shape, irregular shape, etc., can be virtually projected or displayed and automatically or using a virtual or other user interface, moved, oriented, or aligned to coincide with, be tangent to, intersect with, partially or completely overlap with: the patient's anatomy, pathology, anatomical axes, biomechanical axes including mechanical axes, anatomical planes, three-dimensional shapes, two-dimensional and / or three-dimensional geometric shapes, three-dimensional surfaces, and / or three-dimensional volumes of any internal organ, soft or hard tissue of the patient; after being moved, oriented, or aligned, coordinate information of the two-dimensional and / or three-dimensional virtual shape or contour or surface can be measured. Optionally, additional intraoperative measurements can be performed based on the coordinate information, and / or the information can be used to develop or modify the virtual surgical plan.

[0511] In some embodiments of the present invention, the methods described herein (including anatomical landmarks) can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan formulated for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same method described above or any other alignment method described in this specification or any other alignment method known in the art. Optionally, different anatomical landmarks can also be used for the first alignment and any subsequent alignment. Alternatively, the same anatomical landmark can be used for the first alignment and any subsequent alignment.

[0512] Use light sources to reference anatomical landmarks

[0513] The tracker or pointing device can also be a light source that can produce a laser-generated red or green dot on the patient's tissue, highlighting anatomical landmarks for registration. The light source can be selected to have an intensity and / or color that will easily distinguish it from the patient's living tissue.

[0514] A laser or other light source is optionally integrated into or attached to the optical head-mounted display, for example, the laser or light source can be integrated into or connected to a bridge piece of the frame that connects the left eye and right eye portions of the optical head-mounted display, for example, on the nose area.

[0515] For example, an image capture device integrated into, attached to, or coupled to an optical head-mounted display can be used to identify the location of light on patient tissue or anatomical landmarks. Once the light is directed to the desired location on the patient's live data, specifically, a live anatomical landmark, registration can be performed by executing a registration command to align the live patient data with the virtual data, such as a laser or other light reflected from the live anatomical landmark and the patient's corresponding virtual anatomical landmark. This process can be repeated for different anatomical landmarks, for example, by directing the light source to the next live anatomical landmark on the patient, confirming the exact location or direction of the light. For example, a red or green laser dot reflected from a live patient anatomical landmark can be captured by the optical head-mounted display's image capture device and software, and the next anatomical live anatomical landmark can be registered with the patient's corresponding virtual anatomical landmark. The virtual data and live data may include osteophytes, bone spurs, or other bony anatomy or deformities. In this way, the optical head-mounted display, the live patient data, and the virtual patient data can be registered in a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0516] In some embodiments, more than one real-time and virtual anatomical landmark of the patient will be used, eg, two, three, or more.

[0517] In some embodiments of the present invention, an ultrasound or radio frequency transmitter can be used to precisely locate certain in-situ anatomical landmarks. For example, the ultrasound transmitter or radio frequency transmitter can be integrated into a point device, such as the tip of a pointing device. When the tip contacts the desired in-situ anatomical landmark, the transmitter can emit an ultrasound signal or radio frequency signal that can be captured at a receiving site, optionally integrated into an optical head-mounted display. Optionally, multiple receiving sites can be used at different spatial locations, for example, as a means to improve the accuracy of real-time data registration. The virtual data and the real-time data can include osteophytes or bone spurs or other bony anatomy or deformities.

[0518] In some embodiments of the present invention, the dimensions of the pointer have previously been scanned and registered with the optical head-mounted display. An image and / or video capture system connected to, integrated with, or coupled to the optical head-mounted display can identify the pointer in the real-time data and can identify the tip of the pointer. When the tip of the pointer touches a live anatomical landmark on the patient that corresponds to an anatomical landmark in the virtual data, the surgeon can click to indicate that the cross-reference has been successful. The two data points are then optionally fused or superimposed in a common coordinate system. The virtual and real-time data and data points can include, or can be generated from, osteophytes or bone spurs or other bony anatomical structures or deformities. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0519] Anatomical landmarks may include unaltered surface shapes, such as skin, facial features such as the tip of the nose, the distance between the eyes, the position of the ears, the shape of the ears, etc.

[0520] The anatomical landmarks may also be bony anatomical landmarks, such as the medial or lateral malleolus, tibial tuberosity, medial or lateral epicondyle, trochlear notch, spinous process, etc. The virtual and real-time data and virtual and live anatomical landmarks may include osteophytes or bone spurs or other bony anatomy or deformities.

[0521] Alternatively, a live anatomical surface can be used for registration purposes. In this embodiment, a real-time anatomical surface can be derived during surgery using optical scanning, infrared scanning, or ultrasound or ultrasonic scanning techniques. The patient's live surface detected and generated in this manner can be matched or aligned with a patient's virtual surface, for example, obtained preoperatively using imaging studies such as X-ray imaging, ultrasound, computed tomography, or magnetic resonance imaging, or any other technique known in the art. The virtual and real-time data and anatomical surfaces can include osteophytes or spurs or other bony anatomy or deformities.

[0522] In some embodiments of the present invention, the methods described herein can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan developed for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same methods described above or any other alignment methods described in this specification or any other alignment methods known in the art.

[0523] The patient virtual data and the patient real-time data are registered using implanted or attached markers or a calibration or registration phantom, or a device including optical markers.

[0524] In some embodiments of the present invention, the surgeon can optionally register the patient's live and virtual data using implantable or attached markers. This is useful, for example, if the surgery is extensive and results in the removal of tissue from the surgical site, as is the case with neurosurgery, such as removing a brain tumor, as well as liver surgery, such as removing a liver tumor, joint replacements, and many other types of surgery. The virtual and live data may include osteophytes or spurs or other bony anatomy or deformities.

[0525] The terms implantable marker, attached marker, skin marker, soft tissue marker, calibration or registration phantom or device, and image capture marker as used throughout this application include optical markers, such as optical markers having different geometric shapes or patterns, optical markers with QR codes, bar codes, or alphanumeric codes. Implantable or attached markers or calibration or registration phantoms or devices can be implanted prior to the actual surgery and can be included in preoperative, intraoperative, and / or postoperative images. Implantable or attached markers or calibration or registration phantoms or devices can be implanted or attached to osteophytes or bone spurs or other bony anatomical structures or deformities.

[0526] If implanted or attached markers or calibration or registration phantoms or devices are present in the virtual image data, the surgeon can readily identify the implanted or attached markers or calibration or registration phantoms or devices after the cut because the surgeon has access to the target tissue and the implanted markers placed next to or within the target tissue. Such implanted or attached markers or calibration or registration phantoms or devices may include radiation beets or metallic beets, for example, which are also used for stereoscopic imaging or registration.

[0527] Alternatively, an implantable or attached marker or calibration or registration phantom or device can be placed during surgery, for example, by image capture through an optical head-mounted display, or by image capture through an attached optical head-mounted display, an integrated or coupled optical head-mounted display, and the position of the implantable or attached marker or calibration or registration phantom or device can be determined. The position of the implantable or attached marker or calibration or registration phantom or device in the patient's physical data can then be matched to the position of the anatomical structure of the implantable or attached marker or calibration or registration phantom or device attached to the patient's virtual data. For example, the anatomical structure in the virtual and real-time data may include osteophytes or bone spurs or other bony anatomy or deformity. In some embodiments, a pointer or pointing device is optionally placed on the implantable or attached marker or calibration or registration phantom or device, and image capture is then performed through the optical head-mounted display and other image capture devices connected to the optical head-mounted display and connected or coupled to the optical head-mounted display, and the pointer tip is registered. In this manner, an optical head-mounted display, an implanted or attached marker or calibration or registration phantom or device comprising an optical marker, and anatomical structures, pathological structures, instruments, implant components, and any other objects to which one or more implanted or attached markers or calibration or registration phantoms or devices comprising optical markers are attached, and patient virtual data can be registered in a common coordinate system. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0528] Implantable or attached markers or calibration or registration phantoms or devices may include rigid or fixed registration markers. Such rigid or fixed registration markers may be used to maintain registration as the surgical field changes. The rigid or fixed registration markers may be screws or pins. The virtual data and real-time data may include osteophytes or spurs or other bony anatomy or deformity. The rigid or fixed registration markers may be attached to osteophytes or spurs or other bony anatomy or deformity. In some embodiments, the medical device being implanted or a component thereof is already temporarily or permanently attached to patient tissue, and during surgery, for example, while subsequent steps of the surgery are completed, the osteophytes or spurs or bony anatomy or deformity, or the anatomical site or surgical site may be used as implantable or attached markers or calibration or registration phantoms or devices. These subsequent steps may include implanting additional components of the medical device. For example, in spinal fusion surgery, a first pedicle screw may be implanted. The real-time data and virtual data of the first pedicle screw may be registered. By maintaining registration between the real-time data and the virtual data using the registered first pedicle screw, subsequent pedicle screws or other components can be virtually displayed in the optical head-mounted display, including their intended paths, locations, positions, or orientations. Any other rigid or fixed registration markers or implantable devices can be used in this manner for different types of surgical procedures on the human body.

[0529] One or more implantable or attached markers or calibration or registration phantoms or devices are attached to bone, cartilage, soft tissue, organs or pathological tissue, such as osteophytes or bone spurs or other bony anatomical structures or deformations.

[0530] The one or more implanted or attached markers or calibration or registration phantoms or devices optionally include optical markers, retroreflective markers, infrared markers or radiofrequency markers or any other marking devices described in the art.

[0531] An optical marker is a marker that reflects light within the visible spectrum, i.e., the portion of the electromagnetic spectrum visible to the human eye, with wavelengths between approximately 390 and 700 nm or a frequency band between approximately 430 and 770 THz. An optical marker may also reflect a mixture of light comprising different wavelengths within the visible spectrum. The light reflected by the optical marker can be detected by an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display. The location, position, orientation, alignment, and / or direction of movement of the optical marker can be detected using an image and / or video capture system that is integrated into, connected to, or separate from the optical head-mounted display and capable of performing relevant image processing, along with optional pattern recognition software and systems. The optical marker may include a marker having a selected geometric pattern and / or shape that is recognizable by an image and / or video capture system (e.g., integrated into, attached to, or separate from the optical head-mounted display), for example, using image processing and / or pattern recognition techniques. The optical indicia may be an alphanumeric code or pattern and / or a numeric code or pattern and / or an alphanumeric code or pattern or other code or pattern combination, such as a barcode or QR code that can be recognized by an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display, for example, using image processing and / or pattern recognition techniques. A QR code or quick response code may be any current or next-generation matrix code, including a barcode. Barcodes and QR codes are machine-readable optical labels that contain information such as patient identifier, patient condition, type of surgery, surgical site, spinal level if spinal surgery was performed, patient side of surgery, one or more surgical instruments, one or more trial implants, one or more implant components, including implant type and / or implant size, polyethylene type, acetabular liner type (e.g., standard, labled, offset, other), and if hip replacement is contemplated. QR codes can use various standardized encoding schemes, such as numeric, alphanumeric, byte / binary, and / or Chinese characters, to store data. Other encoding schemes may be used. Any current and / or future version of a QR code may be used. Single-color or multi-color QR codes may be used. Other graphical markers, such as those supported by the Vuforia (PTC, Needham, Mass) augmented reality platform, may also be used.

[0532] A barcode, QR code or other graphic mark may be an optical mark. A barcode, QR code or other graphic mark may be part of an optical mark or may be integrated into an optical mark.

[0533] - Information related to the patient and / or surgical site, such as patient identifier, age, sex, BMI, medical history, risk factors, allergies, site and side (left, right), spinal segment to be operated on

[0534] - Information related to inventory management, including surgical instruments and / or implants or implant components, e.g., left and right components, selected component sizes (matched to the virtual surgical plan and / or template and / or size)

[0535] And can be used to obtain information about the location, position, orientation, alignment and / or movement direction (if applicable) of the surgical site, surgically altered tissue, one or more surgical instruments and one or more trial implants and / or implant components.

[0536] Geometric patterns, geometric shapes, letters, numbers, alphanumerics, and other codes or patterns of bar codes and two-dimensional codes included in one or more optical markers or a portion thereof can be predefined and, optionally, stored in a database accessible by an image and / or video capture system and associated image processing software and pattern recognition software. Geometric patterns, geometric shapes, letters, numbers, alphanumerics, and other codes or patterns of bar codes and two-dimensional codes included in one or more optical markers or a portion thereof can be two-dimensional images, while some are three-dimensional images. For example, one or more planes or two-dimensional patterns can be used in selected embodiments. Alternatively, selected three-dimensional geometric shapes can be used, such as cubes, cuboids, prisms, cones, cylinders, spheres. Any three-dimensional shape can be used, including irregular and / or asymmetric shapes. Three-dimensional geometric shapes can include two-dimensional geometric patterns and / or letters, numbers, alphanumerics, and other codes or patterns including bar codes and two-dimensional codes on one or more surfaces. For example, if a cuboid is used, the same or different geometric patterns and / or letters, numbers, alphanumeric and other codes or patterns (including bar codes and QR codes) may be included on, attached to or integrated into one or more surfaces thereof, such as two opposing surfaces or two adjacent surfaces, for example, oriented vertically. Two-dimensional geometric patterns and / or letters, numbers, alphanumeric and other codes or patterns (including bar codes and QR codes) may be included on, fixed to or integrated into one or more surfaces of a three-dimensional geometric shape, which may be used to determine the orientation and / or alignment of the surface of the geometric shape including optical markers, and the orientation and / or alignment of the surface, and the geometry, for example, in relation to the surgical site and to the surgical alteration, specifically including a cut bone surface or a reamed bone surface, a surgical instrument and / or one or more implant components including a trial implant.

[0537] Geometric patterns and / or geometric shapes, letters, numbers, alphanumeric characters, and other codes or patterns, including barcodes and QR codes, can be in color or black and white. Geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric characters, and other codes or patterns, including barcodes and QR codes, can include both colored and black and white portions, only colored portions, and only black and white portions. Geometric shapes can include both colored and black and white surfaces, only black and white surfaces, and only colored surfaces. Different colors and codes can be used on different surfaces of the geometric shape portion of an optical mark. Different colors and codes can be used for different geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric characters, and other codes or patterns, including barcodes and QR codes. Different colors and codes can be used for different optical marks. Different colors, such as red, blue, green, orange, cyan, etc., can be used for different geometric patterns and / or geometric shapes and / or letters, numbers, alphanumeric characters, and other codes or patterns, including barcodes and QR codes. Different colors, such as red, blue, green, orange, yellow, pink, cyan, etc., can be used for different optical marks. Different optical markers can optionally be associated with different surgical steps and / or different surgical instruments and / or different implant components; the use of a specific marker can be identified by using standard image processing and / or pattern recognition software integrated into the optical head-mounted display, attached to the optical head-mounted display, or separate from the optical head-mounted display, and the image and / or video capture system optionally includes a pattern database, such as a database of images associated with specific surgical steps and / or surgical instruments. As the image and / or video capture system recognizes a specific optical marker in the field of view, for example, based on a specific geometric pattern and / or geometric shape and / or letters, numbers, alphanumerics, and other codes or patterns including bar codes, it can then selectively display the corresponding surgical step and / or surgical instrument and / or implant component associated with the optical marker.

[0538] Two-dimensional geometric patterns, letters, numbers, alphanumerics and other codes or patterns including bar codes and 2D codes or combinations thereof, optionally color and / or black and white coded, included in, attached to or integrated into one or more three-dimensional geometric surfaces, can be used to determine the orientation of a selected surface of a geometry and / or the alignment of a selected surface of a geometry, the orientation and / or alignment of a geometry and / or optical marker, for example, with respect to anatomical landmarks, surgical sites, surgical alternations, in particular: cut bone surfaces or reamed bone surfaces, surgical instruments and / or one or more implant components including trial implants, etc. One or more two-dimensional geometric patterns, letters, numbers, alphanumerics and other codes or patterns including bar codes and 2D codes or combinations thereof, optionally color and / or black and white coded, included in, attached to or integrated into optical markers, can be used to determine the orientation and / or alignment of optical markers fixed to or integrated into anatomical landmarks, surgical sites, surgical alternations, in particular: cut bone surfaces or reamed bone surfaces, surgical instruments and / or one or more implant components including trial implants, etc. Optical markers can be affixed to anatomical landmarks, surgical sites, or surgical changes (e.g., cut or reamed bone surfaces that can be identified in the patient virtual data, or patient drill holes and corresponding anatomical landmarks, surgical sites, or surgical changes), thereby enabling registration of the patient's virtual data and real-time data in the same coordinate system. If multiple optical head-mounted displays are used during surgery, optical markers can also be attached to the optical head-mounted displays, including multiple optical head-mounted displays. Optionally, optical markers (e.g., using QR codes) can be used to distinguish between the first, third, fourth, and / or more optical head-mounted displays. Optionally, one or more optical markers can be attached to the operating table and they can be registered in a coordinate system, for example, one or more optical head-mounted displays, the patient, and portions of the surgical site can be registered in the same coordinate system. One or more optical markers can optionally be attached to other structures in the operating room, including fixed structures (e.g., walls) and movable structures (e.g., operating room lights), and they can be registered in a coordinate system, for example, one or more optical head-mounted displays, the patient, and portions of the surgical site can be registered in the same coordinate system. In this example, the optical marker may also be mounted to a fixed structure on the retaining arm or extender, optionally movable, eg, with known size, orientation, length, and angle.

[0539] Optical markers attached to fixed structures, such as operating room walls, can be used to enhance the accuracy of room identification and spatial mapping, particularly when the coordinates and / or angles and / or distances between different optical markers are known. Optical markers attached to fixed structures (e.g., operating room walls) can also be used to enhance the determination of the position and posture, as well as changes in position or posture, or changes in coordinates and coordinates, of one or more optical head-mounted displays, which can help increase the accuracy of virtual data displays and overlays on corresponding real-time data.

[0540] Optical markers attached to movable structures can be used to track their position in the operating room. Optical markers attached to operating room lights can be used to estimate the direction of light and the direction and / or trajectory of shadows in the operating room or room. If the direction and / or trajectory of shadows in the operating room or room is known, virtual shadows with the same or similar direction or trajectory can be applied to the virtual data display of the optical head-mounted display.

[0541] For example, one or more optical markers including one or more geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes, or combinations thereof, can be attached to the medial femoral epicondyle, for example, using pins or screws or adhesive. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, orientation, alignment, and / or direction of movement of the optical marker relative to the image and / or video capture system; as the distal femur moves, the image and / or video capture system can detect the marker (e.g., based on its pre-programmed geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes) and can monitor and optionally register the movement. If, in the same example, a second optical marker comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns (including bar codes and QR codes or combinations thereof) is attached to the lateral femoral condyle, the image and / or video capture system can also monitor and optionally register the position, and / or orientation, and / or alignment, and / or direction of movement of the second optical marker relative to the image and / or video capture system; by monitoring the position, and / or orientation, and / or alignment, and / or direction of movement of the first optical marker on the medial femoral epicondyle and the position, and / or orientation, and / or alignment, and / or direction of movement of the second optical marker on the lateral femoral epicondyle, the image and / or video capture system and associated image processing and pattern recognition software can also monitor and optionally register the movement of the femoral epicondyle axis (e.g., during flexion and extension of the knee) . One or more optical markers comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof can be attached to the proximal tibia (e.g., the proximal tibia), the anterior tibial edge, the medial and / or lateral tibial spine, the lowest point of the medial plateau, and / or the highest point of the lateral tibial plateau. For example, in the same example. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, orientation, alignment, and / or movement direction of the optical markers attached to the tibia relative to the image and / or video capture system and to one or more femoral optical markers, thereby monitoring and optionally registering tibiofemoral motion, for example, during surgery. One or more optical markers comprising one or more geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof can be attached to the patella (e.g., the uppermost, lowermost, lateralmost, and / or medialmost side in the same example).An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, and / or orientation, and / or alignment, and / or direction of movement of an optical marker attached to the patella relative to the image and / or video capture system and to one or more femoral optical markers, thereby monitoring and optionally registering patellofemoral motion, such as during surgery. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can be used to monitor the position, and / or orientation, and / or alignment, and / or direction of movement of an optical marker attached to the patella relative to one or more tibial optical markers, thereby monitoring and optionally registering patellar motion relative to the tibia, such as during tibial adduction or abduction.

[0542] In some embodiments of the present invention, optical markers using one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof can be assigned to virtual surgical steps. The markers can include written text that defines the surgical step or corresponds to a surgical step, which can be the immediately preceding surgical step or the next surgical step, for example, in a virtual surgical plan. In some embodiments, the text can be a number (e.g., a digit corresponding to a specific surgical step), such as 1 for distal femoral cut, 2 for anterior femoral cut, 3 for posterior femoral cut, 4 for first chamfer cut, and 5 for second chamfer cut. The number can be recognized by an image and / or video capture system, which can then display a virtual view of the corresponding surgical step, such as 1 for a cutting plane for the distal femoral cut or a virtual outline of a corresponding physical distal femoral cutting block. A combination of numbers and text can be used, and the image and / or video capture system and associated software and optional pattern recognition software and system can recognize the numbers and text and trigger commands to display corresponding virtual views of the corresponding virtual surgical steps, such as, 1F-distal femoral cut, 2F-anterior femoral incision, 1T-proximal tibial cut, 2T-tibial keel, etc.

[0543] In another example, an optical marker having one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof can be assigned to the "distal femoral cut" step in a virtual surgical plan for a total knee replacement of a patient; the optical marker may include the text "distal femoral cut". For example, the surgeon may affix the marker to the cut bone surface of the distal femur or somewhere adjacent thereto. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can detect the optical marker having one or more specific geometric patterns and / or specific geometric shapes assigned to "distal femoral cut", indicating that the distal femoral cut has been completed; the image capture signal can then command the optical head-mounted display to display the next surgical step (e.g., the anterior cutting plane or the outline of the anterior cutting block or cutting guide when the surgeon is preparing to make the next cut), such as the anterior femoral cut in this example.

[0544] In some embodiments, optical markers using one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns, including bar codes and QR codes, or combinations thereof, can be integrated into, included in, or attached to surgical instruments used for surgical steps in a virtual surgical plan. For example, the optical marker can be included in, integrated into, or attached to a surgical cutting block or cutting tool, such as for a proximal tibial cut. Alternatively, the marker can include written text that defines or corresponds to a surgical step, such as in a virtual surgical plan. In the preceding example, an optical marker having one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns, including bar codes and QR codes, or combinations thereof, can be assigned to the "proximal tibial cut" step in a virtual surgical plan for a total knee replacement of a patient; the optical marker can include the text "proximal tibial cut" that the surgeon can read and ensure the correct marker is used for the next surgical step he or she is considering, in this example, the proximal tibial cut.

[0545] When the optical marker enters the surgeon's field of view, an image and / or video capture system of an optical head-mounted display integrated into or connected to the surgeon's head can detect the optical marker and display the next virtual surgical step, such as an outline of a virtual proximal tibial cutting block corresponding to the physical proximal tibial cutting block, so that the surgeon can align or superimpose the physical surgical cutting block or instrument onto the outline of the virtual surgical cutting block or instrument. Alternatively, when the optical marker enters the surgeon's field of view, an image and / or video capture system of an optical head-mounted display integrated into or connected to the surgeon's head can detect the optical marker and display the next virtual surgical step, such as a virtual cutting plane having a predetermined resection level, varus or valgus angle, and / or inclination, so that the surgeon can align or superimpose the physical surgical cutting block and / or physical surgical saw onto the virtual cutting plane. Once the surgical step is completed, e.g., a proximal tibial cut, and the surgeon removes the physical surgical instrument with the integrated, included, or attached optical marker from the surgical field and / or the field of view of the image and / or video capture system, the image and / or video capture system can detect that the optical marker is no longer in the field of view, and the software can generate a command to turn off the optical head mounted display screen or the screen that completes the virtual surgical step. Optionally, a command can be generated at this time, optionally automatically, to display the next surgical step (e.g., tibial keel punch), e.g., to set tibial rotation. Alternatively, the display of the optical head mounted display device can display the next surgical step for the next surgical instrument while the next surgical instrument with the corresponding optical marker for the next surgical step enters the field of view, e.g., in the hands of the surgeon.

[0546] In a similar example, an optical marker can be attached to an acetabular reamer for hip replacement. An image and / or video capture system integrated into or connected to the optical head-mounted display can detect the optical marker when it enters the surgeon's field of view, triggering a command to display a reaming axis or a virtual display of the reamer with the intended alignment or orientation for the reaming step; when the surgical instrument's optical marker leaves the surgeon's field of view, the image and / or video capture system can detect it, triggering a command to stop displaying the reamer or the virtual display of the reamer, and optionally switching to the next surgical step.

[0547] In some embodiments, one or more optical markers may be included in, integrated into, or attached to an insert for a cutting block or guide. The insert can be configured to fit into one or more slots or guides within the cutting block or guide to guide the saw blade. For example, representative cutting blocks or guides are those used for knee replacements, shoulder replacements, hip replacements, and ankle replacements. For example, these cutting blocks or guides are used to remove bone at the joint surface to fit the patient's bone to the bone-facing side of an implant or implant assembly. The insert can be designed to partially or substantially fill the entire slot or guide, for example, in the X and Y directions, or in the X and Z directions, or in the Y and Z directions, depending on the shape and / or design of the cutting block or guide. If the insert partially fills or substantially fills the slot or guide in the X and Y directions, the insert can be configured to extend beyond the slot or guide in the Z direction. If the insert partially fills or substantially fills the slot or guide in the X and Z directions, the insert can be configured to extend beyond the slot or guide in the Y direction. If the insert partially fills or substantially fills the slot or guide in the Y and Z directions, the insert can be configured to extend beyond the slot or guide in the X direction. Depending on the configuration of the cutting block or guide and the configuration of the associated slot or guide, any orientation is possible, including oblique, orthogonal, and non-orthogonal. For example, an oblique slot can be used for a chamfered resection in a total knee replacement or an oblique talar cut in a total ankle replacement.

[0548] The portion of the insert that extends beyond the slot or guide may include one or more integrated or attached optical markers. If more than one optical marker is used, the optical markers may be arranged at predetermined angles and positions, e.g., 90 degrees or less or greater than 90 degrees. The insert may have dimensions similar to a representative saw blade for a cutting block or guide. The insert may indicate the location, position, orientation, alignment, and direction of travel of a saw blade that will subsequently be inserted. The surgeon may place the insert within the slot or guide of a physical cutting block or guide and align the insert with a virtual cutting plane or virtual contour of the insert or cutting block or guide projected by an optical head-mounted display onto the surgical site, e.g., the distal femur in a total knee replacement or the proximal femur in a total hip replacement. Once the insert is substantially aligned and / or superimposed with the virtual cutting plane or virtual contour of the insert or cutting block or guide, the surgeon can secure the physical cutting block or guide to the bone, thereby securing the cutting block or guide to the bone. In this secured position, the virtual surgical plan, such as the virtual cutting plane or virtual contour of the insert or cutting block or guide, is substantially aligned with the physical cutting plane and / or the physical insert or cutting block or guide. The surgeon can then insert the physical saw blade and perform the physical cut. The insert can be configured to have a shape substantially similar to a physical saw blade, acting as a dummy saw blade.

[0549] Alternatively, the surgeon can place a physical saw blade within a slot or guide of a physical cutting block or guide, and the surgeon can align the physical saw blade with a virtual cutting plane or virtual contour of the saw blade or cutting block or guide projected by the optical head-mounted display onto the surgical site, e.g., the distal femur in a total knee arthroplasty or the proximal femur in a total hip arthroplasty. Once the physical saw blade is substantially aligned and / or superimposed with the virtual cutting plane, virtual contour of the saw blade or cutting block or guide, the surgeon can pin the physical cutting block or guide to the bone, thereby securing the cutting block or guide to the bone, and in a fixed position, the virtual surgical plan, e.g., the virtual cutting plane or virtual contour of the saw blade or cutting block or guide, is substantially aligned with the physical cutting plane and / or the physical saw blade or cutting block or guide. The surgeon can then advance the physical saw blade and perform the physical cut.

[0550] The optical marker can be included in a cutting block or guide or insert, integrated into or attached to an insert or insert, such as a dummy saw blade. The optical marker can also be attached or fixed to a saw blade. The optical marker can include text or alphanumeric codes for the surgeon that specify a specific surgical step, such as a specific surgical step, 1F-distal femoral cut, 2F-anterior femoral cut, 1T-proximal tibial cut, 2T-tibial keel punch, etc. The optical marker can also include one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics and other codes or patterns including bar codes and QR codes or combinations thereof. One or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics and other codes or patterns (including bar codes and QR codes or combinations thereof) can be specific to a surgical step, such as corresponding to a letter or alphanumeric code representing the surgeon's surgical step. When the optical marker enters the field of view, an image and / or video capture system integrated into, attached to, or separate from the optical head mounted display can detect one or more specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof; image processing and / or pattern recognition software can be used to recognize the specific geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns, for example, to command the display of a corresponding virtual surgical step in the optical head mounted display superimposed on a surgical field having a surgeon's perspective consistent with the surgical field or the target anatomy or bone cut. When the cutting block or guide, insert (e.g., dummy saw blade), or physical saw blade with the optical marker is removed, the image and / or video capture system can detect that the optical marker is no longer in the field of view, triggering a command to turn off the optical head mounted display or to complete the display of the surgical step or to switch to the display of the next surgical step and the corresponding virtual display.

[0551] In some embodiments of the invention, one or more optical markers (which are selected at angles, e.g., 90 degrees or less or greater or parallel to the axis or on the axis) can be included in, integrated into, or attached to the cutting block or guide.

[0552] In some embodiments, one or more optical markers can be used in conjunction with spinal surgery, such as vertebroplasty, kyphoplasty, posterior spinal fusion, anterior spinal fusion, lateral spinal fusion, and / or disc replacement. For example, one or more optical markers can be included in, integrated into, or attached to a needle, nail, awl, probe, ball-handled probe, straight probe, curved probe, tap, ratchet, screwdriver, rod template, rod inserter, rod clamp, tube bender, plug starter, compressor, retractor, circuit breaker, obturator, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, caliper, plate holder, plate bender, forceps, etc. The foregoing list is merely exemplary and should not be construed as limiting the present invention. One or more optical markers may be used to designate the patient's left side and the patient's right side, and / or to designate a spinal segment of the patient, such as one or more geometric shapes, geometric patterns, letters, numbers, alphanumeric and other codes or patterns detected using an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display and recognized using image processing and / or pattern recognition.

[0553] The location, position, orientation, alignment and / or orientation of a needle, pin, awl, probe, ball-handled probe, straight probe, curved probe, screw, ratchet, screwdriver, rod template, rod inserter, rod clamp, bender, plug starter, compressor, retractor, circuit breaker, obturator, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, calipers, plate holder, plate bender, tweezers, milling cutter, saw, reamer, broach, impactor, cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant components can be determined using an image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display. For example, after initial or subsequent registration of the patient, the surgical site, the optical head-mounted display, and optionally, an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display, and the patient's virtual data and / or real-time data, the image and / or video capture system can detect an optical marker included in, integrated into, and / or attached to a surgical instrument. Because the location, position, alignment, and / or orientation of the optical marker on the surgical instrument is known, and the dimensions (e.g., at least one of them) or geometry of the surgical instrument is known, the image and / or video capture system can track the location, position, orientation, alignment, and / or direction of movement of the optical marker and the surgical instrument.

[0554] In another example, two or more optical markers can be integrated into or attached to different, optionally defined locations along the long axis of a needle, nail, awl, probe, ball-handled probe, straight probe, curved probe, screw, ratchet, screwdriver, rod template, rod inserter, rod clamp, bender, plug starter, compressor, retractor, circuit breaker, obturator, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, caliper, plate holder, plate bender, forceps, milling cutter, saw, reamer, broach, impactor, cutting or drilling block, and / or other surgical instrument and / or trial implant and / or implant assembly, such as an instrument or trial implant or implant assembly in a knee replacement or hip replacement. The image and / or video capture system can detect the two or more optical markers and can determine their respective locations. By positioning two or more optical markers captured and defined by an image and / or video capture system, the long axis of a needle, pin, cone, probe, tap, milling cutter, saw, reamer, broach, impactor and / or surgical instrument and / or trial implant and / or implant assembly can be determined; other axes can be determined in addition to or instead of the long axis. With the positions of optical markers on the needle, pin, cone, probe, tap, mill, saw, reamer, broach, impactor and / or other surgical instrument and / or trial implant and / or implant component known, the major axis or other axis of the needle, pin, cone, probe, tap, mill, saw, reamer, broach, impactor and / or other surgical instrument and / or trial implant and / or implant component known, and the dimensions of the needle, pin, cone, probe, tap, mill, saw, reamer, broach, impactor and / or other surgical instrument and / or trial implant and / or implant component known, portions of the needle, pin, cone, probe, tap, mill, saw, reamer, broach, impactor and / or trial implant and / or implant component hidden by tissue, e.g., beneath the skin and / or inside muscle, can be estimated and optionally displayed by the optical head mounted display in addition to the virtual or intended path or projected path or any other aspect of the virtual surgical plan. Instead of using two or more optical markers in the aforementioned embodiments, an optical marker that is long enough or wide enough or deep enough can also be used to define one or more axes of a needle, pin, cone, probe, tap, milling cutter, saw, reamer, broach, impactor and / or other surgical instrument and / or trial implant and / or implant assembly.

[0555] Optionally, when two or more optical markers are used that are included in, integrated into, or attached to a surgical instrument, the optical markers can be arranged at the same angle, e.g., parallel or on the same axis, or at different angles, e.g., orthogonal or non-orthogonal angles. This can be particularly useful when the optical markers include one or more of geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including bar codes and QR codes or combinations thereof. By arranging the optical markers and any associated geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns (including bar codes and QR codes or combinations thereof) in this manner, the angular orientation of the surgical instrument can be determined more accurately. For example, a first optical marker on a surgical instrument may be partially visible or not visible at all due to its angular orientation when viewed from a certain viewing angle of an image and / or video capture system integrated into or attached to an optical head-mounted display, and the viewing angle of the second optical marker from the image and / or video capture system integrated into or attached to the optical head-mounted display may allow for a complete or more complete display of one or more geometric shapes, geometric patterns, letters, numbers, alphanumerics, and other codes or patterns including barcodes and QR codes or combinations thereof when a second optical marker is oriented at a different angle, position, and / or orientation on the same surgical instrument, thereby allowing for a more accurate determination of the angular orientation of the second optical marker and, thereby, the surgical instrument. Furthermore, the respective projections of the first and / or second optical markers measured by the image and / or video capture system, when using two or more cameras, may be selected with any parallax information pairing (e.g., one positioned near the left eye and the other positioned near the right eye), which may be used to more accurately determine their relative positions and the position of the surgical instrument.

[0556] An image and / or video capture system integrated with, connected to, or detached from an optical head-mounted display detects an optical marker contained in, integrated into, or attached to: a needle, nail, awl, probe, ball-handled probe, straight probe, curved probe, screw, ratchet, screwdriver, rod template, rod inserter, rod clamp, bender, plug starter, compressor, retractor, breaker, obturator, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, caliper, plate holder, plate bender, forceps, milling cutter, saw, reamer, broach, impactor, cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant components, and triggers a command to display the needle, nail, awl, probe, ball-handled probe, straight probe, curved probe, screw, ratchet, screwdriver, rod template, rod inserter, rod clamp, bender, plug starter, compressor, retractor, breaker, obturator, anti-torque, quick joint, driver, retractor, retraction frame, implant positioner, caliper, plate holder, plate bender, forceps, milling cutter, saw, reamer, broach, impactor, cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant components when the optical marker enters the surgeon's field of view , a rod template, a rod inserter, a rod clamp, a bender, a plug starter, a compressor, a retractor, a circuit breaker, an obturator, a counter torque, a quick joint, a driver, a retractor, a retraction frame, an implant positioner, a caliper, a plate holder, a plate bender, a forceps, a milling cutter, a saw, a reamer, a broach, an impactor, a cutting or drilling block, and / or other surgical instruments and / or trial implants and / or implant components, for example, with an intended position, location and / or alignment and / or orientation for an intended surgical step; when an optical marker of a surgical instrument leaves the surgeon's field of view, the image and / or video capture system can detect it, triggering a command to stop displaying the predetermined path or virtual display of the surgical instrument or other aspects of the virtual surgical plan, optionally switching to the next surgical step and corresponding virtual display. In spinal surgery and selected other surgeries, the next surgical step can involve the same side of the same spinal segment of the patient or the opposite side of the same spinal segment of the patient, wherein the virtual display corresponding to the next surgical step for a given level and side can be initiated by the OHMD display. The next surgical step may involve the same or opposite side of an adjacent or different spinal segment of the patient, wherein a corresponding virtual display of the next surgical step for a given level and side may be initiated by the optical head mounted display display.

[0557] The optical marking may include one or more two-dimensional codes. The two-dimensional code may be part of a geometric pattern or geometric figure included in the optical marking or may be embedded therein. The optical marking may be a two-dimensional code.

[0558] If the optical marker is attached to a surgical instrument, the attachment can occur at a defined location and / or position and / or alignment, such as at the end of the surgical instrument. The accessory can include an opening with a stop to define the location and / or position and / or alignment of the optical marker on the surgical instrument. For example, the optical marker is provided with an opening with a stop that is large enough to accommodate the end of a nail or drill bit facing the surgeon, such as for insertion into a spinous process or a facet joint or a portion of a pedicle. By using this type of accessory and other accessories to secure the marker at a defined location, position and / or orientation on the surgical instrument, an image and / or video capture system can detect the optical marker and its location, position and / or orientation, which can be used to determine the location, position and / or orientation of the surgical instrument, such as a needle, due to its defined spatial relationship and the known geometry of the surgical instrument, including its tip or front portion within the patient's body.

[0559] In some embodiments of the present invention, optical markers can be used to determine or identify the location, position, orientation, alignment, size, axis or axes, plane or planes of a surgical change. For example, if a bone cut is made during a surgical procedure, one or more optical markers can be attached to the cut bone to determine its location, position, orientation, alignment, size, shape, geometry, axis or axes, plane or planes, or one or more thereof. For example, one, two or more optical markers can be placed around the perimeter or edge of the cut bone or surgical change; an image and / or video capture system integrated into, connected to, or separate from the optical head-mounted display can detect the location, position, and / or orientation of the optical markers, and software can be used, for example, to analyze the location, position, and / or orientation information of the optical markers to obtain information about the perimeter and / or edge and / or shape of the cut bone or surgical change. One, two, or more optical markers can be placed near or on the bone cut or surgical change; an image and / or video capture system integrated into, connected to, or separate from the optical head-mounted display can detect the location, position, and / or orientation of the optical markers, and software can be used, for example, to analyze the location, position, and / or orientation information of the optical markers to obtain information about the shape or geometry of the cut bone or the surgical change. If the bone cut is planar, one or more optical markers on the surface facing the planar bone or one or more optical markers attached to the surface facing the planar bone of a carrier or instrument (e.g., a plastic sheet) can be held against, fixed, or attached to the bone cut surface; an image and / or video capture system integrated into, connected to, or separate from the optical head-mounted display can then be used to detect the one or more optical markers, and software can be used, for example, to analyze the location, position, and / or orientation information of the one or more optical markers to obtain information about the location and / or position and / or orientation and / or alignment of the bone cut plane, including in relation to other anatomical landmarks and / or other optical markers. The carrier or instrument for the optical marker can be transparent or translucent so that the surgeon can check or confirm that the carrier or instrument and the attached optical marker are flush with the bone cut, e.g., the bone plane, before determining or confirming the plane of the bone cut. Once the bone cut plane has been determined or confirmed in this manner, the optical marker attached to the bone cut and / or the determined plane of the bone cut can be used to plan the next surgical change, e.g., the next fracture or surgical change, e.g., anterior or posterior femoral cut after distal femoral cut in knee replacement, or cutting a chamfer after anterior and posterior femoral cut after knee replacement, or cutting an opposing joint surface. By determining, confirming, and / or referencing previous surgical changes, e.g., bone cuts, in this manner, the accuracy of subsequent surgical steps can be improved, thereby ultimately improving the overall accuracy of the surgical procedure.

[0560] In some embodiments of the present invention, one or more optical markers can be attached or fixed to the patient's thigh or distal femur. For example, one or more optical markers can be attached to the skin of the distal thigh, for example, above the knee joint space. The attachment can be performed using, for example, an adhesive that attaches the one or more optical markers to the patient's skin. The one or more optical markers are optionally sterile. The one or more optical markers are optionally magnetic. In this example, a magnetic base is optionally attached to the patient's skin, for example, using an adhesive. A transparent, translucent, or opaque surgical drape can then be placed on the magnetic base, and the magnetic optical markers can then be attached to the magnetic base attached to the patient's skin. Alternatively, once the skin incision is made, the one or more optical markers can be rigidly attached to one or more bones, such as the distal femur and / or proximal tibia. The rigid attachment can be accomplished using nails or screws or other attachment mechanisms.

[0561] An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of one or more optical markers, for example, when the leg is in a neutral position and / or an extended position and / or any other position, including any position or a position selected by the surgeon and / or operator. The surgeon and / or operator can then move the leg and thigh to a plurality of different positions and / or orientations and / or alignments, and / or the surgeon and / or operator can move the leg and thigh in a circular or semi-circular manner. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of a plurality of optical markers with a plurality of different positions and / or orientations and / or alignments of the leg or thigh, and / or register the position and / or location and / or orientation and / or alignment of a plurality of optical markers during different circular or semi-circular movements. The resulting information can be used to determine the center of rotation, which in this example may be the center of the hip joint.

[0562] In some embodiments, an ankle clamp can be applied to the ankle of the patient's leg. The ankle clamp can include one or more optical markers, including one or more QR codes. The ankle clamp and / or optical markers can be disposable. Using an image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display, the ankle clamp and integrated or attached optical markers can be used to determine the position of the medial and lateral malleolus, for example, using the center or 1 / 3 or 2 / 3 distance points between the ankles of the ankle joint. Alternatively, one or more optical markers can be applied to the medial and / or lateral malleolus. In some embodiments of the present invention, a magnetic base can be secured to the medial and lateral malleolus. The ankle can then be prepared and draped using sterile techniques, and one or more sterile magnetic optical markers can be applied to a drape or surgical cover, with the built-in drape or surgical cover secured to the magnetic base using the one or more optical markers. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can then be used to identify optical markers on the medial and lateral malleolus and the center, 1 / 3, or 2 / 3 distance points of the ankle joint.

[0563] After determining the center of the hip joint using one or more optical markers at the thigh or distal femur, and determining the center or 1 / 3 or 2 / 3 distance point of the ankle joint using an ankle clamp and / or one or more optical markers, the system can obtain the patient's mechanical axis and plan surgical intervention, such as correcting varus or valgus deformity with corresponding femoral and / or tibial and / or talar cutting, and then projecting it using an optical head-mounted display.

[0564] In some embodiments of the present invention, one or more optical markers can be attached or fixed to the patient's arm. For example, one or more optical markers can be attached to the skin of the upper arm or thigh, such as above the elbow. The attachment can be performed using, for example, an adhesive that attaches the one or more optical markers to the patient's skin. The one or more optical markers are optionally sterile. The one or more optical markers are optionally magnetic. In this example, a magnetic base is optionally attached to the patient's skin, such as using an adhesive. A transparent, translucent, or opaque surgical drape can then be placed on the magnetic base, and the magnetic optical markers can then be attached to the magnetic base attached to the patient's skin. Alternatively, once the skin incision is made, the one or more optical markers can be rigidly attached to one or more bones, such as the proximal humerus. The rigid attachment can be accomplished using nails or screws or other attachment mechanisms.

[0565] An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of one or more optical markers, such as when the arm is in a neutral position and / or an extended position and / or an abducted position and / or any other position, including any position or positions selected by the surgeon and / or operator. The surgeon and / or operator can then move the arm to a plurality of different positions and / or orientations and / or alignments, and / or the surgeon and / or operator can move the arm in a circular or semi-circular manner. An image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display can register the position and / or location and / or orientation and / or alignment of a plurality of optical markers with a plurality of different positions and / or orientations and / or alignments of the arm, and / or register the position and / or location and / or orientation and / or alignment of a plurality of optical markers during different circular or semi-circular movements. The resulting information may be used to determine a center of rotation, which in this example may be the center of rotation of the shoulder joint.

[0566] In some embodiments of the present invention, one or more optical markers can be attached to an operating room (OR) table. If the optical marker is parallel to the table, a single marker is sufficient to determine the principal plane of the table, e.g., a horizontal plane, which can be the plane in which the patient is positioned, e.g., supine, prone, lateral, or oblique, or other positions known in the art. This can be assisted by using optical markers that include surfaces or planes that are parallel or perpendicular to the table, or at a defined angle, and that are large enough to be detected by a camera, image, or video capture system that is integrated into, attached to, or separate from an optical head-mounted display. For example, such a plane of the optical marker can measure 1×1 cm, 2×2 cm, 2×3 cm, 4×4 cm, 4×6 cm, etc. Alternatively, multiple (e.g., two, three, or more) optical markers can be used to determine the plane, specifically by having markers corresponding to the principal plane of the table or parallel to the principal plane of the table, or, for example, perpendicular to the table, or, for example, at an angle to the table. If the operating table is covered by a surgical drape, one or more magnetic or attachable bases can be attached to the operating table before the drape is placed. After the drape is placed, one or more magnetic or attachable optical markers can be attached to the magnetic base or attachment mechanism with the inserted surgical drape. Alternatively, one or more retaining arms or extenders of known geometry can be attached to the operating table, and one or more optical markers can be attached to or integrated into the retaining arms or extenders. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can then identify the location, position, orientation, and / or alignment of the one or more optical markers. The resulting information can be used to determine the principal plane of the operating table on which the patient is lying. The image and / or video capture system integrated into or attached to the optical head-mounted display can be used to reference the one or more optical head-mounted displays with respect to the operating table and / or the attached optical markers. Once the principal plane of the operating table is determined in the system, virtual surgical steps can be planned in the patient's virtual surgical plan with respect to the principal plane of the operating table. For example, one or more bone cuts can be planned and / or performed perpendicular to the principal plane of the operating table, e.g., with the patient in a supine or prone position or any other desired position. One or more bone cuts can be planned and / or performed at defined angles other than 90 degrees relative to the horizontal plane of the operating table, e.g., with the patient in a supine or prone position or any other desired position. One or more bone cuts can be planned and / or performed in non-orthogonal planes or directions relative to the principal plane or horizontal plane of the operating table, e.g., with the patient in a supine or prone position or any other desired position, optionally with reference to a plane displayed by the optical head mounted display that is perpendicular to the operating table.The principal planes of the operating table can be used as a reference in this manner, including for comparing or referencing virtual patient data and real-time patient data, and including for comparing or referencing a virtual surgical plan. Bone cuts at orthogonal or non-orthogonal angles (e.g., relative to the operating table or relative to the patient's anatomy, anatomical landmarks, anatomical axes, or biomechanical axes) can be performed using one or more virtual surgical guides or cutting blocks and / or one or more physical surgical guides or cutting blocks. The virtual surgical guide or cutting block can include one or more dimensions that correspond to those of the physical surgical guide or cutting block.

[0567] One or more optical markers attached to or linked to the operating table can also serve as a fixed reference for one or more optical head-mounted displays during a surgical procedure. This can be useful, for example, when the patient and / or limb and / or surgical site moves during surgery. A fixed reference to the operating table can help maintain registration of the one or more optical head-mounted displays with the virtual surgical plan and real-time patient data and / or operating room data.

[0568] In some embodiments of the present invention, one or more optical markers may be placed in the surgical field and / or in an area remote from the surgical field or attached to the patient. An image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display may be used to identify the one or more optical markers and determine their location, position, orientation, and / or alignment. The image and / or video capture system may also optionally determine the location, position, orientation, and / or alignment of one or more optical markers attached to or referenced to the operating table. The system may reference the coordinates and / or spatial relationship of one or more optical markers attached to the patient in the surgical field and / or in an area remote from the surgical field and one or more optical markers attached to or used as a reference to the operating table. In this way, if the patient's body moves during surgery, such as during proximal femoral broaching or acetabular reaming in hip replacement, or during impaction of the femoral or tibial components in knee replacement, or during nailing or cutting of bone, or during placement of a spinal device (e.g., a fusion cage or pedicle screw), movement between one or more optical markers attached to the patient in the surgical field and / or in areas remote from the surgical field and one or more optical markers attached to or referenced to the operating table, as well as changes in the coordinates of one or more optical markers attached to the patient in the surgical field and / or in areas remote from the surgical field, can be detected, and the amount, direction, and magnitude of movement can be determined; for example, the resulting information can be used to update or adjust or modify the virtual surgical plan or update or adjust or modify the virtual surgical plan or virtual surgical steps or the display of the virtual display for the patient's movement, including by updating, moving, or adjusting one or more aspects and / or portions of the virtual surgical plan, including one or more virtual surgical steps. The virtual surgical instrument comprises: a virtual surgical guide or cutting block, a virtual trial implant, a virtual implant assembly, a virtual implant or virtual device, a predetermined starting point, a predetermined starting position, a predetermined starting orientation or alignment, a predetermined intermediate point, a predetermined intermediate position, a predetermined intermediate orientation or alignment, a predetermined end point, a predetermined final position, a predetermined end orientation or alignment, a predetermined path, a predetermined plane, a predetermined cutting plane, a predetermined profile or cross-section or surface feature or shape or projection, a predetermined depth mark or depth gauge, a predetermined angle or orientation or rotation mark, a predetermined axis, such as a rotation axis, a bending axis, an extension axis, a predetermined axis of the virtual surgical tool, a virtual surgical instrument (including a virtual surgical guide or cutting block, a virtual trial implant, a virtual implant assembly, a virtual implant or device), one or more of a predetermined tissue change or variation for one or more devices or implants or implant assemblies or surgical instruments or surgical tools, and / or using new patient coordinates or new coordinates of the surgical field.

[0569] In some embodiments of the present invention, portions or the entire optical marker may be radiopaque, such that the optical marker may also be visible in radiographs or other imaging examinations utilizing ionizing radiation, including fluoroscopy, digital tomosynthesis, cone-beam computed tomography, and / or computed tomography. Different levels or degrees of radiopacity may be present in different portions or regions of the optical marker. Different levels or degrees of radiopacity may be utilized to encode information. For example, different levels of radiopacity may be used to encode information stored in an optically readable alphanumeric code, barcode, or QR code. The different levels of radiopacity may optionally be arranged as thickness profile bars, which may optionally mirror some or all of the information contained in the barcode. The different levels of radiopacity may optionally be arranged as dots or a grid, which may optionally mirror some of the information contained in the QR code. Different radiopacities may be achieved by varying the thickness of a metal (e.g., lead). Radiopaque optical markers encoding information in this manner may be manufactured using a 3D metal printer. They may also be machined using a CNC machine, e.g., from rod stock or ingot.

[0570] The radiopaque portion of the optical marker may include laterality information, such as left for left and right for right, visible on a radiograph, such as by different material thicknesses, such as lead; the same information may be contained in an additional alphanumeric code or text, a bar code, or a QR code, readable by a bar code or QR code reader or an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display. The radiopaque portion of the optical marker may include anatomical site information, such as L5 or L4, T1 or T2, C3 or C7, knee, hip, visible on a radiograph, such as by different material thicknesses, such as lead; the same information may be contained in an additional alphanumeric code or text, a bar code, or a QR code, readable by a bar code or QR code reader or an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display. Image processing techniques and / or software can be applied to radiographic information including optical markers and radiographically encoded information such as transverseness and / or position, and the information included in the radiograph can be compared with the information included on the optical scan. If any discrepancies are detected, an alarm can be triggered, such as by displaying an alarm in an optical head-mounted display.

[0571] Multiple partially or fully radiopaque optical markers can be used. Radiopaque optical markers can be applied at different locations and in different planes around the surgical site. In spinal surgery, for example, one, two, three, or more radiopaque optical markers can be applied to the skin surrounding the spinal segment for the intended procedure; one, two, three, or more radiopaque optical markers can be attached to a pin, drill, or screw inserted into the spinous process and / or pedicle or other spinal element; and one, two, three, or more radiopaque optical markers can be applied to the patient's flank or abdomen. In hip replacement surgery, one, two, three or more radiopaque optical markers can be applied to the anterior superior iliac spine on the patient's intended surgical side, for example, by adhesively bonding to the skin or attaching to a pin or attaching a drill to the bone; one, two, three or more radiopaque optical markers can be applied to the anterior superior iliac spine on the patient's contralateral side, for example, by adhesively bonding to the skin or attaching to a pin or attaching a drill to the bone; one, two, three or more radiopaque optical markers can be attached to the pubic symphysis, for example, by adhesively bonding to the skin or attaching to a pin or attaching a drill to the bone; one, two, three or more radiopaque optical markers can be applied to the acetabulum attached to the patient's intended surgical side, for example, by attaching a pin or drill to the bone. The accuracy of three-dimensional spatial registration and cross-referencing of optical markers in different modalities (e.g., radiographs, image capture) can be increased by using multiple radiopaque optical markers at multiple different locations and in different planes around a surgical site, for example, by obtaining multiple x-rays at different angles, such as anteroposterior, lateral, and / or oblique, from multiple viewpoints, and / or by using an image and / or video capture system integrated into, connected to, or separate from an optical head-mounted display to scan and image the radiopaque optical markers from multiple viewpoints. The accuracy of three-dimensional spatial registration of optical markers can be increased by using multiple optical markers at multiple different locations and in different planes around a surgical site, for example, by using an image and / or video capture system integrated into, connected to, or separate from an optical head-mounted display to scan and image the optical markers from multiple viewpoints. Additionally, the accuracy of registration can be better maintained when the viewpoints or radiographic angles change, for example, during a surgical procedure or due to patient movement.

[0572] In some embodiments of the present invention, system performance can be tested. For example, the system performance test can measure a phantom including two or more optical markers at a known location, position, orientation and / or alignment. Using the known coordinates of the two or more optical markers and the distance and angle between the markers, the accuracy of the distance measurement and / or angle measurement and / or area measurement and / or volume measurement can be determined using an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display in the following manner. In addition, by repeated measurements, the reproducibility and / or accuracy of the distance measurement and / or angle measurement and / or area measurement and / or volume measurement can be determined using an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display. Using an image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display, the accuracy, reproducibility, and / or precision of distance, angle, area, and / or volume measurements under static and dynamic conditions can be determined. Static conditions may be conditions where the patient, spine, limb, joint, and / or skeleton are not moving. Dynamic conditions may be conditions where the patient, spine, limb, joint, and / or skeleton are moving during image capture. For example, dynamic conditions can be used to determine the center of rotation of a joint. Measurements of static and dynamic conditions can be performed for different viewing angles and distances using the image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display. Measurements of static and dynamic conditions can be performed when the optical head-mounted display is stationary, rather than when it is moving. Measurements of static and dynamic conditions can be performed when the optical head-mounted display is not stationary, rather than when it is moving, such as with the operator's head.

[0573] Table 5 illustrates exemplary tests with various combinations of test conditions and test parameters in which the accuracy and reproducibility and / or precision of measurements may be determined. Any combination is possible. Other parameters may be measured, such as reproducibility of color temperature (e.g., in Kelvin). Other statistical tests may be applied. All measurements and all statistical determinations and parameters may be evaluated under static, dynamic, optical head-mounted display stationary, and optical head-mounted display moving conditions, including at various angles and distances between the image and / or video capture system and the target anatomy and / or test device and / or phantom.

[0574]

[0575]

[0576] Once the accuracy and / or reproducibility and / or precision of distance measurements and / or angle measurements and / or area measurements and / or volume measurements and / or coordinate measurements are determined using an image and / or video capture system integrated into, attached to, or separate from an optical head-mounted display, thresholds can be defined to indicate when the system is operating outside a clinically acceptable performance range. Standard statistical methods known in the art can be used to determine the thresholds. For example, when the angle of view and / or distance or movement speed indicator measurements of the image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display fall outside two standard deviations of the system performance comprising the overall system performance, a warning can be triggered to the surgeon that the display of virtual data, such as a portion of a virtual surgical plan, a virtual projection path, or a virtual plane (e.g., a virtual cutting plane), may be inaccurate. A binary (e.g., yes, no) system can be used to trigger an alert when the image and / or video capture system and / or OHMD display is operating outside a clinically acceptable performance range, such as exceeding a certain angle of view, exceeding or falling below a certain distance to the target anatomy, or exceeding an acceptable movement speed. Alternatively, a sliding scale can be used as the system gradually enters a range outside of a clinically acceptable performance range. The sliding scale can be a color scale ranging from green to red, with mixed colors in between. The sliding scale can be an acoustic signal that further increases in intensity or frequency when the system operates outside of a clinically acceptable range. The sliding scale can be a vibration signal that further increases in intensity or frequency when the system operates outside of a clinically acceptable range. In some embodiments of the present invention, when one or more test data indicates that the system is operating outside of its clinically acceptable performance range, the optical head-mounted display can optionally turn off the display of patient virtual data, such as virtual planning information, a virtual surgical guide or cutting block, or a virtual plane or intended path. When the test data indicates that the system is again operating within a clinically acceptable performance range, the OHMD display can be turned back on. System testing, including accuracy and reproducibility testing, can be performed intermittently, for example, every 3 seconds, 5 seconds, 10 seconds, 20 seconds, 30 seconds, 1 minute, 2 minutes, etc. System testing can also be performed continuously. System testing can be performed intermittently or continuously, but only during the time the optical head-mounted display is displaying virtual data. System testing can be performed intermittently or continuously, but only during surgical steps that require high precision or reproducibility. These steps requiring high precision or reproducibility can be identified by the surgeon through voice commands or other commands, or they can be identified in the virtual surgical plan, for example, automatically or by the surgeon's selection.

[0577] In some embodiments of the present invention, radiopaque and non-radiopaque optical markers are optionally attached or applied to an extender that increases the distance of the optical marker from the patient's skin. For example, such an extender can be anchored in the spinous process, pedicle or other spinal element by a nail, drill or screw. The use of an extender with attached radiographic optical markers can increase the registration accuracy between radiographic data and image capture data, for example, when using anteroposterior and lateral radiographs. When using image capture, the use of an extender with attached optical markers can illustrate the definition of anatomical axes or instrument axes and other information. When two or more markers are used with an extender and the distance separating the markers is greater than the spatial resolution of the image and / or video capture system, the accuracy of determining the axis between the two markers, for example, the length of the extender and the distance between the markers, can be increased.

[0578] Optical markers can be visualized using other imaging modalities, such as nuclear magnetic resonance, nuclear scintigraphy, single photon emission tomography, or positron emission tomography. For example, optical markers can be doped with nuclear magnetic resonance contrast agents, such as gadolinium-DTPA, to make them visible by nuclear magnetic resonance. Optical markers can be doped with isotopes or positron emitters to make them visible by single photon emission tomography or positron emission tomography.

[0579] When the optical marker includes a QR code or when a QR code is used as an optical marker, it can also address inventory management issues and quality issues before, during and after surgery. Errors in the surgical side of the patient are common quality issues associated with surgery and can have devastating consequences for the patient. Similarly, in spinal surgery, operating on the wrong spinal segment can result in serious injury to the patient. Optical markers used to determine the location, position, orientation, alignment and / or direction of travel of a patient, limb, joint, surgical site, surgical instrument, trial implant and / or implant component can also include any of the following information, such as using a bar code or QR code contained in or attached to the optical marker:

[0580] - Patient identifier

[0581] - Patient demographics, such as age, sex, height, and body mass index (BMI)

[0582] -Patient medical records

[0583] - Patient risk factors

[0584] -Patient allergies

[0585] - The side of the body to be operated on, for example, left vs. right

[0586] - The area to be operated on, e.g. knee and hip, spinal segments L1 and L2, etc.

[0587] - Spinal segment to be operated on

[0588] -Part of virtual surgery planning, e.g.

[0589] ○Resection amount,

[0590] ○ Level of resection for a given surgical step,

[0591] ○ Location and / or direction of bone cuts

[0592] ○The slope of the tibial cut,

[0593] ○ Implant rotation, e.g., femoral component rotation, tibial component rotation

[0594] ○ Implant flexion, e.g., femoral component flexion

[0595] ○Predetermined depth, location, position, orientation, and coordinates of deburring

[0596] ○Predetermined depth, location, position, orientation, and coordinates of the reaming hole

[0597] ○ Expected depth, location, position, orientation, coordinates of milling

[0598] ○ Femoral neck notch angle,

[0599] ○Acetabular angle,

[0600] ○ Acetabular anteversion,

[0601] ○ Thigh tilted forward,

[0602] ○Offset

[0603] ○Femoral shaft

[0604] ○ Intended implant component axis / alignment

[0605] ○ Intended polyethylene composition and thickness (e.g., hip acetabular liners, knee tibial inserts, shoulder glenoid inserts)

[0606] - Information about templates or dimensions

[0607] o Dimensions of selected implant components, e.g., knee femoral, tibial, or patellar components, hip flexor shells, acetabular liners, femoral stems, femoral heads, and femoral necks with removable bearing components

[0608] ○One side of the implant component, left and right

[0609] -Inventory management information, such as

[0610] ○Version, type, and model of the device used

[0611] ○Batch number of the device used

[0612] ○The manufacturing location of the equipment used

[0613] ○Manufacturing date of the device used

[0614] ○The first sterilization date of the instrument used

[0615] ○ Number of sterilization cycles applied to the device used

[0616] ○The last sterilization date of the instrument used

[0617] ○Date the device was delivered to the hospital or surgical center

[0618] ○ Version, type, and model of the implant component used

[0619] ○Batch number of the implant component used

[0620] ○The manufacturing location of the implant components used

[0621] ○Manufacture date of the implant components used

[0622] ○Sterilization date of the implant components used

[0623] ○Date the implant components were delivered to the hospital or surgical center

[0624] ○Any other information related to inventory management

[0625] Optionally, some of the QR codes included in this information may also be separated from the optical markers. In some embodiments of the present invention, a separate bar code and / or QR code reader may be used to read the information contained on the bar code and / or QR code before, during, and / or after surgery. In some embodiments of the present invention, an image and / or video capture system integrated into an optical head-mounted display, attached to an optical head-mounted display, or separated from an optical head-mounted display may be used to read the information contained on the bar code and / or QR code. Then, for example, the information read from the bar code and / or QR code may be compared with portions of the virtual surgical plan and / or the sides of the patient prepared for surgery (e.g., left side vs. right side), the patient site prepared for surgery, e.g., spinal segments L4 vs. L5 (e.g., as seen in radiographs), the physical surgery performed, the physical instruments selected, the physical implant trials selected, and the physical implant components selected.

[0626] When a pin or screw is placed in a surgical site including a joint and / or bone, such as in the spinous process or pedicle of a spinal segment, having an integrated or attached optical marker with a QR code, or when an instrument, a trial implant and / or an implant assembly having an integrated or attached optical marker with a QR code enters the field of view of a bar code and / or QR code reader and / or an image and / or video capture system integrated or attached to an optical head-mounted display, or enters the surgical field or vicinity of surgically altered tissue, the information on the bar graph can read the code or QR code on the physical pin or screw, the physical instrument, the physical trial implant and / or the physical implant assembly and compare it with the expected surgical site information and / or expected laterality information and / or the virtual surgical plan and / or expected size information and / or expected template information. In the example of spinal segments, a barcode and / or QR code reader and / or an image and / or video capture system integrated or attached to an optical head-mounted display can read a QR code (left or right) identifying the intended spinal segment and the side (left or right) for the nail, pedicle screw, or other device. This information can be compared with the patient's virtual surgical plan and / or X-ray information. For example, the system can use intraoperative X-rays to automatically or semi-automatically or user-operatedly identify the spinal segment (e.g., counting from the sacrum upward), for example, by detecting the sacral endplate and the opposing endplate and / or pedicle. If the system detects a discrepancy in spinal segment or laterality between the information read from the nail, screw, or device and the integrated or attached optical marker and barcode or QR code and the virtual surgical plan and / or radiographic information, an alarm can be triggered to check the surgical plan and / or reconfirm the spinal segment and / or side. The above examples are not limited to radiographic information; other imaging tests known in the art, such as computed tomography, magnetic resonance imaging, etc., can be used to determine or identify anatomical sites and sides, including spinal segments.

[0627] If the reading of the QR code indicates a discrepancy in any of the information embedded in the QR code, e.g., the location, laterality, segment, multi-part or multi-aspect, size or template information of the virtual surgical plan, compared to the physical real-time data during the surgery, e.g., the physical location or spinal segment or laterality of the inserted nails or screws, the physical instruments used, the physical trial implants used and / or the physical implant components used, an alarm can be triggered, e.g., in an optical head mounted display or on a computer monitor used to plan, display or modify the virtual surgical plan. The alarm can be visual, e.g., a red warning sign or stop sign or a displayed alarm sign, or sound, or vibration, or a combination thereof. Any other alarm known in the art can be used.

[0628] For example, when a surgeon is operating on a patient to replace the patient's left knee, one or more implant components or attached retainers or packaging labels or sterile packaging may include an optical indicia in the form of a QR code. The QR code may indicate laterality, e.g., left femoral component versus right femoral component. If a scrub technician accidentally implants the surgeon's right femoral component into the patient's left knee, an image and / or video capture system integrated into or attached to an optical head-mounted display worn by the surgeon can read the QR code as the surgeon removes the femoral component and when the femoral component, with the attached optical indicia and QR code, enters the surgeon's field of view or near the surgical field. The image and / or video capture system and associated system software can read the QR code to identify whether the implant component is for the right knee. The system software can then compare this information with the patient's virtual surgical plan or a template and / or sizing information indicating a left knee plan, triggering an alarm indicating that an incorrect femoral component has entered the field of view or that the surgeon has entered the vicinity of the surgical field, e.g., as distinguished by another optical indicia. The alarm can assist the surgeon in correcting the error by switching to the correct side component.

[0629] In another example, when the surgeon operates on the patient to replace the patient's left knee, one or more implant assemblies or attached retainer or packaging label or sterile packaging may include the optical markings of a Quick Response Code.Quick Response Code can indicate the size of an implant assembly, such as, size 5 or 6 or other femoral components, or size 5 or 6 or other tibial components, or size 2 or 3 or other patellar components. If scrub technician accidentally hands the surgeon the femoral component (being templated as the femoral component having a size of 6) that is transplanted into the patient's left knee, the surgeon may then wear an image and / or video capture system that is integrated into or is attached to an optical head-mounted display and can, when the surgeon takes the femoral component, and when the femoral component with attached optical markings and Quick Response Code enters the surgeon's field of view or near the operating field, read the Quick Response Code. The image and / or video capture system and associated system software can read a QR code identifying an implant component size 4; the system software can then compare this information with the patient's virtual surgical plan or template and / or sizing information, which can indicate that a femoral component size 6 has been planned, and then trigger an alarm that an incorrect femoral component has entered the surgeon's field of view or has entered the vicinity of the surgical field (e.g., as distinguished by another optical marker). The alarm can assist the surgeon in correcting the error by switching to the correct size component.

[0630] An image and / or video capture system and / or a barcode and / or QR code reader integrated into, attached to, or separate from an optical head-mounted display can also be used to read embedded information about virtual surgical instruments and / or implant components for inventory management and billing and invoicing purposes. For example, the image and / or video capture system and / or barcode and / or QR code reader can detect which instruments are used, monitor their frequency of use, and when a certain recommended frequency of use is reached, the system can trigger an alert to send the instruments for repair. In some embodiments, the image and / or video capture system and / or barcode and / or QR code reader can detect which instruments are used and trigger an alert to send the instruments for repair. In some embodiments, the image and / or video capture system and / or barcode and / or QR code reader can detect which disposable instruments are used and trigger a system alert to replenish the supply and send new, additional disposable instruments to replace the used instruments. In some embodiments, an image and / or video capture system and / or a bar code and / or QR code reader can detect which implant components and other rechargeable components have been used and trigger an alarm in the system to replenish supplies and send new, additional implants to replace used implants; the alarm can also trigger a command to generate an invoice for the hospital and / or surgical center and monitor payment.

[0631] Any of the foregoing embodiments may be applied to any surgical step and any surgical instrument or any implant component during any type of surgery, such as, knee replacement, hip replacement, shoulder replacement, ligament repair, including anterior cruciate ligament repair, spinal surgery, spinal fusion (e.g., anterior and posterior), vertebroplasty and / or kyphoplasty.

[0632] In some embodiments of the present invention, a nail or other implantable or attached marker or calibration or registration phantom or device comprising an optical marker can be initially placed in, for example, a bone or osteophyte or spur or other bony anatomical structure or deformity. Registration of virtual image data can be performed, for example, using anatomical landmarks or positions or osteophytes or spurs or other bony anatomy or deformity where the pin has been physically placed and optionally marked on an electronic image, and processing patient real-time data. The nails can then be selectively removed, for example, if they would interfere with the steps of the surgical procedure. After performing the steps of the surgical procedure (e.g., bone cutting), the nails are optionally reinserted into the nail holes in the residual bone left below the bone cutting, and the nails can be used to register the virtual data. Even if the surgical site and anatomical structure have been changed by the surgical procedure, the patient can obtain patient real-time data.

[0633] In some embodiments of the present invention, the techniques described herein can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan developed for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same techniques described above or any other alignment techniques described in this specification or any other alignment technology devices known in the art.

[0634] Use patient-specific markers or templates to register patient virtual data with patient real-time data

[0635] Various techniques have been described for registering patient virtual data to patient real-time data using patient-specific markers or templates, including those described in WO9325157A1, which is expressly incorporated herein by reference.

[0636] In some embodiments of the present invention, preoperative imaging is performed to obtain three-dimensional data of the patient. For example, preoperative imaging requires ultrasound, computed tomography, or magnetic resonance imaging, as described above, and optionally, injection of a contrast agent.

[0637] Preoperative imaging may include a single area or location, such as the lumbar spine or a portion of the lumbar spine or one or more spinal segments, or a single joint (e.g., a knee, hip, ankle, shoulder, elbow or wrist). Alternatively, preoperative imaging may include scanning part or all of one or more adjacent joints. This approach may be beneficial when information about limb length or axis alignment or rotational alignment is required. For example, when planning a hip replacement, image information obtained through the distal femur and optionally the knee and / or ankle joints can help determine leg length, etc. When planning a knee replacement, it is beneficial to obtain image information through the hip and ankle joints. For example, in this way, the hip center and ankle joint are determined. This information can be used to determine the patient's mechanical axis alignment and, optionally, to plan a mechanical axis correction.

[0638] Preoperative imaging can also be performed in one or more positions, such as prone, supine, upright, flexion, extension, lateral bending, etc. The data obtained from scanning the patient in different positions can be optionally combined or fused. For example, an upright weight-bearing partial or full leg X-ray can be used to determine the mechanical axis alignment of the leg. For example, three-dimensional data of the knee joint from a computed tomography or magnetic resonance imaging can be used to obtain detailed anatomical information about the joint, for example, to obtain surface shape information and design patient-specific markers or templates. The information from the upright scan can be used to align patient-specific markers or templates or their orientation relative to the mechanical axis. The information from the three-dimensional knee scan can be used to derive one or more patient-specific surfaces that are tailored to the patient's unique shape.

[0639] In patients with spinal symptoms, three-dimensional data of the spine can be obtained by computed tomography or magnetic resonance imaging or rotational fluoroscopy or C-arm scanning. For example, upright imaging in flexion and extension can be used to determine the presence and extent of spinal instability, for example, before an intended spinal fusion surgery with pedicle screws and / or fusion cages. The degree of instability or slip can be determined and used to determine the extent of the intended correction (if any) or the extent of the required foraminotomy, both of which can selectively predetermine the three-dimensional data. Lateral bending views can optionally be used to determine the extent and angle of partial vertebral corpectomy and the desired position and / or height of the intervertebral pedicles. Therefore, data from upright imaging can be combined or selectively fused with data from supine or prone imaging. Data from two-dimensional imaging can be combined or fused with data from three-dimensional imaging. Three-dimensional data can be used to derive one or more patient-specific surfaces that are adapted to the patient's unique shape, for example, the unique shape of one or more patient spinous processes, one or more patient transverse processes, one or more more patient laminas, one or more patient articular processes, or one or more patient vertebral bodies.

[0640] The patient-specific marker or template may include one or more surfaces designed and manufactured to fit the corresponding surface of the patient, typically similar to or substantially similar to the negative. The surface may be optionally smoothed. Alternatively, the surface may be intentionally "roughened" to include more surface features than the segmented three-dimensional surface of the patient's target anatomical structure. Such surface features may include spikes or needle-like structures to allow for enhanced fixation of the patient-specific marker or template on the patient's tissue surface.

[0641] Patient-specific markers or templates can be developed from computed tomography, magnetic resonance or ultrasound scans, and X-ray imaging. In principle, any multi-planar two-dimensional or three-dimensional imaging modality is suitable, especially if it provides information about the shape of a surface or provides information to derive an estimate of the shape of the surface of an anatomical region. A patient-specific marker or template may include one or more surfaces designed or manufactured to correspond to any joint or spinal or other anatomical location.

[0642] - The patient's cartilage surface

[0643] - The patient's subchondral bone surface

[0644] - The patient's cortical bone surface

[0645] - Patients with bone disease or bone spurs

[0646] - The patient's bone defect

[0647] - Patients with bone hyperplasia

[0648] - Patients with subchondral cysts

[0649] - Soft tissue shape, such as the shape of the thigh or calf or lower back, or chest area, or neck area, or foot or ankle area, or shoulder area, etc.

[0650] -Soft tissue shape in different body postures or positions, for example, in prone, supine, or lateral positions

[0651] -Patient's ligaments

[0652] - The patient's upper lip

[0653] - The patient's meniscus

[0654] - The shape of the patient's organs

[0655] -Organ margins or patient margins, e.g., liver margins or spleen margins

[0656] For a given tissue, different imaging tests may be particularly appropriate. For example, if patient-specific markers or templates are designed to fit the patient's cartilage shape, MRI and ultrasound or computed tomography arthrography are best suited to provide surface information. If patient-specific markers or templates are designed to fit the shape of subchondral bone or cortical bone, computed tomography can be used, although MRI and ultrasound can also provide information about bone shape.

[0657] Patient-specific markers or templates can be made of different materials, such as ABS or nylon or different types of plastics or metals. They can be machined from blanks, where a CAD / CAM machining system transfers the patient-specific shape information to a milling machine. They can also be produced using stereolithography or three-dimensional printing techniques known in the art. If three-dimensional printing is used, air purge operations and / or water baths can be used to remove residual powder. Three-dimensional printing can be performed using powder-based or liquid resin-based methods, including but not limited to continuous liquid interface production.

[0658] The patient-specific marker or template can include or be combined with an optical marker, such as an optical marker having a different geometric shape or pattern, a QR code, a barcode, or an alphanumeric code. The geometric shape or pattern, QR code, barcode, or alphanumeric code can optionally be printed, such as when three-dimensional printing is used to create the patient-specific marker or template. Three-dimensional printing can be performed using software such as that available from Materialize Magics (Materialise, Leuven, Belgium) and hardware known in the art, such as 3D printers from 3D Systems, Rock Hill, SC, or Concept Laser, Lichtenfels, Germany.

[0659] Patient-specific markers or templates can be made with different material properties. For example, they can be inelastic, semi-elastic, or elastic. They can be rigid. They can be solid, or include hollow spaces or openings. They can be opaque. Patient-specific markers or templates can be translucent. Patient-specific markers can be transparent. In some embodiments, patient-specific markers or templates can be semi-opaque or translucent. However, when the patient-specific marker or template is in contact with the patient and the patient-specific surface of the marker or template is well matched to the corresponding surface of the patient, the patient-specific marker or template is transparent due to the presence of tissue moisture on the corresponding surface of the patient.

[0660] One or more patient-specific markers or templates can be used on the first surface of the joint. One or more patient-specific markers can be used on the second surface of the joint. The first and second surfaces can be located on the same weight-bearing side of the joint. The first and second surfaces can be located on opposite sides of the joint. The one or more patient-specific markers or templates on the first surface of the joint cannot be connected to the one or more patient-specific markers or templates on the second surface of the joint. In some embodiments, the one or more patient-specific markers or templates on the first surface of the joint are optionally connected or attached to the second surface of the joint. Thus, the one or more patient-specific markers or templates are optionally referenced to each other.

[0661] Patient-specific markers or templates can be designed for any joint, any part of the spine, and any tissue in the human body. Patient-specific markers or templates generally include one or more surfaces or shapes designed to fit the corresponding surface or shape of the patient. Representative, non-limiting examples of patient surfaces on which patient-specific markers or templates can be designed and / or mounted include:

[0662] spine:

[0663] - Part or all of the spinous process

[0664] - Partial or complete spinal plate

[0665] - Part or all of the spinal articular process

[0666] - Part or all of a facet joint

[0667] - Part or all of the transverse process

[0668] - Part or all of the pedicle

[0669] - Part or all of a vertebral body

[0670] - Part or all of the intervertebral disc

[0671] - Osteophytes on part or all of the spine

[0672] - Bone spurs on part or all of the spine

[0673] - Fracture of part or all of the spine

[0674] - Fracture of part or all of the vertebral body

[0675] - Any combination of the above

[0676] Hip:

[0677] - Part or all of the acetabulum

[0678] - Part or the entire rim of the acetabulum

[0679] -Multiple segments of the acetabular rim

[0680] -Part of the iliac wall

[0681] -Part of the pubic bone

[0682] -Part of the ischium

[0683] -Part of the greater rotor or the entire greater rotor

[0684] -Part of or the entire lesser trochanter

[0685] - Part of or the entire femoral shaft

[0686] - Part or all of the femoral neck

[0687] - Part or all of the femoral head

[0688] -Bone concave

[0689] - Transverse acetabular ligament

[0690] - Posterior thalamic tubercle

[0691] -ligament

[0692] -upper lip

[0693] - One or more osteophytes, femoral and / or acetabular

[0694] - Any combination of the above

[0695] knee:

[0696] - Part or all of the medial condyle

[0697] - Part or all of the lateral femoral condyle

[0698] - Partial or complete femoral notch

[0699] -Part or all of the pulley

[0700] -Part of the anterior femoral cortex

[0701] - A portion of the anterior femoral cortex and the adjacent portion of the trochlea

[0702] - A portion of the anterior femoral cortex, with adjacent portions of the trochlea and osteophytes when present

[0703] - One or more osteophytes on the femur and / or tibia

[0704] - One or more bone spurs in the femur and / or tibia

[0705] - Epicondylar eminence

[0706] - Part or all of the mid-tibial plateau

[0707] - Part or all of the lateral tibial plateau

[0708] - Part or all of the middle tibial spine

[0709] - Partial or entire lateral tibial spine

[0710] -Part of the anterior tibial cortex

[0711] - Part of the anterior tibial cortex and part of the tibial plateau, medial or lateral or both

[0712] - Part of the anterior tibial cortex and part of the tibial plateau, medial or lateral or both and the presence of osteophytes

[0713] - Part or all of the patella

[0714] - Medial edge of the patella

[0715] - Lateral edge of the patella

[0716] - Upper pole of the patella

[0717] -Inferior pole of the patella

[0718] - Patellar osteophytes

[0719] -Anterior cruciate ligament

[0720] -Posterior cruciate ligament

[0721] -Medial collateral ligament

[0722] - Lateral collateral ligament

[0723] - Part or all of the medial meniscus

[0724] - Part or all of the lateral meniscus

[0725] - Any combination of the above

[0726] Shoulders:

[0727] - Partial or complete glenoid

[0728] - Part or all of the coracoid process

[0729] - Part or all of the acromion

[0730] -Part of the clavicle

[0731] - Part or all of the humeral head

[0732] - Part or all of the humeral neck

[0733] -Part of the humeral shaft

[0734] - One or more humeral osteophytes

[0735] - One or more glenoid osteophytes

[0736] - Part or all of the glenoid labrum

[0737] - Part or all of the shoulder ligaments, such as the coracoacromial ligament, superior, middle, or inferior glenohumeral ligament

[0738] -Part of the shoulder blade

[0739] - Any combination of the above

[0740] Skull and Brain:

[0741] -Part of the skull

[0742] -Part of the occipital bone

[0743] -Part of the temporal bone

[0744] -Part of the occipital bone

[0745] -Part of the parietal bone

[0746] -Part of the frontal bone

[0747] -Part of the facial bones

[0748] - Part or all of the bone structure inside the skull

[0749] - Select some or all of the gyri

[0750] - Select part or all of the sulci

[0751] -Part of the sinus

[0752] -Part of the venous sinus

[0753] -Part of the blood vessel

[0754] organ:

[0755] - Part of an organ, such as the upper or lower pole of a kidney

[0756] -The edges or margins of the liver, spleen, or lungs

[0757] -Part of the liver lobe

[0758] -Part of the blood vessel

[0759] - A part of a crack, for example, in the liver or spleen

[0760] -Part of the uterus

[0761] If applicable to a specific anatomical region, such as cartilage, subchondral bone, cortical bone, osteophytes, etc., patient-specific markers or templates can be designed or adapted for any of the aforementioned tissues. Patient-specific markers or templates can be designed or adapted only for normal tissue. Patient-specific markers or templates can be designed or adapted only for abnormal or diseased tissue. Patient-specific markers or templates can be designed or adapted for a combination of normal tissue and abnormal or diseased tissue. For example, patient-specific markers can be designed for normal cartilage, or diseased cartilage, or a combination of normal and diseased cartilage, such as on the same or opposing joint surfaces. Patient-specific markers can be used to align one or more normal or pathological tissues or structures in a common coordinate system, such as with one or more optical head-mounted displays and virtual data of the patient. Virtual and physical surgical instruments and implant components can also be aligned in a common coordinate system.

[0762] Patient-specific markers or templates can be designed using patient virtual data, such as from preoperative imaging studies such as computed tomography scans, magnetic resonance imaging scans, or ultrasound scans. The patient-specific markers or templates include one or more surfaces designed and / or manufactured to closely mate with corresponding surfaces of the patient.

[0763] In some embodiments of the present invention, a surgeon or operator can apply a patient-specific marker or template to the corresponding tissue of the patient. Once a satisfactory fit is achieved and the two corresponding surfaces are substantially in contact, the patient-specific marker or template can be used to register the patient's virtual data and, optionally, the virtual surgical plan with the patient's real-time data. By applying the patient-specific marker or template to the corresponding surface of the patient, the surgeon can effectively identify the corresponding structures or surfaces in the virtual data and the patient's real-time data.

[0764] The position, location, and / or orientation of the patient-specific marker or template relative to the optical head-mounted display can then be determined. Any of the embodiments described herein can be applied to determine the position, location, and / or orientation of the patient-specific marker or template relative to the optical head-mounted display. For example, the sides of the patient-specific marker or template that are opposite the patient-specific surface can include certain standardized geometric features, such as rectangles, triangles, circles, etc., to facilitate recognition by an image and / or video capture system integrated into, connected to, or coupled to the optical head-mounted display. In alternative embodiments, the patient-specific marker or template can include one or more inertial measurement units, including accelerometers, magnetometers, and gyroscopes, similar to the optical head-mounted display. In some embodiments, the patient-specific marker or template can include one or more radio frequency tags or markers or retroreflective markers, and their position, location, and / or orientation can be captured by the surgical navigation system. Radio frequency tags can be active or passive. The optical head-mounted display can also optionally include one or more radio frequency tags or markers or retroreflective markers, and their position, location, and / or orientation can also be captured by the surgical navigation system and cross-referenced with the patient-specific marker or template. The patient-specific marker or template may also include a light source, such as a laser or light-emitting diode (LED). For example, a laser may be projected onto a wall or ceiling, and the optical head-mounted display may be referenced relative to it. The LED attached to or integrated into the patient-specific marker or template may be identified, for example, by an image and / or video capture system integrated into or connected to the optical head-mounted display.

[0765] In other embodiments, one or more surgical instruments and / or one or more implantable devices used during a surgical procedure may include certain standardized geometric features, such as rectangles, triangles, circles, etc., that are easily recognized by an image and / or video capture system integrated into, connected to, or coupled to an optical head-mounted display. In alternative embodiments, one or more surgical instruments and / or one or more implantable devices used during a surgical procedure may include one or more inertial measurement units, including accelerometers, magnetometers, and gyroscopes, similar to the optical head-mounted display. In some embodiments, one or more surgical instruments and / or one or more implantable devices used during a surgical procedure may include one or more radio frequency tags or markers or retroreflective markers, and their location, position, and / or orientation may be captured by a surgical navigation system. The optical head-mounted display may also optionally include one or more radio frequency tags or markers or retroreflective markers, and their location, position, and / or orientation may also be captured by the surgical navigation system and cross-referenced with patient-specific markers or templates and / or the one or more surgical instruments and / or one or more implantable devices used in the surgical instrument. One or more surgical instruments and / or one or more implantable devices used during surgery may also include a light source, such as a laser or a light emitting diode. For example, a laser can be projected onto a wall or ceiling, and the optical head-mounted display and the patient can be referenced relative to the laser. For example, an image and / or video capture system integrated into, connected to, or coupled to the optical head-mounted display can identify the light emitting diodes of one or more surgical instruments and / or one or more implantable devices attached to or integrated into the surgical instrument. Multiple light emitting diodes may optionally be used. Optionally, two or more of the multiple light emitting diodes emit light of different wavelengths or colors. The two or more light emitting diodes can be located in spatially defined positions and orientations, such as at a predefined or fixed distance and at one or more predefined or fixed angles. In this way, when the image and / or video capture system is close to the operator's eyes, the two or more light emitting diodes can be located using the image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display and the measured distances and / or angles seen by the image and / or video capture system. For example, it can be used to determine the distance and / or direction of the operator to the target anatomical structure. By using LEDs with different wavelengths or colors, the image and / or video capture system can distinguish between the different LEDs; when the LEDs are arranged in a known spatial orientation, this information can help improve the accuracy of the registration and / or be used to obtain accurate distance, angle, direction and / or speed measurements. In all embodiments that include or are suitable for the use of LEDs, two or more LEDs with different wavelengths and colors, measurements or registrations are used throughout this specification.

[0766] Optionally, the selective patient-specific markers or templates used during surgery and one or more surgical instruments and / or implantable devices may also include color markings, selected color markings of different geometries or color markings at different, known positions and different known angles, such as by an image and / or video capture system integrated into, attached to, or coupled to an optical head-mounted display to recognize such patterns and, for example, estimate distances and angles (e.g., from the surgical site to the optical head-mounted display, or the distance and angle between two markers, two surgical instruments, or medical device components).

[0767] Optionally, the patient-specific markings or templates, as well as one or more surgical instruments and / or one or more implantable devices used during surgery, may also include multiple scales, such as metric distances, inches, or angles, such as by recognizing such scales or angles using an image and / or video capture system integrated into, attached to, or coupled to an optical head-mounted display, and estimating distances and angles, such as from a surgical site to the optical head-mounted display, or between two surgical instruments or medical device components.

[0768] In some embodiments of the invention, the patient-specific marker or template can be attached to a corresponding surface of the patient or to an adjacent surface of the patient, for example, using tissue glue (such as fibrin glue or needles or staples).

[0769] In some embodiments, the patient-specific markings or template may include openings or guides, for example, for receiving surgical instruments or tools, such as drills, saws, reamers, pins, screws, and any other instruments or tools known in the art.

[0770] By cross-referencing patient virtual data and patient real-time data using patient-specific markers or templates and one or more selective surgical instruments and / or one or more implantable devices used during surgery and an optical head-mounted display, coordinate information, distance information, axis information, and functional information contained in the patient virtual data can now be obtained and used during surgery.

[0771] In some embodiments of the present invention, the techniques described herein can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan developed for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same techniques described above or any other alignment techniques described in this specification or any other alignment technology devices known in the art.

[0772] Registering virtual patient data with live patient data using intraoperative imaging

[0773] In some embodiments of the present invention, intraoperative imaging can be performed, for example, using X-ray imaging or computed tomography imaging and / or ultrasound imaging. The patient virtual data obtained using intraoperative imaging can be used to align the patient virtual data obtained preoperatively, for example, using preoperative X-ray, ultrasound, computed tomography or magnetic resonance imaging. The patient's preoperative and intraoperative virtual data and the patient's real-time data can be aligned in a common coordinate system with one or more optical head-mounted displays by identifying and selectively marking corresponding anatomical landmarks, surfaces, object shapes, for example. In the patient's preoperative virtual data, the patient's intraoperative virtual data, for example, one or more electronic two-dimensional or three-dimensional images described above, and the patient's real-time data can be the surgical site or target tissue. The virtual preoperative, virtual intraoperative and real-time data may include osteophytes or bone spurs or other bony anatomy or deformities. Virtual and physical surgical instruments and implant components can also be aligned in a common coordinate system.

[0774] This embodiment may be advantageous when the amount of information obtainable through intraoperative imaging is more limited, anatomically or otherwise, than the amount of information obtainable through preoperative imaging, or vice versa.

[0775] For example, intraoperative imaging can be performed using X-ray imaging, which is typically only two-dimensional. X-ray imaging can be enhanced by image acquisition in more than one plane, for example, orthogonal planes or one or more planes separated by defined angles. Intraoperative X-ray images can be used to identify certain landmarks or shapes, which can then be registered to the patient's preoperative imaging and / or real-time data during surgery. Preoperative imaging can optionally include three-dimensional image data, for example, obtained by computed tomography or magnetic resonance imaging. Acquiring intraoperative images in multiple planes can help to more accurately define the location of certain anatomical landmarks, contours or shapes used for registration of preoperative virtual data, intraoperative virtual data and patient real-time data. The patient's intraoperative virtual data can be an intraoperative image of the patient in two or three dimensions for clarification.

[0776] For example, in spinal surgeries such as vertebroplasty, kyphoplasty, pedicle screw placement, or placement of anterior spinal devices including artificial discs or fusion cages, intraoperative X-ray imaging can be used to identify the spinal segment being operated on, certain landmarks or contours in the anteroposterior projection, such as the tips of the spinous processes, facet joints, the upper or lower tips of the facet joints, the cortical margins of the lamina, the upper or lower endplates or osteophytes or spurs or other bony anatomy or deformities. Alternatively, X-ray magnification resulting from the distance between the X-ray tube and the patient can be taken into account in any registration to improve the registration accuracy of the patient's virtual preoperative data, the patient's virtual intraoperative data, or the real-time patient data. The intraoperative X-ray image can then be registered and, optionally, superimposed onto the patient's preoperative data or the patient's real-time data via an optical head-mounted display. The patient's intraoperative virtual data, such as the spinous process tip, facet joint, the superior or inferior tip of the facet joint, the cortical edge of the lamina, the superior or inferior endplate, can be registered to the real-time data, for example, by contacting corresponding anatomical landmarks of a pointing device or a needle or nail inserted through the skin, and by cross-referencing the position of the tip of the pointing device with the patient's intraoperative virtual data in the real-time data. In this manner, the patient's preoperative virtual data, the patient's intraoperative virtual data, and the patient's real-time data, or any combination thereof, can be co-registered. Two or three data sets, the patient's preoperative virtual data, the patient's intraoperative virtual data, and the patient's real-time data, can be selectively viewed in the optical head-mounted display. However, in many embodiments, intraoperative imaging can be used only to enhance the registration accuracy of the patient's preoperative virtual data and the patient's real-time data, and, for example, the preoperative virtual data of the patient and / or a medical device to be placed at the surgical site, along with the real-time data of the patient or the surgical site, will be displayed together by the optical head-mounted display.

[0777] In some embodiments of the present invention, the techniques described herein can be used to repeat the registration of patient virtual data and patient real-time data after one or more surgical steps have been performed, and optionally repeat intraoperative imaging. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient or the patient's intraoperative repeated image data can be matched, superimposed and / or registered to the patient's virtual data (such as in a virtual surgical plan prepared for the patient), the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the patient's virtual data. Matching, superimposing and / or registering patient real-time data and patient virtual data after surgical tissue alteration can use the same techniques described above or any other registration techniques described in this specification or any other registration technology devices known in the art.

[0778] Use skin markers or soft tissue markers to register virtual patient data with real-time patient data

[0779] In some embodiments of the present invention, skin markers and soft tissue markers, a calibration or registration phantom, or a device can be used to register preoperative virtual data, and optionally, intraoperative virtual data, such as data obtained from intraoperative X-ray imaging, and real-time data viewed through an optical head-mounted display, in a common coordinate system with one or more optical head-mounted displays. Virtual and physical surgical instruments and implant assemblies can also be registered in the common coordinate system. For example, initial registration between preoperative virtual data and real-time patient data can occur at the beginning of the procedure. For example, initial registration can be performed using corresponding anatomical landmarks, surfaces, or shapes, or using intraoperative imaging to generate intraoperative virtual data or other embodiments described herein. For example, registration can be used to place virtual data and real-time data, as well as an optical head-mounted display, into the common coordinate system. Skin markers, a calibration or registration phantom, or a device can then be applied. Virtual and physical surgical instruments and implant assemblies can also be registered in the common coordinate system. Alternatively, or in addition, soft tissue markers, a calibration or registration phantom, or a device can be applied. Typically, more than one, such as two, three, four or more, skin markers and soft tissue markers, calibration or registration phantoms or devices are applied. For clarity, the terms implantable markers, attached markers, skin markers, soft tissue markers, calibration or registration phantoms or devices used in this application may include optical markers, such as optical markers with different geometric shapes or patterns, with QR codes, with bar codes, with alphanumeric codes. Skin markers and soft tissue markers, calibration or registration phantoms or devices can be applied to the skin or soft tissue using a tissue-compatible adhesive (including fibrin glue, etc.). In some embodiments, one, two, three, four or more skin markers and soft tissue markers, calibration or registration phantoms or devices can be included in a surgical sheet or dressing transparent film applied to the skin before surgery. The skin markers and soft tissue markers, calibration or registration models or devices can then be registered in real-time data and referenced against the virtual data. Skin markers and soft tissue markers can then be used to calibrate or register the model or device, for example, when the surgical site is altered and anatomical landmarks, surfaces, or shapes used for initial registration of the virtual and real-time data are altered or removed and cannot be used or cannot be reliably used to maintain registration between the virtual and real-time data. The virtual pre-operative, virtual intra-operative, and real-time data may include osteophytes or bone spurs or other bony anatomy or deformities.

[0780] In some embodiments of the present invention, the techniques described herein can be reused to align patient virtual data and patient real-time data after one or more surgical steps have been performed. In this case, surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched, superimposed and / or aligned with surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume in the patient virtual data or other tissue features in the virtual surgical plan developed for the patient. Matching, superimposing and / or aligning patient real-time data and patient virtual data after surgical tissue alteration can use the same techniques described above or any other alignment techniques described in this specification or any other alignment technology devices known in the art.

[0781] If the marker or phantom is still present, the same skin marker or soft tissue marker or calibration phantom or registration phantom can be used after performing one or more surgical steps. Alternatively, a re-registration of the patient's live data and the patient's virtual data can be performed after one or more surgical steps or surgical changes. After the re-registration, one or more new skin markers or soft tissue markers or calibration phantoms or registration phantoms can be applied and the re-registered live and virtual data can be cross-referenced after the surgical step or change. The skin markers or soft tissue markers or calibration phantoms or registration phantoms can then be used for subsequent matching, superimposition, movement, and registration of the patient's live data and the patient's virtual data.

[0782] A calibration or registration phantom of defined size or shape is used to register the patient virtual data with the patient real-time data.

[0783] In some embodiments of the present invention, a calibration or registration phantom of a defined size or shape may be used to perform registration of virtual patient data with real-time patient data. The calibration or registration phantom may be primarily of two-dimensional or three-dimensional nature. For example, a calibration or registration phantom may be primarily arranged or positioned in a single plane. Other calibration phantoms may be located in multiple planes, thereby creating the opportunity to use multiple planes for registration. For clarity, the terms calibration or registration phantom, implantable marker, attached marker, skin marker, soft tissue marker, calibration or registration phantom or device used in this application may include optical markers, such as optical markers with different geometric shapes or patterns, with QR codes, with bar codes, with alphanumeric codes.

[0784] Such a calibration or registration phantom can be attached to the patient's skin. The calibration or registration phantom can be integrated into or attached to a surgical drape. The calibration or registration phantom can be attached to the patient's tissue. The calibration or registration phantom can be part of a medical device or a component thereof. Components of medical devices typically have known dimensions. By using the calibration or registration phantom and other markers, real-time patient data and virtual patient data can be registered in a common coordinate system, for example, with one or more optical head-mounted displays. Virtual and physical surgical instruments and implant components can also be registered in a common coordinate system.

[0785] In some embodiments, the calibration or registration phantom includes known dimensions, angles, or geometric two-dimensional or three-dimensional shapes. For example, the calibration or registration phantom may include the following structures:

[0786] -Circle, oval, ellipse, square, rectangle, complex 2D geometry, 2D geometry with one or more defined distances, 2D geometry with one or more defined angles

[0787] - spheres, egg-shaped structures, cylinders, cubes, cuboids, complex 3D geometric shapes or shapes, 3D geometric shapes with one or more defined distances, 3D geometric shapes with one or more defined angles, 3D geometric shapes with one or more defined surfaces

[0788] Optionally, the calibration or registration phantom can be radiopaque if pre-operative or intra-operative imaging is performed using an imaging modality with ionizing radiation, such as x-ray imaging, two-dimensional or three-dimensional fluoroscopy, computed tomography, cone-beam computed tomography, etc.

[0789] In some embodiments, the calibration or registration phantom can be visible to MRI or radionuclide imaging or single photon emission tomography or positron emission tomography, for example, by including a portion or container in the phantom containing isotope emitting water doped with gadolinium-DTPA or doped with a radionuclide or positron emission tomography (PET). Any contrast agent or MRI or radionuclide imaging or single photon emission tomography or PET visible agent known in the art can be used in this manner.

[0790] In some embodiments, the calibration or registration phantom includes retroreflective markings or features that aid in detection by image and / or video capture systems. The calibration or registration phantom can also highlight patient tissue, including blood and surgical drapes, using selected colors, such as bright green, bright blue, bright yellow, bright pink, etc. Any color combination known in the art can be used.

[0791] The calibration or registration phantom may optionally include a light emitting diode, optionally battery powered. A plurality of light emitting diodes may be used. The light emitting diodes may emit light of a known color, hue, and intensity, preferably selected to be readily identifiable by an image and / or video capture system, as well as a segmentation technique or algorithm for detecting the location, position, and / or orientation of the light emitting diodes.

[0792] The light emitting diodes can be arranged in a spatially defined manner, wherein two or more light emitting diodes are arranged at defined distances and angles in substantially the same plane or in different planes. If the light emitting diodes are arranged in different planes, the spatial directions of the planes are known and defined.

[0793] When two or more LEDs are used, the two or more LEDs can emit light using different wavelengths, colors, intensities, and optionally blink rates. In this manner, an image and / or video capture system integrated into, attached to, or separate from the optical head-mounted display can identify each different LED based on one or more of their different wavelengths, colors, intensities, and / or blink rates. When the LEDs are arranged in a spatially defined and known manner, such as using known distances or angles within the same plane or within different planes, the identification of each individual LED and the changes in distances and angles measured by the image and / or video capture system can be used to determine the position, location, and / or orientation of the optical head-mounted display and / or the operator's head (e.g., if the image and / or video capture system is integrated into or connected to the optical head-mounted display), or in some applications, to calibrate or register the movement of the phantom and the patient or body part to which the LEDs are attached.

[0794] The LEDs used throughout this specification can be reusable. The LEDs used throughout this specification can also be disposable, optionally with an integrated disposable battery unit / battery. The LEDs can be wired, e.g., connected to a power source and / or connected to a wired user interface or control unit. The LEDs can be wireless, e.g., not connected to a power source (e.g., battery-powered) and / or connected to a wireless (e.g., WiFi, Bluetooth) control unit.

[0795] The light emitting diodes can be connected and / or organized in a LIF network. For example, one or more LIF networks can be used to transmit or receive data or information to and from one or more optical head-mounted displays to a control unit or computer, optionally with a user interface. In this example, the light emitting diodes participating in or connected to the one or more LIF networks can be integrated into or attached to the optical head-mounted display. The light emitting diodes participating in or connected to the one or more LIF networks can be connected to or integrated, where applicable, to the surgeon, operating room staff, the patient, the surgical site, one or more optical head-mounted displays, one or more navigation systems at any location or site, one or more navigation markers, such as retroreflective markers, infrared markers, radio frequency markers; one or more optical markers, calibration or registration phantoms, and the like.

[0796] The LIF network can also be used to transmit or receive data or information regarding the spatial position, orientation, movement direction, and speed of individual LEDs. For example, using an image and / or video capture system to measure the position, orientation, movement direction, and speed of the same LED relative to the surgeon, patient, or surgical site, the same LED can be used to transmit or receive information within the LIF network, optionally using different wavelengths, colors, frequencies, and blink patterns, depending on the type of data being transmitted. This information can include information regarding the position, orientation, movement direction, and speed of the individual LEDs. This information can also include data being transmitted or received by an optical head-mounted display. This information can include information or data being displayed by the optical head-mounted display. This information can include information generated or received by navigation markers or radio frequency tags. This information can also include information captured by one or more image and / or video capture systems or cameras.

[0797] 1, 2, 3, 4 or more light emitting diodes can be connected or attached to the patient, the target anatomy, the surgical site, the surgical site (e.g., performed using virtual surgical planning), the optical head mounted display, a second, third and / or additional optical head mounted displays, e.g., worn by a second surgeon, a scrub nurse, other operating room personnel, the hand, forearm, upper arm and / or other body part of the surgeon / operator after a first, second or more surgical changes.

[0798] For example, when one or more LEDs emit light using different wavelengths, colors, intensities, and optional blinking frequencies, the relative position, orientation, movement, direction of movement, and speed of movement of each LED can be determined (e.g., using one or more image and / or video capture systems integrated into, connected to, or separate from one or more optical head-mounted displays).

[0799] The calibration or registration phantom may optionally include one or more lasers, optionally battery powered. Multiple lasers may be used. The lasers may emit light of a known color, hue, and intensity, e.g., light selected to be easily identifiable by an image and / or video capture system and any segmentation technique or algorithm for detecting the location, position, and / or orientation of the lasers.

[0800] The lasers can be arranged in a spatially defined manner, wherein two or more lasers are arranged at a defined distance or distances in substantially the same plane or in different planes at a defined angle or angles. If the lasers are arranged in different planes, the spatial orientation of the planes can be known and defined.

[0801] The calibration or registration phantom may optionally include a radio frequency (RF) transmitter, optionally battery powered. Multiple RF transmitters may be used. The RF transmitter may transmit one or more signals selected to be readily recognizable by an RF receiver system for detecting the location, position, and / or orientation of the RF transmitter. The one or more RF transmitters may transmit signals having different frequencies and intensities, thereby allowing the RF receiver system to distinguish between different RF transmitters.

[0802] The radio frequency transmitters can be arranged in a spatially defined manner, wherein two or more radio frequency transmitters are arranged at a defined distance or distances in substantially the same plane or in different planes at a defined angle or angles. If the radio frequency transmitters are arranged in different planes, the spatial orientation of the planes can be known and defined.

[0803] The calibration or re...

Claims

1. A system for preparing a physical joint for reconstruction of anterior cruciate ligament in a patient, the system comprising: at least one processor, at least one see-through optical head-mounted display, and at least one user interface, wherein the system is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual axis, wherein the virtual axis is a three-dimensional digital representation that indicates a predetermined position, a predetermined orientation, or a combination thereof of at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof relative to at least one anatomical structure of the physical joint, wherein the system is configured to allow the virtual axis to be aligned with at least one of a predetermined femoral tunnel, a predetermined tibial tunnel, a predetermined position of an anterior cruciate ligament graft, or a predetermined orientation of an anterior cruciate ligament graft, wherein the system is configured to display the virtual axis on the surface of the physical joint at the predetermined position, predetermined orientation, or a combination thereof by the at least one see-through optical head-mounted display, wherein the system is configured to allow superposition and alignment between the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof and at least a portion of the virtual axis displayed by the at least one see-through optical head mounted display, and Wherein the system is configured to register the at least one anatomical structure in a coordinate system during surgery.

2. The system of claim 1 , wherein the system is configured to maintain the display of the virtual axis on the surface of the physical joint at a predetermined position, predetermined orientation, or a combination thereof relative to the at least one anatomical structure of the physical joint as the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof is moved in the user's field of view.

3. A system according to claim 1, wherein the system is configured to adjust the display of the virtual axis in response to the movement of the physical joint to maintain the display of the virtual axis at the predetermined position, predetermined orientation, or a combination thereof relative to the at least one anatomical structure of the physical joint.

4. A system according to claim 3, wherein the system is configured to adjust the display of the virtual axis in response to movement of the first and second joint sides to maintain the display of the virtual axis in the predetermined position, predetermined orientation, or a combination thereof relative to the at least one anatomical structure of the physical joint.

5. The system of claim 1 , wherein the at least one user interface comprises a graphical user interface, voice recognition, gesture recognition, a virtual interface displayed by the at least one see-through optical head-mounted display, a virtual keyboard displayed by the at least one see-through optical head-mounted display, a physical keyboard, a physical computer mouse, a physical tracking pad, or a combination thereof.

6. The system of claim 1 , wherein the virtual axis is a position indicator of the at least one physical surgical drill, drill guide, peg, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, shaver, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or combination thereof, wherein the position indicator is a three-dimensional digital representation corresponding to at least a portion or axis of the at least one physical surgical drill, drill guide, peg, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, shaver, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or combination thereof.

7. The system of claim 1, wherein the system is used to prepare the physical joint for implantation of the anterior cruciate ligament comprising a single bundle technique or a double bundle technique.

8. The system of claim 1, wherein the system is configured to identify the at least one anatomical structure intraoperatively through an arthroscopic approach.

9. The system of claim 1, wherein the system is configured to intraoperatively register the at least one anatomical structure of the physical joint in the coordinate system using at least one of a pointer, a camera, a 3D scanner, an arthroscope, or a combination thereof.

10. The system of claim 1, wherein the system is configured to intraoperatively register the at least one anatomical structure of the physical joint in the coordinate system using an image capture system, a video capture system, a 3D scanner, or a combination thereof.

11. The system of claim 1 , wherein the predetermined position comprises a predetermined starting position, a predetermined ending position, or a combination thereof, or wherein the predetermined orientation comprises a predetermined starting orientation, a predetermined starting alignment, a predetermined ending orientation, a predetermined starting point and ending point, a predetermined ending alignment, or a combination thereof.

12. The system of claim 1, wherein the predetermined position comprises a predetermined starting point, a predetermined ending point, or a combination thereof, or wherein the predetermined orientation comprises a predetermined angle.

13. The system of claim 1 , wherein the system is configured to obtain the coordinates of the at least one anatomical structure using a radio frequency marker, an IMU, an optical marker, a geometric pattern, a patient-specific marker or template, a pointer, a surgical navigation system, a camera, a video system, an arthroscope, a depth sensor, a laser scanner, a 3D scanner, x-rays, or a combination thereof.

14. The system of claim 1, wherein the system is configured to use infrared markers to obtain coordinates of the at least one anatomical structure.

15. The system of claim 1, wherein the system is configured to obtain the coordinates of the at least one anatomical structure using an LED, an image capture system, an imaging system, or a combination thereof.

16. The system of claim 1, wherein the at least one anatomical structure comprises an anatomical landmark, an anatomical plane, or a combination thereof.

17. The system of claim 1, wherein the at least one anatomical structure comprises an articular surface, a cartilage surface, a cortical bone surface, a cut bone surface, a reamed bone surface, a milled bone surface, an impacted bone surface, a tissue resection, a tissue surface, or a combination thereof.

18. The system of claim 1, wherein the at least one anatomical structure comprises a subchondral bone surface.

19. The system of claim 1 , wherein the at least one anatomical structure comprises information about an anatomical axis, a biomechanical axis, a mechanical axis, a dimension between two or more anatomical landmarks, a dimension between anterior and posterior anatomical structures, a dimension between medial and lateral anatomical structures, a dimension between superior and inferior anatomical structures, a curvature, a surface, an edge, or a combination thereof.

20. The system of claim 1, wherein the at least one anatomical structure comprises information regarding an anatomical axis, a biomechanical axis, a mechanical axis, a distance between two or more anatomical landmarks, a shape, a length, a width, a depth, or a combination thereof.

21. The system of claim 16, wherein the at least one anatomical landmark comprises one or more of: a portion or all of the femoral shaft, a portion or all of the medial femoral condyle, a portion or all of the lateral femoral condyle, a portion or all of the femoral notch, the origin of the anterior cruciate ligament, the insertion of the anterior cruciate ligament, a portion or all of the trochlea, a portion of the anterior femoral cortex, a portion of the anterior femoral cortex and an adjacent portion of the trochlea, a portion of the anterior femoral cortex and an adjacent portion of the trochlea and osteophytes, one or more femoral osteophytes, one or more tibial osteophytes, an epicondyle eminence, a medial epicondyle eminence, a lateral epicondyle eminence, a portion or all of the medial tibial plateau, a portion or all of the lateral tibial plateau, a portion or all of the medial tibial spine, a portion or all of the lateral tibial spine, an anterior cruciate ligament remnant, a portion of the anterior tibial cortex, a portion of the anterior tibial cortex, and a portion of the tibial plateau , medial wall of the femoral notch, lateral wall of the femoral notch, top of the femoral notch, medial wall of the medial condyle, lateral wall of the lateral condyle, medial femoral condyle shape, medial femoral condyle radius, medial femoral condyle convexity, medial femoral condyle concavity, lateral femoral condyle shape, lateral femoral condyle radius, lateral femoral condyle convexity, lateral femoral condyle concavity, intercondylar notch shape, intercondylar notch surface features, anterior medial tibial margin, anterolateral tibial margin, medial tibial margin, lateral lateral tibial edge, lowest point of the medial plateau, lowest point of the lateral plateau, highest point of the medial plateau, highest point of the lateral plateau, medial tibial plateau shape, lateral tibial plateau shape, medial tibial plateau surface, medial tibial plateau surface characteristics, medial tibial plateau radius, medial tibial plateau convexity, medial tibial plateau concavity, lateral tibial plateau shape, lateral tibial plateau surface, lateral tibial plateau surface characteristics, lateral tibial plateau radius, lateral tibial plateau convexity, or lateral tibial plateau concavity.

22. The system of claim 1, wherein the surfaces of the physical joint include cartilage, opposing joint surfaces, skin, subcutaneous tissue, ligament remnants, meniscus, and labrum.

23. The system of claim 1, wherein the surface of the physical joint comprises one or more of bone, ligament, and intra-articular structure.

24. The system of claim 3, wherein movement of the patient's physical joint comprises movement of at least a portion of one or more of: a cartilage surface, a cortical bone surface, an articular surface, osteophytes, a cut bone surface, a reamed bone surface, a milled bone surface, skin, subcutaneous tissue, a femoral condyle, a femoral notch, a trochlea, anterior cortex of the femur, an epicondylar eminence, a medial tibial plateau, a lateral tibial plateau, a tibial spine, a tibial cortex, a patella, anterior cruciate ligament, anterior cruciate ligament remnant, a posterior cruciate ligament, a medial collateral ligament, a lateral collateral ligament, and a meniscus.

25. The system of claim 3, wherein movement of the patient's physical joint comprises movement of a subchondral bone surface.

26. The system of claim 3, wherein movement of the patient's physical joint comprises movement of one or more of: a deburred joint surface, at least a portion of a surgically altered surface.

27. A system according to claim 13, wherein one or more markers are attached to a first joint side and a second joint side, wherein the position and / or orientation of the one or more markers on the first joint side and the second joint side are tracked during movement of the joint, and wherein the one or more markers include one or more of an IMU, an optical marker, a geometric pattern, a barcode, a QR code, an LED, a radio frequency marker, a retroreflective marker, an active marker, and a passive marker.

28. A system according to claim 13, wherein one or more markers are attached to a first joint side and a second joint side, wherein the position and / or orientation of the one or more markers on the first joint side and the second joint side are tracked during movement of the joint using a camera, an image capture system, an imaging system, a 3D scanner, or a combination thereof.

29. The system of claim 1 , wherein the system is configured to track the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof in the coordinate system.

30. The system of claim 29, wherein the system is configured to display the virtual surgical guide in one or more colors on the at least one see-through optical head-mounted display when the at least one physical surgical drill, drill guide, pin, guide rod, pointer, obturator, reamer, milling cutter, reamer, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof.

31. The system of claim 1 , wherein the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof is handheld.

32. The system of claim 1 , wherein the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof is attached to the robot.

33. A system according to claim 32, wherein the system is configured to display a safety area on the surface of the physical joint through the at least one perspective optical head-mounted display, wherein the safety area represents a predetermined operating range of the at least one physical surgical drill, drilling guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel groover, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof.

34. The system of claim 33, wherein the system is configured to display a visual alert via the at least one see-through optical head-mounted display, and / or wherein the system is configured to trigger an acoustic alert when the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof is within or outside the safe zone.

35. The system of claim 32, wherein the robot has a robotic arm.

36. The system of claim 32, wherein the robot is configured to guide a physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, drill, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof.

37. The system of claim 1, wherein the system is configured to intermittently display the virtual surgical guide.

38. The system of claim 1, wherein the at least one see-through optical head-mounted display comprises the at least one processor.

39. The system of claim 1, wherein the at least one processor is separate from the at least one see-through optical head-mounted display.

40. The system of claim 1, wherein the system is used to prepare the physical joint for anterior cruciate ligament reconstruction including a transtibial technique.

41. A system for preparing a physical joint for reconstruction of anterior cruciate ligament in a patient, the system comprising: at least one processor, at least one see-through optical head-mounted display, and at least one user interface, wherein the system is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual axis, wherein the virtual axis is a three-dimensional digital representation that indicates a predetermined position, a predetermined orientation, or a combination thereof of at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof relative to at least one anatomical structure of the physical joint, wherein the system is configured to allow the virtual axis to be aligned with at least one of a predetermined femoral tunnel, a predetermined tibial tunnel, a predetermined position of an anterior cruciate ligament graft, or a predetermined orientation of an anterior cruciate ligament graft, wherein the system is configured to display the virtual axis on the surface of the physical joint at the predetermined position, predetermined orientation, or a combination thereof by the at least one see-through optical head-mounted display, wherein the system is configured to allow superposition and alignment between the at least one physical surgical drill, drill guide, pin, guide rod, pointer, arthroscope, obturator, reamer, milling cutter, reamer, razor, parallel guide, parallel guide sleeve, tunnel notcher, tibial or femoral tunnel orientation device, femoral aiming piece, tibial aiming piece, compression screw, or a combination thereof and at least a portion of the virtual axis displayed by the at least one see-through optical head mounted display, The system is configured to arthroscopically identify the at least one anatomical structure and intraoperatively register the at least one anatomical structure in a coordinate system.

Images