Systems and methods for executing extended reality registration and pose estimation

Abstract

A system defines a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached. The system identifies, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion. The system generates, for display using the AR device, a three-dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further comprises the virtual marker and at least one of the virtual anatomical portion and the virtual guide. The system executes a registration process comprising aligning the virtual marker with the first physical marker.

Description

SYSTEMS AND METHODS FOR EXECUTING EXTENDED REALITY REGISTRATION AND POSEESTIMATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Application No. 63 / 712,431, filed on October 26, 2024, and United States Provisional Application No. 63 / 560,772, filed on March 4, 2024, which is herein incorporated by reference.FIELD OF TECHNOLOGY

[0002] The present disclosure relates to the field of computer vision, and, more specifically, to systems and methods for executing registration of a virtual medical overlay and subsequent pose estimation of a surgical instrument relative to the virtual medical overlay.BACKGROUND

[0003] Accurate intraoperative transfer of preoperative planning data is critical for proper execution of surgical procedures. Projection of preoperative planning data on the patient using extended reality may enable better surgical outcomes. The key step in use of extended reality (XR) for surgical guidance is registration, which refers to the superposition of three-dimensional preoperative planning data in the anatomically correct position relative to the actual patient.

[0004] Current methods of intraoperative XR registration typically rely on one of three general approaches: (1) precise indication of anatomical landmarks through a selection process using a stylus or tool of known geometry tracked by the XR headset, and alignment of the indicated points to the same landmark points determined preoperatively on virtual models; (2) use of so-called "markerless" techniques that utilize computer vision techniques to detect landmark points automatically without human intervention with a tool or stylus but performing point-to-point alignment as with method (1); (3) use of an adjustable standard fixture that remains rigidly attached to a portion of the patient anatomy and uses a fiducial marker or other indicator to provide a location.

[0005] All of these general methods have shortcomings. For instance, use of a stylus or tool to indicate landmarks relies on the judgment and physical coordination of the operator to accurately select points, and discrepancy between the physical patient and virtual models (e.g., due to swelling) can lead to registration error. Furthermore, this approach is time consuming in the operating room and disrupts the surgical workflow. General purpose fixtures or clamps can suffer from accuracy issues due to rotational error or poor fixation of the clamp to the patient anatomy. As a result, this approach is generally utilized for scenarios where the patient is immobile or in specific instances where rigid fixation to multiple bony landmarks can be reliably achieved. Markerless alignment techniques are almost universally the least accurate due to reprojection error and perspective shifts in landmark identification, coupled with processing limitations of most XR headsets.SUMMARY

[0006] The present disclosure describes a patient-specific guide design tailored for use as a part of an extended reality (XR) system. The patient-specific guide includes a clamp, whose design facilitates ease of use for a surgeon with a high degree of accuracy and precision in placement, promoting confidence for the operator, and accurate intraoperative registration and subsequent re-registration (i.e., tracking) of the preoperative data projected in the operator's view using extended reality. This general principle of manufacturing a patient-specific clamp or guide that attaches to the patient's skin or bone in a unique manner is not limited to the preferred embodiment described herein.

[0007] Additionally, the present disclosure describes a method for the production and assembly of the patient-specific guide and clamp design. For example, a volumetric surface representation of the patient's bone (e.g., femur) may be extracted from preoperative imaging using segmentation techniques, and this volumetric representation along with other reference information (such as a drilling guide) are created as 3D model files that may be displayed in extended reality. Additionally, geometric reference data necessary for placement of preoperative data in an extended reality system are determined. An optical tracker, such as an ArUco marker or AprilTag, is selected and a virtual surrogate is created. These 3D digital models are assembled into a virtual scene that is displayed in extended reality. The surface representation of the femur is used to construct patient-specific clamps for the optical tracker as described herein.

[0008] The patient-specific clamp or guide may be a single, unitized part manufactured using techniques such as additive manufacturing. The clamp may allow for small adjustments intraoperatively to compensate for manufacturing tolerance or anatomical changes. In other cases, the clamp orguide may include multiple components or parts. It should be appreciated that patientspecific guides and clamps used for XR registration purposes may take many forms and be comprised of varying material compositions depending on the application.

[0009] The present disclosure describes a method for use of a patient-specific clamp and integration with an XR system. In a preferred embodiment, an XR system may include a mixed reality headset, such as the Microsoft HoloLens 2, running software capable of detecting the presence of optical tracker(s) and registering preoperative planning data to them. During surgery, a surgeon may attach a patient-specific clamp or guide onto a designated portion of the patient's skin or bone. The guide may be constructed in a manner as to be unambiguously and uniquely placed in a specific position and rotational orientation in space. The action of placing the guide should be simple and compatible with the surgical workflow. Then, through interaction with the XR software, such as through voice commands or gaze interaction, the surgeon may initiate a registration procedure to align the preoperative data to the patient in the anatomically correct position. The design of the clamp ensures that the optical marker is positioned correctly on the patient and provides confidence to the surgeon.

[0010] The preoperative data that is registered and subsequently tracked to the patient may include, but not be limited to, virtual cutting guides, virtual drilling guides, geometrical references for bony anatomy, virtual overlays of anatomical structures, and a virtual representation of the clamp or portions thereof. The system may enable the operator to toggle on and off different components based on relevance for each phase of the surgical procedure.

[0011] In an exemplary aspect, the techniques described herein relate to a method for registration using a physical guide and an augmented reality (AR) device, the method including: defining a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached; identifying, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion; generating, for display using the AR device, a three-dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further includes the virtual marker and atleast one of the virtual anatomical portion and the virtual guide; executing a registration process including aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion.

[0012] In some aspects, the techniques described herein relate to a method, wherein a second physical marker is attached to a surgical instrument for operating on the physical anatomical portion, further including: defining an offset between the second physical marker and a designated point of contact on the surgical instrument; and tracking the second physical marker in real-time using the AR device.

[0013] In some aspects, the techniques described herein relate to a method, further including: receiving a user input to create a fixed offset between the second physical marker and a virtual target object; in response to the user input, determining pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset, wherein the virtual target object is at least a portion of the virtual anatomical portion.

[0014] In some aspects, the techniques described herein relate to a method, further including: generating, for display using the AR device, at least one of a second virtual marker corresponding to the second physical marker and a virtual overlay corresponding to the surgical instrument.

[0015] In some aspects, the techniques described herein relate to a method, wherein the physical anatomical portion is hidden or occluded from plain sight of a user of the AR device.

[0016] In some aspects, the techniques described herein relate to a method, wherein aligning the virtual marker with the first physical marker includes minimizing a difference in a position and orientation between the virtual marker and the first physical marker.

[0017] In some aspects, the techniques described herein relate to a method, wherein the physical guide and the first physical marker is constructed to match the virtual guide and the virtual marker.

[0018] In some aspects, the techniques described herein relate to a method, wherein the physical guide is produced using additive manufacturing techniques and is made up of polymeric materials with sufficient mechanical strength to maintain a threshold compressive clamping force during surgery.

[0019] In some aspects, the techniques described herein relate to a method, wherein the physical guide includes a clamp equipped with the first physical marker which is detectable by a camera of the AR device.

[0020] In some aspects, the techniques described herein relate to a method, wherein the virtual marker is one of an optical code, an infrared retroreflective sphere, an electromagnetic tracker, an AprilTag, an ArUco marker, or a QR code.

[0021] In some aspects, the techniques described herein relate to a method, wherein the virtual marker matches a visual appearance of the first physical marker.

[0022] In some aspects, the techniques described herein relate to a method, further including outputting an alert indicative of the alignment in response to detecting that the first physical marker aligns with the virtual marker.

[0023] In some aspects, the techniques described herein relate to a method, wherein the alert is one or more of: a visual indication, an audio indication, and a haptic indication.

[0024] In some aspects, the techniques described herein relate to a method, wherein the three-dimensional representation of the medical imaging data is generated using a plurality of individual medical images of at least the physical anatomical portion.

[0025] It should be noted that the methods described above may be implemented in a system comprising at least one hardware processor and memory. Alternatively, the methods may be implemented using computer executable instructions of a non-transitory computer readable medium.

[0026] In some aspects, the techniques described herein relate to a system for registration using a physical guide, including: an augmented reality (AR) device; at least one memory; and at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to: define a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached; identify, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion; generate, for display using the AR device, a three-dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further includes the virtual marker and at least one of the virtual anatomical portion and the virtual guide; execute a registration process including aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion.

[0027] In some aspects, the techniques described herein relate to a non-transitory computer readable medium storing thereon computer executable instructions for registration using a physical guide and an augmented reality (AR) device, including instructions for: defining a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached; identifying, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion; generating, for display using the AR device, a three- dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further includes the virtual marker and at least one of the virtual anatomical portion and the virtual guide; executing a registration process including aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion.

[0028] The above simplified summary of example aspects serves to provide a basic understanding of the present disclosure. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key or critical elements of all aspects nor delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present one or more aspects in a simplified form as a prelude to the more detailed description of the disclosure that follows. To the accomplishment of the foregoing, the one or more aspects of the present disclosure include the features described and exemplarily pointed out in the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate one or more example aspects of the present disclosure and, together with the detailed description, serve to explain their principles and implementations.

[0030] FIG. 1 illustrates a preferred embodiment of a patient-specific clamp design for the femoral neck.

[0031] FIG. 2 illustrates a 3D model of a bony anatomy of a patient, exemplified by a femoral bone model.

[0032] FIG. 3 illustrates a 3D virtual volume constructed around a portion of a patient anatomy.

[0033] FIG. 4 illustrates the concept of a virtual negative of a patient anatomy.

[0034] FIG. 5 illustrates the method of using the virtual negative to construct a patient-specific guide or clamp and to establish a virtual relationship between the guide and other anatomical or planning data.

[0035] FIG. 6 illustrates the patient-specific clamp or guide attached to the patient anatomy, exemplified by a femoral guide clamped to a femoral neck.

[0036] FIG. 7 illustrates the use of the patient-specific guide to register anatomical overlays, planning data, virtual guides, and other objects onto the physical anatomy of the patient.

[0037] FIG. 8 is a flowchart illustrating a method for executing registration using a physical guide.

[0038] FIG. 9 provides a flowchart showing key steps in the method of utilizing the patientspecific guide or clamp with an extended reality system.

[0039] FIG. 10 provides a flowchart showing key steps in the method of manufacture and assembly of the patient-specific clamp for the extended reality system.

[0040] FIG. 11 shows the core principle of manipulating a structure using augmented reality, as viewed from the perspective of the wearer of the head-mounted display.

[0041] FIG. 12 illustrates the concept of registration of virtual anatomy to a patient.

[0042] FIG. 13A illustrates affixing a fiducial marker(s) to an instrument, and FIG. 13B illustrates the detection of the affixed fiducial for the purpose of tracking the instrument's pose.

[0043] FIG. 14 illustrates further the principle of manipulating a structure using augmented reality, as viewed from the perspective of the wearer of the head-mounted display.

[0044] FIG. 15 is a flowchart illustrating a method for pose estimation of a surgical instrument relative to a registered anatomical portion.

[0045] FIG. 16 shows a flowchart indicating the core steps in the method of using augmented reality to manipulate a structure during a surgical procedure.

[0046] FIG. 17 is a flowchart illustrating a method for pose estimation of a surgical instrument relative to a registered anatomical portion.

[0047] FIG. 18 is a block diagram illustrating a computer system on which aspects of systems and methods of the present disclosure may be implemented in accordance with an exemplary aspect.DETAILED DESCRIPTION

[0048] Exemplary aspects are described herein in the context of a system, method, and computer program product for executing registration of a virtual medical overlay and subsequent pose estimation of a surgical instrument relative to the virtual medical overlay. Those of ordinary skill in the art will realize that the following description is illustrative only and is not intended to be in any way limiting. Other aspects will readily suggest themselves to those skilled in the art having the benefit of this disclosure. Reference will now be made in detail to implementations of the example aspects as illustrated in the accompanying drawings. The same reference indicators will be used to the extent possible throughout the drawings and the following description to refer to the same or like items.

[0049] FIG. 1 shows a three-dimensional view of a preferred, non-limiting embodiment of the patient-specific guide or clamp assembly. Patient-specific guide 100 is a single component or unitized assembly comprised of clamp 110 for attaching to a predefined region of the patient anatomy (e.g., a neck segment with attached optical marker 120).

[0050] In a preferred embodiment, guide 100 is produced using additive manufacturing techniques, principally polymeric materials with sufficient mechanical strength to maintain adequate compressive clamping force by clamp 110 during surgery, while accommodating for some expansion during install. Additionally, selected materials should have sufficient thermal and chemical stability to withstand sterilization processes. Such materials may include, but are not limited to, poly(lactic acid) (PLA), acrylonitrile butadiene styrene (ABS) polymer, poly(phenylene sulfide) (PPS), poly(ether ether ketone) (PEEK), poly(propylene) (PP) and formulated cross-linkable derivatives thereof, known in the art as "resin" in the context of additive manufacturing.

[0051] For simplicity, the present disclosure describes a specific example of guide 100 shaped after a femur. However, guide 100 may be created for any part of the human anatomy and is not limited to the femur (e.g., dental splints, nasal bridge saddles, etc.).

[0052] By way of example, FIG. 2 illustrates a 3D model 200 of the patient's femur determined from preoperative clinical imaging (e.g., computed tomography (CT) data, magnetic resonance imaging (MRI) data, etc.) and using methods widely known in the art to segment this data into 3D volumes (i.e., meshes) for further processing. 3D model 200 may include several 3D model subcomponents obtained from different sources using different processes. For instance, model 200 may include a three-dimensional tessellated surface representation of the patient's own hip, pelvic, and femoral bones' outer and inner surfaces.

[0053] Model 200 may use common representations of 3D models, such as .STL or .OBJ format files. Model 200 may be converted from a surface mesh representation into a volumetric representation to facilitate further processing as described herein. The model 200 may be used as input to the process of designing the patient-specific guide as described herein.

[0054] Now referring to FIGs. 3A and 3B, where the concept of a virtual volume is described through two illustrative examples. In one embodiment, shown in FIG. 3A, the virtual volume 210 encapsulating the anatomical volumetric model includes a spherical component 220 and a truncated cone component 230. Spherical component 220 may be sized according to the size of the femoral head, the physical properties of the manufacturing method (e.g., additive manufacturing), and patient-or-disease specific features (e.g., osteophytes or bone spurs that necessitate further tolerances). The relative placement of the components comprising the virtual volume 210 is defined within a common reference frame 211.

[0055] Further referring to FIG. 3A, spherical component 220 may be centered at the coordinates of the femoral head center in virtual space. Truncated cone component 230 may have a base diameter that is identical to the spherical component 220, and a top diameter that is selected according to the femoral neck diameter. The length of truncated cone component 230 may be adjusted to the length of the femoral neck to ensure connectivity. Truncated cone component 230 may be based at the femoral head center and oriented along the femoral neck angle. Once the virtual volume 210 is in the desired position, an intersection volume 240 can be defined between the anatomical volumetric mesh and the virtual volume 210. Intersection volume 240 may be used in subsequent steps to construct the patient-specific guide 100.

[0056] FIG. 3B provides an alternative embodiment of a virtual volume 250. Virtual volume 250 is an elliptical cylinder located over the humerus virtual volumetric model 202 at a specified orientation and position. Similar to FIG. 3A, an intersection between virtual model 202 and virtual volume 250 can be computed to create an intersection volume 260.

[0057] Components of virtual volume 210 can be constructed in 3D Computer Aided Design (CAD) software including, but not limited to, parametric design software such as SolidWorks, Siemens NX and others. The virtual volume may be combined in Computer Aided Manufacturing (CAM) software such as, but not limited to, PTC Creo, Fusion 360, Siemens NX CAM, and others.

[0058] FIG. 4 provides an illustration of the concept of a virtual negative. Using the virtual volume 210 illustrated in FIG. 3A and subtracting the intersection volume 240 illustrated in FIG. 3A or 260 in FIG. 3B, a virtual negative 270 may be created which provides a concave 'imprint' of theshape and surface morphology of the femoral head and neck. Creation of the virtual negative 270 requires virtual volume 210 and model 200 (volumetric) to be imported into a common coordinate system with a shared spatial reference and may require adjustment of tolerances according to the specified manufacturing process and / or material selection.

[0059] FIG. 5 shows two views of a preferred embodiment of the virtual model of the patientspecific guide 300 which is used to construct physical patient specific guide 100. In this exemplary embodiment, the model of the patient-specific guide 300 includes a clamp 310 and neck 320 which has a fiducial marker 330 attached. By way of example, the fiducial marker may be an AprilTag, an ArUco marker, a QR code, or other binary optical code. The femoral clamp section 310 is designed to fit specifically and without interference to the patient's femoral neck contour, and the fit can be verified virtually using the virtual anatomical model 340 created from model 200 (virtual surface mesh) when aligned within the common virtual coordinate space 211.

[0060] By defining a virtual relationship between the patient anatomy (e.g., virtual femur) and the virtual patient-specific guide, a registration of virtual objects can be achieved relative to the fiducial marker position 330. Specifically, a position and orientation of the virtual anatomical model 340 can be defined relative to position 330, which allows for holographic overlays of the anatomical model for reference. Further, planning data 350 and other reference data 351 can be placed in the virtual scene relative to marker position 330, which allows for holographic overlays of said planning data during surgery. Planning data may include, but is not limited to, implant positions, cutting guides (planes or curves), drilling locations (trajectory and hole placement), anatomical planning angles (e.g., varus angle), and other data.

[0061] FIG. 6 provides an illustration of the patient-specific guide attached to the patient anatomy in an exemplary embodiment. Guide 100 is attached to the patient anatomy 130 through a mechanical action and is attached in a position and orientation that mirrors the arrangement in the virtual scene as described in FIG. 5.

[0062] Referring to FIG. 7, the principle of holographic registration using guide 100 with attached physical optical marker 120 is illustrated. First, guide 100 is attached to target anatomy 130. Then, the surgeon 400 wearing extended reality headset 410 issues a command through a gesture or voice interface to perform registration of the holographic patient and planning data onto the physical patient based on the detection of the optical marker 120. Holographic overlay 340 and planning data virtual overlays 350 and 351 are then registered to the physical patient as viewed by surgeon 400 through extended reality headset 410.

[0063] FIG. 8 is a flowchart illustrating a method 800 for executing registration using a physical guide. Method 800 may be a executed by a registration module / application installed on an AR device. At 802, the registration module defines a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached. In some aspects, the virtual marker is one of an optical code, an infrared retroreflective sphere, an electromagnetic tracker, an AprilTag, an ArUco marker, or a QR code.

[0064] At 804, the registration module identifies, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion.

[0065] Consider a scenario where a surgeon is using an augmented reality (AR) device to perform a complex procedure. At step 802, the registration module within the AR system defines the position and orientation of a virtual marker, such as an ArUco marker, which is fixed on a virtual guide. This virtual guide is aligned with a virtual anatomical model of the patient's knee. For instance, the virtual marker is positioned at coordinates (10, 15, 5) with an orientation of 30 degrees relative to the virtual knee joint. At step 804, the AR device identifies a physical guide placed on the patient's actual knee. This physical guide has a first physical marker, such as a QR code, which is positioned at the same coordinates (10, 15, 5) and shares the same 30-degree orientation relative to the physical knee joint. This alignment ensures that the virtual and physical guides are synchronized, allowing the surgeon to accurately overlay digital information onto the patient's anatomy during the procedure.

[0066] In some aspects, the physical guide and the first physical marker is constructed to match the virtual guide and the virtual marker. Accordingly, the virtual marker matches a visual appearance of the first physical marker. In some aspects, the physical guide is produced using additive manufacturing techniques and is made up of polymeric materials with sufficient mechanical strength to maintain a threshold compressive clamping force during surgery. In some aspects, the physical guide comprises a clamp equipped with the first physical marker which is detectable by a camera of the AR device.

[0067] At 806, the registration module generates, for display using the AR device, a three- dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further comprises the virtual marker and at least one of the virtual anatomical portion and the virtual guide. In some aspects, the three-dimensionalrepresentation of the medical imaging data is generated using a plurality of individual medical images of at least the physical anatomical portion.

[0068] At 808, the registration module executes a registration process comprising aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion. In some aspects, aligning the virtual marker with the first physical marker comprises minimizing a difference in a position and orientation between the virtual marker and the first physical marker.

[0069] At step 808, the registration module performs a registration process to align the virtual marker with the first physical marker, which is attached to the patient's back. For example, this alignment process involves adjusting the virtual marker's position and orientation to match those of the physical marker, such as ensuring both are at coordinates (10, 15, 5) with an orientation of 30 degrees. By minimizing the differences in position and orientation between the virtual and physical markers, the system ensures that the virtual anatomical model is accurately aligned with the patient's physical anatomy.

[0070] In some aspects, the registration module further outputs an alert indicative of the alignment in response to detecting that the first physical marker aligns with the virtual marker. For example, the alert is one or more of: a visual indication, an audio indication, and a haptic indication.

[0071] FIG. 9 describes a method 900 for utilizing a patient-specific guide with an extended reality system to provide registered holographic overlays. Method 900 details use of the assembled clamp embodiment 100 as illustrated in FIG. 1, but it should be appreciated that this method may be extensible to other clamp designs not explicitly disclosed.

[0072] Referring to the steps of FIG. 9, in step 902, preoperative virtual surgical planning is performed using methods well-known in the art. Virtual surgical planning may produce geometrical information about the placement of patient-specific features, such as femoral head center, femoral reference angle, femoral neck angle, femoral head diameter, and others, using mathematical representations such as quaternions and homogeneous matrix representations well-known in the art. Preoperative surgical planning data determined in step 902 may be used in subsequent steps of the method.

[0073] Consider steps 904 and 906 of FIG. 9 together. Relevant patient-specific geometrical data may be imported into a virtual coordinate space 211, which allows for different data sources including 3D models and geometrical data to be referenced to a common coordinate system. Virtualvolumes 210 may be constructed with geometric primitives to represent key aspects of the femoral structure and used to further define reference information such as the femoral neck axis, the femoral head center, the anteversion angle, and femoral cup angle. Other representations may include procedure specific representations, such as specific implant size (selected from a catalog of anatomically adjusted implants), axis of implantation, acetabular cup placement angle, and other surgical planning representations.

[0074] In step 908, clamp 110 is affixed as described herein to the patient's femur. Further, clamp 110 is equipped with optical marker element 120, which can be detected in real-time by the extended reality system (e.g., the photo-video camera of Microsoft HoloLens 2). In prior steps 904 and 906, a virtual optical marker is aligned to the geometric data and virtual volumes within virtual reference space 211. As per step 910, this correspondence can allow for real-time visualization of the preoperative data during a surgical procedure, as the extended reality system can detect the physical optical marker 120 and use its estimated pose to present the preoperative data aligned in the correct position and orientation relative to the patient's femur.

[0075] Referencing FIG. 10, a flowchart for the method of manufacture, assembly and use 1000 of the patient-specific guide for extended reality registration design is provided. Steps 1002 to 1010 provide key milestones in the process of manufacture and use; however, it should be appreciated that additional steps are required for manufacture that are not explicitly disclosed on the flowchart, and certain steps may be combined or skipped based on the method of manufacture chosen.

[0076] Further referencing FIG. 10, the method begins with step 1002 in retrieving imaging data and segmenting into a 3D volume (mesh) representation as described herein. The output of Step 1002 is a 3D volumetric model of the patient anatomy 200 as illustrated in FIG. 2. In step 1004, virtual negative 270 is produced as described herein and as illustrated for a preferred embodiment in FIG. 4.

[0077] Step 1006 discloses use of CAD or CAM software tools to utilize patient-specific virtual negative 270 to contour the clamping surfaces of the patient-specific guide by using CAD or CAM software to generate extrusions from the virtual negative. Step 1008 discloses that the patientspecific guide 100 may be produced by additive manufacturing and an optical marker 120 attached for detection by the XR headset. The attachment of an optical marker is not limiting and other physical objects that may provide guidance, such as a drilling guide, may be attached to the clamp body. As described herein, a variety of material compositions, design variations and CAD / CAM tools may be employed in the preceding stages of this method. In step 1010, patient-specific guide 100is attached to the corresponding anatomical location on the physical patient to provide a visual reference for registration of holographic overlays of anatomical features and planning data as described herein.

[0078] It should be appreciated that method 1000 may be partially or fully automatable by, e.g., use of image processing and / or machine learning algorithms for segmentation, clamp design, geometric representations, and / or optical marker design and selection. The method 1000 is not intended to be limiting but merely illustrative of the method of manufacture of a preferred embodiment of a patient-specific guide design for use with XR guidance. Variations in both design and manufacture as disclosed herein may modify the sequence of steps in method 1000 and possibly add or subtract steps. It is anticipated that those skilled in the art may recognize the impact of changes to the patient-specific guide design within the scope of the present disclosure may impact method 1000 as explicitly taught in FIG. 10.

[0079] Many applications of augmented reality in surgery consider the use of fiducials to track instrumentation, thereby allowing a surgeon to "navigate" or determine their position in space with an instrument. Such applications utilize a known offset between a fiducial marker, such as an ArUco marker or set of infrared-reflective spheres, and a physical tip of an instrument. This enables the instrument's tip (or other point of interest) to be navigated relative to a three-dimensional overlay within an augmented reality display. This offset is determined in the process of designing an instrument or is reverse engineered from an existing instrument, outside of the context of its utilization in a surgical procedure, and is a fixed parameter used in the application.

[0080] By contrast, this present disclosure claims a method for establishing an a priori unknown offset between a point of interest (e.g. an implant or an anatomical structure) and a fiducial marker(s), within the context of utilization of the augmented reality application for a surgical procedure. In this present disclosure, the surgeon can create an offset on-demand between a fiducial marker(s) and the point of interest. This means the offset that was designed into the instrument or reverse engineered from an existing one is not needed, and only the fiducial marker(s) which will be used to establish the offset will be required. This variability in offset enables valuable new use cases with augmented reality in surgery, particularly the ability to perform novel surgical techniques that have the potential to reduce harm to the patient.

[0081] The present disclosure is a system comprised of an augmented reality (AR) headmounted display with the ability to provide a registered view of three-dimensional virtual anatomy, an instrument with a fiducial marker or set of fiducial markers affixed, software with machine-readable instructions capable of measuring the position in space of the fiducial marker(s), and at least one target object to be manipulated by the instrument. The system advantageously combines these components to allow the surgeon to visualize a virtual overlay, through the AR head-mounted display, of the target object even hidden from view (e.g., obscured by skin).

[0082] The AR head-mounted display is equipped with a set of sensors, which may include but are not limited to optical cameras, infrared cameras and inertial measurement units. A set of machine-readable instructions are provided to the software on the AR headset to recognize fiducial marker(s) attached to the instrument. Using techniques known in the art, the AR headset can thereby track the position of the fiducial marker(s) and estimate the pose of the instrument in realtime. The tip position of said instrument, should one exist, may be part of the machine-readable instructions, but it is not necessary for the purposes of this present disclosure to establish a known correspondence between the fiducial marker(s) and the instrument tip. Additionally, using methods known in the art, a virtual overlay of the patient anatomy can be registered, or superimposed on, the physical patient using sensor measurements from the AR head-mounted display and following instructions provided by the software on the AR head-mounted display.

[0083] The present disclosure describes a method of use for this system. This method is carried out preferably by a surgeon, surgical resident or other medical practitioner in the context of a surgical operation. In the context of its utilization, the user provides inputs including but not limited to voice commands or gesture inputs are used to establish an offset between the instrument and the target object. Such user inputs are interpreted and responded to by software on the AR headset using techniques known in the art. For instance, once the instrument's tip or a portion of the instrument is in contact with a target anatomical structure, the user can issue a voice command (or other input) to the AR head-mounted display to "capture" or lock a target anatomical structure relative to the instrument. Following the pre-programmed input (sequence), the user can then visualize movement of a portion of a virtual overlay on the patient relative to the remainder of the overlay that remains fixed in position. A subsequent command can be provided to "break" the association between the instrument with affixed fiducial marker(s) and the target.

[0084] Use of this system according to the method enables a surgeon to perform tasks previously difficult or impossible to carry out with hidden or occluded structures. As a non-limiting example, a screw handle may have a set of fiducial markers attached and the position of the screw body relative to the fiducial markers programmed into the machine-readable instructions. Then, using the AR head-mounted display, a surgeon performs a surgical procedure, such as azygomaticomaxillary complex (ZMC) fracture reduction. The surgeon registers a virtual overlay to a patient, including a virtual display of the patient's ZMC, begins tracking the screw handle with fiducials, and upon establishing contact between the tracked screw and the fractured portion of the patient's ZMC, issues a command to the AR head-mounted display to establish an offset between the ZMC fragment and the tracked instrument overlay. This allows the surgeon to see the hidden fragment which is occluded by the patient's skin, and perform fracture reduction using novel minimally-invasive techniques that eliminates the need for an open surgical procedure.

[0085] The present disclosure includes a system and method for manipulating structures in a surgical procedure. The core principle of the present disclosure utilizes a principle of a variable offset between a fiducial marker(s) and a target(s) virtual structure(s), which may be hidden (e.g., Zygomaticomaxillary (ZMC) complex and subcondylar fracture reduction while occluded by the skin), and may optimally correspond to the locations of physical anatomical structures. It should be noted that although examples of hidden structures are provided, the systems and methods may be generalized to structures that are not "hidden" by setting of a bone anchor to manipulate a fragment after the initial registration, such as the manipulation of the femur after dislocation from the acetabulum.

[0086] Referring to FIG. 11, the core principle of the present disclosure is illustrated. A patient 1100 is undergoing a medical procedure and the practitioner, ideally a surgeon or surgical resident, is carrying out the procedure with augmented reality (AR) guidance, and viewing patient 1100 through the lens of the AR head-mounted display. The wearer of the AR headset carries out a registration procedure which establishes a virtual overlay 1120 to patient 1100. The virtual overlay may show a variety of three-dimensional data including segmented anatomical objects, implants, data annotations, and the like. A target virtual anatomical object 1130 is one of the objects displayed on the patient after the registration is performed. Typical sources of three-dimensional overlays 1120 and 1130 are medical imaging data, such as computed tomography (CT) and magnetic resonance imaging (MRI), using segmentation methods known in the art. The virtual overlay will be established using methods known in the art and disclosed elsewhere.

[0087] Still referring to FIG. 11, an instrument with fiducial marker 1110 is used by the surgeon to manipulate target virtual anatomical object 130. The position in space, also known as the pose, of instrument 1110 is tracked using sensors on the AR head-mounted display. In a method described herein, the surgeon creates an offset 1111 between the instrument with fiducial marker 1110 and the target virtual anatomical object 1130. The offset can be established by use of a variety of inputmethods available with the AR head-mounted display, such as voice commands or gesture inputs, which are well known in the art. Subsequent to establishing offset 1111, virtual target object 1130 moves with the motion of instrument 1110. Specifically, the pose computed of instrument with fiducial marker 1110 is used to compute the pose of virtual object 1130, based on the offset position established by the user.

[0088] It should be noted that other registration methods instead of the one taught by FIGS. 8-10 can be used to align anatomy overlays into place before using the methods described in FIGS. 11-16 to manipulate a bone fragment relative to the rest of the anatomy overlays.

[0089] Referring now to FIG. 12, the principle of registration is illustrated. User 1200, typically a surgeon carrying out a procedure, wears AR head-mounted display 1210. Upon looking at patient 1100 through the AR headset 1210, user 1200 can execute a sequence of steps to register virtual overlay 1120 and target anatomy overlay 1130 onto the patient 1100. Common approaches used in the art include automatic registration, landmark-based registration, and manual registration. This present disclosure does not require a specific method of registration but is preferentially done using the most accurate method available.

[0090] Referring now to FIGs. 13A and 13B, the principles of the instrument tracking portion of the present disclosure are illustrated. In FIG. 13A, instrument 1112 gets fiducial marker 1114 affixed to it using a variety of methods, such as using a clamp, adhesive, or even an integrated fiducial into the marker. Instrument 1112 and fiducial marker 1114 may have a variable offset 1116 between them. By contrast with methods of AR instrument tracking described in the prior art, this method is tolerant to changes in the instrument position relative to the fiducial marker 1114. This is advantageous, for example, in an instance where a sliding clamp may be used to affix a fiducial marker to an osteotome.

[0091] Still referring to FIG. 13A, the fiducial marker can take a variety of forms, shapes and sizes. A fiducial marker may be a binary optical marker, such as an ArUco marker, AprilTag, QR Code, or other optical marker with a known pattern and size. Binary optical markers can be efficiently detected by an AR head-mounted display using built-in optical cameras, and software packages such as OpenCV can be used to compute the pose of said markers for the purposes of real-time instrument tracking. Alternatively, and in a preferred embodiment, retroreflective spheres that reflect infrared light can be used to define a plane, which can be computed to infer the pose of the instrument. Many AR head-mounted displays, such as the Microsoft HoloLens 2, are equipped with infrared sensors for depth measurement, and this sensor can be used to track objects includingreflective spheres. These examples are not meant to be limiting in any way of the types of fiducial markers that can be used for instrument tracking to practice this present disclosure.

[0092] Now referring to FIG. 13B, real-time instrument tracking is shown. On AR headset 1210, sensors 1220 track the position of fiducial marker 1114 as described in the preceding paragraph. As the position of fiducial marker 1114 is tracked, a virtual fiducial overlay 1150 may appear along with a virtual instrument position overlay 1152. The virtual overlays 1150, 1152 help provide the user confirmation of tracking in real-time, along with feedback on tracking accuracy and latency.

[0093] Still referring to FIG. 13B, as the physical instrument with fiducial 1110 is moved by the user, the headset 1210 computes the pose in real-time of the instrument 1110, optionally providing overlays 1150, 1152. Instrument 1110 begins in pose 1160, and in a matter of a few seconds, the user moves the instrument into pose 1161, which is determined by software running on the AR headset. This process happens in real-time. In a preferred embodiment, the user sees the virtual overlays 1150, 1152 update at approximately 60 frames per second as they move the instrument 1110 in space. This tracking speed is not required to practice the present disclosure, and for certain cases, overlays 1150, 1152 may not be desired, or may preferentially be toggled on and off by the user. It should be appreciated that many technical details regarding instrument tracking are configurable to the particular use case and surgeon preference, but the core functionality of the present disclosure requires the position (pose) of instrument with fiducial 1110 to be known to establish an offset to a target virtual object(s).

[0094] Now referring to FIG. 14, the principle of manipulating a hidden object is further illustrated building from the previous discussion and shown through the point of view of the user of the AR headset. The instrument with fiducial 1110 is tracked by the AR headset along with virtual overlay of fiducial 1150 and instrument 1152 as desired. At some point deemed optimal by the surgeon, a user input is provided to create a fixed offset 1111 between the fiducial marker 1110 and the virtual target anatomy 1130. Subsequent to this offset being established, the pose changes computed for the tracked instrument are used to compute a pose of the virtual target object 1130. Initial pose 1160, as computed by the AR headset's tracking of instrument 1110, now corresponds to the initial pose of the virtual target object 1130. As instrument 1110 moves to new pose 1161, the pose of the virtual target object 1130 is now changed in a corresponding manner. This correspondence can be maintained indefinitely or broken and re-established at the will of the user.

[0095] The illustration of FIG. 14 highlights the core utility of the present disclosure. Once the user creates offset 1111 between the instrument 1110 and the virtual target 1130, they can observethe motion of virtual target 1130 corresponding to the instrument 1110. In a specific embodiment, the instrument 1110 may be a screw commonly used to attach to bone. Screw with attached fiducial marker, 1110, can then be used to attach to a hidden bone fragment on the patient. In this specific embodiment, this physical bone fragment may be virtually represented by virtual target overlay 1130. While the physical bone fragment is not visible to the user, the virtual overlay 1130 is visible, and the surgeon can use the position of the virtual overlay to place the screw in the correct position, further assessing correct placement through surgical experience and tactile feedback. Once the screw is in the desired position, the surgeon can issue a command to create offset 1111, and from this point, virtual target object 1130 will move with instrument 1110. The surgeon can then use the screw to perform fracture reduction (closure) by observing the movement of virtual target overlay 1130. Specifically in this embodiment, the virtual target overlay 1130 is observed to move relative to the rest of the anatomy overlay 1120.

[0096] FIG. 15 is a flowchart illustrating a method 1500 for pose estimation of a surgical instrument relative to a registered anatomical portion. At 1502, the registration module, which may have pose estimation capabilities, may register a virtual anatomical portion on a physical anatomical portion. In some aspects, the physical anatomical portion is hidden or occluded from plain sight of a user of the AR device. In some aspects, this registration is performed using method 800 described in FIG. 8.

[0097] At 1504, the registration module defines an offset between a second physical marker attached to a surgical instrument and a designated point of contact on the surgical instrument for operating on the physical anatomical portion. For example, the distance between a needle of the surgical instrument and the second physical marker may be 5 inches.

[0098] At 1506, the registration module tracks the second physical marker in real-time using the AR device.

[0099] At 1508, the registration module receives a user input to create a fixed offset between the second physical marker and a virtual target object. For example, consider a scenario where a surgeon is using an AR device to perform a delicate procedure on a patient's liver. The second physical marker is attached to a laparoscopic tool, and the surgeon inputs a command to set a fixed offset of 3 inches between this marker and a virtual representation of a tumor within the liver. This offset ensures that the virtual target object, representing the tumor, maintains a consistent spatial relationship with the physical marker, allowing the surgeon to visualize the tumor's position relative to the tool in real-time.

[0100] In response to the user input, at 1510, the registration module determines pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset. In some aspects, the virtual target object is at least a portion of the virtual anatomical portion. For example, as the surgeon manipulates the laparoscopic tool, the AR system tracks the marker's position and orientation changes. For instance, if the tool is rotated or moved closer to the liver, the virtual target object (the tumor) will adjust its pose accordingly, maintaining the predefined 3-inch offset. This dynamic adjustment allows the surgeon to see an accurate, real-time overlay of the tumor's position relative to the surgical instrument, enhancing precision and control during the procedure. By continuously updating the virtual target object's pose in response to the physical marker's movements, the registration module provides a seamless integration of virtual and physical environments, aiding the surgeon in making informed decisions during the operation.

[0101] In some aspects, the registration module may generate, for display using the AR device, at least one of a second virtual marker corresponding to the second physical marker and a virtual overlay corresponding to the surgical instrument.

[0102] The flowchart of FIG. 16 describes the method for manipulating a hidden object using AR, making reference to schematics in previous figures. In step 1602, preoperative imaging data, such as CT or MRI data, is prepared into a format such that it can be viewed in AR. This will typically involve a process known as segmentation, which yields a three-dimensional mesh in a standard format (e.g., OBJ, STL, FBX). This step can be done manually by a radiologist or technician skilled with interpreting medical radiographs, or can be done automatically using software designed for segmentation (e.g., utilizing machine learning models to identify and segment features). In step 1602, a set of three-dimensional models of primary anatomy 1120 and target anatomy 1130 are created for the registration step. It should be appreciated by those skilled in the art that other methods of three-dimensional representation may be substituted here, such as volumetric rendering.

[0103] Still referring to FIG. 16, in step 1604, the three-dimensional representations yielded from step 1602 are registered to the patient. This involves loading the prepared data into software that runs on the AR headset as machine-readable instructions. Part of the instruction set includes methods for performing registration, along with instructions for anchoring the virtual anatomy overlays 1120 and 1130 to the patient subsequent to registration (e.g., provisions for re-registration or tracking should the patient be moved). It should be recognized to one skilled in the art that known registration techniques along with known re-registration and tracking techniques (e.g., areference array or anatomy anchor) can be used with this method if provisions are needed to account for patient movement during the procedure.

[0104] In step 1606, the AR headset 1210 tracks the position of at least one predefined fiducial marker 1114 as described previously. Pose estimation of the marker(s) can be performed using software tools such as OpenCV which are known in the art. The specific tracking modality and fiducial design may vary depending on case requirements, but in typical embodiments, an ArUco marker approximately 50mm on a side is affixed to an instrument and tracked using the optical camera on the AR head-mounted display. Alternatively, an array of three infrared retroreflective 13mm spheres with a known geometry is used and tracked with an infrared time-of-flight depth camera on the AR head-mounted display.

[0105] In step 1606, an offset between a target virtual object 1130 and the fiducial marker 1114 is created based on a user input. The user input may be a single action such as a voice command or gesture input, or may be a sequence of commands. The offset, once applied, changes the pose of target virtual object 1130 based on pose changes made to the fiducial marker 1114 by the user. The pose changes made to 1130 relative to 1114 will typically be one characteristic of a rigid body connection between the two (representative of, for instance, a screw inserted into the target physical anatomy). However, other relationships may be established with the offset, such as an offset that moves the target anatomy in an opposite direction, or scales motion by a defined scalar factor (e.g., 1.3x). It should be appreciated by those skilled in the art that a variety of offsets and pose relationships can be established between the fiducial marker 1114 and virtual target object 130. Further, the offset created by the user's input can be subsequently broken at a later time, and also reset subsequently after being broken. In a preferred embodiment, the user has the option to reset or create new offsets and adjust offsets dynamically.

[0106] In step 1610, the user views the position of the target virtual model(s) 1130 through the AR headset 1210 change in real-time as the position of the fiducial marker 1114 is changed. For instance, if the fiducial marker 1114 is affixed to an instrument (e.g., a screw) that is fixated to a bone fragment occluded by skin, the surgeon will be able to move the instrument and watch the position (pose) of the target virtual model(s) 1130 change in a predictable way relative to the instrument with affixed fiducial marker. Further, in a preferred embodiment, the remaining anatomy models 1120 remain static or otherwise stay in a stationary position relative to the patient, while target models 1130 move in concert with the fiducial marker and instrument. In this way, the method effectively allows the surgeon to visualize hidden objects beneath the skin or otheroccluding layers, and to use instruments to manipulate these hidden objects, thereby enabling less harmful surgical techniques to be performed under augmented reality guidance.

[0107] FIG. 17 is a flowchart illustrating a method 1700 for pose estimation of a surgical instrument relative to a registered anatomical portion. At 1702, the registration module registers, for display using an augmented reality (AR) device, a three-dimensional representation of medical imaging data on a physical anatomical portion of a patient. In this case, the medical imaging data comprises a virtual anatomical portion that represents the physical anatomical portion. In some aspects, the registration performed in 1702 involves executing method 800.

[0108] At 1704, the registration module defines an offset between a physical marker attached to a surgical instrument for operating on the physical anatomical portion and a designated point of contact on the surgical instrument.

[0109] At 1706, the registration module tracks, using the AR device, a position and orientation of the physical marker in real-time relative to the virtual anatomical portion. For example, the registration module may receive a user input to create a fixed offset between the physical marker and a virtual target object (i.e., a section or point on the virtual anatomical portion). In response to the user input, the registration module determines pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset.

[0110] At 1708, the registration module determines a position and orientation of the designated point of contact on the surgical instruction relative to the physical anatomical portion based on the offset. Because the position and orientation of the physical marker and the virtual anatomical portion is known based on the fixed offset, and because the virtual anatomical portion is properly registered on the physical anatomical portion, the position and orientation of the point of contact of the surgical instrument (e.g., the tip of the needle) can be determined relative to the physical anatomical portion.

[0111] FIG. 18 is a block diagram illustrating a computer system 20 on which aspects of systems and methods for executing registration of a virtual medical overlay and subsequent pose estimation of a surgical instrument relative to the virtual medical overlay may be implemented in accordance with an exemplary aspect. The computer system 20 may be in the form of multiple computing devices, or in the form of a single computing device, for example, a desktop computer, a notebook computer, a laptop computer, a mobile computing device, a smart phone, a tablet computer, a server, a mainframe, an embedded device, and other forms of computing devices.

[0112] As shown, the computer system 20 includes a central processing unit (CPU) 21, a systemmemory 22, and a system bus 23 connecting the various system components, including the memory associated with the central processing unit 21. The system bus 23 may comprise a bus memory or bus memory controller, a peripheral bus, and a local bus that is able to interact with any other bus architecture. Examples of the buses may include PCI, ISA, PCI-Express, HyperTransport™, InfiniBand™, Serial ATA, l2C, and other suitable interconnects. The central processing unit 21 (also referred to as a processor) can include a single or multiple sets of processors having single or multiple cores. The processor 21 may execute one or more computer-executable code implementing the techniques of the present disclosure. For example, any of commands / steps discussed in FIGS. 1-16 may be performed by processor 21. The system memory 22 may be any memory for storing data used herein and / or computer programs that are executable by the processor 21. The system memory 22 may include volatile memory such as a random access memory (RAM) 25 and non-volatile memory such as a read only memory (ROM) 24, flash memory, etc., or any combination thereof. The basic input / output system (BIOS) 26 may store the basic procedures for transfer of information between elements of the computer system 20, such as those at the time of loading the operating system with the use of the ROM 24.

[0113] The computer system 20 may include one or more storage devices such as one or more removable storage devices 27, one or more non-removable storage devices 28, or a combination thereof. The one or more removable storage devices 27 and non-removable storage devices 28 are connected to the system bus 23 via a storage interface 32. In an aspect, the storage devices and the corresponding computer-readable storage media are power-independent modules for the storage of computer instructions, data structures, program modules, and other data of the computer system 20. The system memory 22, removable storage devices 27, and non-removable storage devices 28 may use a variety of computer-readable storage media. Examples of computer-readable storage media include machine memory such as cache, SRAM, DRAM, zero capacitor RAM, twin transistor RAM, eDRAM, EDO RAM, DDR RAM, EEPROM, NRAM, RRAM, SONOS, PRAM; flash memory or other memory technology such as in solid state drives (SSDs) or flash drives; magnetic cassettes, magnetic tape, and magnetic disk storage such as in hard disk drives or floppy disks; optical storage such as in compact disks (CD-ROM) or digital versatile disks (DVDs); and any other medium which may be used to store the desired data and which may be accessed by the computer system 20.

[0114] The system memory 22, removable storage devices 27, and non-removable storage devices 28 of the computer system 20 may be used to store an operating system 35, additionalprogram applications 37, other program modules 38, and program data 39. The computer system 20 may include a peripheral interface 46 for communicating data from input devices 40, such as a keyboard, mouse, stylus, game controller, voice input device, touch input device, or other peripheral devices, such as a printer or scanner via one or more I / O ports, such as a serial port, a parallel port, a universal serial bus (USB), or other peripheral interface. A display device 47 such as one or more monitors, projectors, or integrated display, may also be connected to the system bus 23 across an output interface 48, such as a video adapter. In addition to the display devices 47, the computer system 20 may be equipped with other peripheral output devices (not shown), such as loudspeakers and other audiovisual devices.

[0115] The computer system 20 may operate in a network environment, using a network connection to one or more remote computers 49. The remote computer (or computers) 49 may be local computer workstations or servers comprising most or all of the aforementioned elements in describing the nature of a computer system 20. Other devices may also be present in the computer network, such as, but not limited to, routers, network stations, peer devices or other network nodes. The computer system 20 may include one or more network interfaces 51 or network adapters for communicating with the remote computers 49 via one or more networks such as a local-area computer network (LAN) 50, a wide-area computer network (WAN), an intranet, and the Internet. Examples of the network interface 51 may include an Ethernet interface, a Frame Relay interface, SONET interface, and wireless interfaces.

[0116] Aspects of the present disclosure may be a system, a method, and / or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0117] The computer readable storage medium may be a tangible device that can retain and store program code in the form of instructions or data structures that may be accessed by a processor of a computing device, such as the computing system 20. The computer readable storage medium may be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. By way of example, such computer-readable storage medium can comprise a random access memory (RAM), a read-only memory (ROM), EEPROM, a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), flash memory, a hard disk, a portable computer diskette, a memory stick, a floppy disk, or even a mechanically encoded device such as punch-cardsor raised structures in a groove having instructions recorded thereon. As used herein, a computer readable storage medium is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or transmission media, or electrical signals transmitted through a wire.

[0118] Computer readable program instructions described herein may be downloaded to respective computing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network interface in each computing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing device.

[0119] Computer readable program instructions for carrying out operations of the present disclosure may be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language, and conventional procedural programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a LAN or WAN, or the connection may be made to an external computer (for example, through the Internet). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0120] In various aspects, the systems and methods described in the present disclosure may be addressed in terms of modules. The term "module" as used herein refers to a real-world device, component, or arrangement of components implemented using hardware, such as by an application specific integrated circuit (ASIC) or FPGA, for example, or as a combination of hardwareand software, such as by a microprocessor system and a set of instructions to implement the module's functionality, which (while being executed) transform the microprocessor system into a special-purpose device. A module may also be implemented as a combination of the two, with certain functions facilitated by hardware alone, and other functions facilitated by a combination of hardware and software. In certain implementations, at least a portion, and in some cases, all, of a module may be executed on the processor of a computer system. Accordingly, each module may be realized in a variety of suitable configurations, and should not be limited to any particular implementation exemplified herein.

[0121] In the interest of clarity, not all of the routine features of the aspects are disclosed herein. It may be appreciated that in the development of any actual implementation of the present disclosure, numerous implementation-specific decisions must be made in order to achieve the developer's specific goals, and these specific goals will vary for different implementations and different developers. It is understood that such a development effort may be complex and timeconsuming, but may nevertheless be a routine undertaking of engineering for those of ordinary skill in the art, having the benefit of this disclosure.

[0122] Furthermore, it is to be understood that the phraseology or terminology used herein is for the purpose of description and not of restriction, such that the terminology or phraseology of the present specification is to be interpreted by the skilled in the art in light of the teachings and guidance presented herein, in combination with the knowledge of those skilled in the relevant art(s). Moreover, it is not intended for any term in the specification or claims to be ascribed an uncommon or special meaning unless explicitly set forth as such.

[0123] The various aspects disclosed herein encompass present and future known equivalents to the known modules referred to herein by way of illustration. Moreover, while aspects and applications have been shown and described, it may be apparent to those skilled in the art having the benefit of this disclosure that many more modifications than mentioned above are possible without departing from the inventive concepts disclosed herein.

Claims

CLAIMS1. A method for registration using a physical guide and an augmented reality (AR) device, the method comprising: defining a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached; identifying, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion; generating, for display using the AR device, a three-dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further comprises the virtual marker and at least one of the virtual anatomical portion and the virtual guide; executing a registration process comprising aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion.

2. The method of claim 1, wherein a second physical marker is attached to a surgical instrument for operating on the physical anatomical portion, further comprising: defining an offset between the second physical marker and a designated point of contact on the surgical instrument; and tracking the second physical marker in real-time using the AR device.

3. The method of claim 2, further comprising: receiving a user input to create a fixed offset between the second physical marker and a virtual target object; in response to the user input, determining pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset, wherein the virtual target object is at least a portion of the virtual anatomical portion.

4. The method of claim 2, further comprising:generating, for display using the AR device, at least one of a second virtual marker corresponding to the second physical marker and a virtual overlay corresponding to the surgical instrument.

5. The method of claim 2, wherein the physical anatomical portion is hidden or occluded from plain sight of a user of the AR device.

6. The method of claim 1, wherein aligning the virtual marker with the first physical marker comprises minimizing a difference in a position and orientation between the virtual marker and the first physical marker.

7. The method of claim 1, wherein the physical guide and the first physical marker is constructed to match the virtual guide and the virtual marker.

8. The method of claim 1, wherein the physical guide is produced using additive manufacturing techniques and is made up of polymeric materials with sufficient mechanical strength to maintain a threshold compressive clamping force during surgery.

9. The method of claim 1, wherein the physical guide comprises a clamp equipped with the first physical marker which is detectable by a camera of the AR device.

10. The method of claim 1, wherein the virtual marker is one of an optical code, an infrared retroreflective sphere, an electromagnetic tracker, an AprilTag, an ArUco marker, or a QR code.

11. The method of claim 1, wherein the virtual marker matches a visual appearance of the first physical marker.

12. The method of claim 1, further comprising outputting an alert indicative of the alignment in response to detecting that the first physical marker aligns with the virtual marker.

13. The method of claim 12, wherein the alert is one or more of: a visual indication, an audio indication, and a haptic indication.

14. The method of claim 1, wherein the three-dimensional representation of the medical imaging data is generated using a plurality of individual medical images of at least the physical anatomical portion.

15. A system for registration using a physical guide, comprising: an augmented reality (AR) device; at least one memory; and at least one hardware processor coupled with the at least one memory and configured, individually or in combination, to: define a position and orientation of a virtual marker, which is fixed on a virtual guide, relative to a virtual anatomical portion to which the virtual guide is attached; identify, by the AR device, a physical guide and a first physical marker fixed to the physical guide, wherein the first physical marker shares the position and the orientation of the virtual marker relative to a physical anatomical portion corresponding to the virtual anatomical portion; generate, for display using the AR device, a three-dimensional representation of medical imaging data overlaid on the physical anatomical portion, wherein the medical imaging data further comprises the virtual marker and at least one of the virtual anatomical portion and the virtual guide; execute a registration process comprising aligning the virtual marker with the first physical marker, wherein alignment of the virtual marker and the first physical marker is indicative of alignment of the virtual anatomical portion and the physical anatomical portion.

16. The system of claim 15, wherein a second physical marker is attached to a surgical instrument for operating on the physical anatomical portion, wherein the at least one hardware processor is further configured to: define an offset between the second physical marker and a designated point of contact on the surgical instrument; and track the second physical marker in real-time using the AR device.

17. The system of claim 16, wherein the at least one hardware processor is further configured to:receive a user input to create a fixed offset between the second physical marker and a virtual target object; in response to the user input, determine pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset, wherein the virtual target object is at least a portion of the virtual anatomical portion.

18. The system of claim 16, wherein the at least one hardware processor is further configured to: generate, for display using the AR device, at least one of a second virtual marker corresponding to the second physical marker and a virtual overlay corresponding to the surgical instrument.

19. The system of claim 16, wherein the physical anatomical portion is hidden or occluded from plain sight of a user of the AR device.

20. A method for pose estimation relative to a registered medical overlay, comprising: registering, for display using an augmented reality (AR) device, a three-dimensional representation of medical imaging data on a physical anatomical portion of a patient, wherein the medical imaging data comprises a virtual anatomical portion that represents the physical anatomical portion; defining an offset between a physical marker attached to a surgical instrument for operating on the physical anatomical portion and a designated point of contact on the surgical instrument; tracking, using the AR device, a position and orientation of the physical marker in real-time relative to the virtual anatomical portion; and determining a position and orientation of the designated point of contact on the surgical instruction relative to the physical anatomical portion based on the offset.

21. The method of claim 20, further comprising: receiving a user input to create a fixed offset between the physical marker and a virtual target object;in response to the user input, determining pose changes of the virtual target object based on pose changes of the second physical marker and the fixed offset, wherein the virtual target object is at least a portion of the virtual anatomical portion.

22. The method of claim 21, further comprising: generating, for display using the AR device, at least one of a virtual marker corresponding to the physical marker or a virtual overlay corresponding to the surgical instrument.

23. The method of claim 20, wherein the physical anatomical portion is hidden or occluded from plain sight of a user of the AR device.

Images