Surgical system and method for positioning an object using augmented reality navigation - Patent Application 20070122999

Abstract

Surgical systems and methods are provided that utilize augmented reality navigation and visualization techniques to transfer aspects of a pre-operative surgical plan to an actual surgical site. The surgical systems and methods may be utilized to achieve precise alignment of surgical positioning objects, such as guide pins to guide a surgical reaming procedure between the pre-operative surgical plan and intra-operative anatomical structures associated with the actual surgical site. Augmented reality may be utilized to achieve visualization of both the entry point and drill trajectory of the surgical positioning objects in a manner that avoids occluding the intra-operative anatomical structures during the procedure.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This disclosure claims priority to U.S. Provisional Patent Application No. 63 / 241,758, filed September 8, 2021, and incorporated herein by reference in its entirety.

[0002] The present disclosure relates to surgical systems and methods that utilize augmented reality navigation and visualization techniques to transfer aspects of pre-operative surgical planning to the actual surgical site. [Background technology]

[0003] The present disclosure relates to the field of surgery, and more particularly to orthopaedic surgical systems and methods for intraoperative positioning of objects, such as surgical guide pins, by utilizing augmented reality navigation and visualization techniques.

[0004] Joint arthroplasty is a type of orthopedic surgical procedure performed to repair or replace a diseased joint. Prior to performing joint arthroplasty, a surgeon may wish to establish a preoperative surgical plan related to surgical site preparation, implant selection, and implant placement at the surgical site.

[0005] In some techniques, surgeons may utilize guide pins to guide the reaming of bone surfaces to position the arthroplasty implant at the surgical site. Surgeons may desire improved intraoperative guidance to transfer aspects of the preoperative surgical plan to the surgical site to improve surgical outcomes. Summary of the Invention [Means for solving the problem]

[0006] An exemplary augmented reality system for a surgical system may include, among other things, an augmented reality visualization device and a processor. The processor may be programmed to control the augmented reality visualization device to provide an augmented reality environment relative to a patient's anatomy. The processor may be further programmed to enable a user to interface with the augmented reality environment to intraoperatively achieve a desired entry point and a desired trajectory of a surgical positioning object relative to the patient's anatomy.

[0007] An exemplary surgical method may include, among other things, intraoperatively defining a desired entry point and a desired trajectory of a surgical positioning object within an augmented reality environment generated by an augmented reality system. A visual indication of accuracy between the actual trajectory and the desired trajectory of the surgical positioning object within the augmented reality environment may be provided.

[0008] Another exemplary surgical method may include, among other things, positioning the tip of a surgical guide pin at a location on a bone surface of an anatomical structure indicated by virtual crosshairs, preparing a dent at the location, pivoting the surgical guide pin about a pivot point established by the dent, and aligning the actual trajectory of the surgical guide pin with the virtual trajectory of the surgical guide pin within an augmented reality environment. [Brief explanation of the drawings]

[0009] [Figure 1] 1 schematically illustrates an exemplary orthopaedic surgical system. [Figure 2] 2A and 2B schematically illustrate aspects of an augmented reality system of the orthopaedic surgical system of FIG. 1. [Figure 3] 10A-10C schematically illustrate a targeting process for guiding the positioning of a surgical positioning object. [Figure 4] 2A-2C are schematic illustrations of exemplary surgical methods that may be pre-operatively planned and intra-operatively performed using the orthopaedic surgical system of FIG. 1; [Figure 5]1 illustrates an augmented reality environment that may be provided by an augmented reality system of an orthopaedic surgical system. [Figure 6] 6 illustrates an example embodiment of a registration module of the augmented reality environment of FIG. 5. [Figure 7] 1 illustrates an exemplary initialization step of a multi-step registration process that may be performed by an augmented reality system of an orthopaedic surgical system. [Figure 8] 1 illustrates an exemplary initialization step of a multi-step registration process that may be performed by an augmented reality system of an orthopaedic surgical system. [Figure 9] 1 illustrates an exemplary 3D scanning step of a multi-step registration process that may be performed by an augmented reality system of an orthopaedic surgical system. [Figure 10] 1 illustrates an exemplary 3D scanning step of a multi-step registration process that may be performed by an augmented reality system of an orthopaedic surgical system. [Figure 11] 10 illustrates another augmented reality environment that may be provided by an augmented reality system of an orthopaedic surgical system. [Figure 12] 1 illustrates an exemplary embodiment of a communication module of an augmented reality environment. [Figure 13] 10 illustrates additional aspects of a communication module of an augmented reality environment. [Figure 14] 10 illustrates additional aspects of a communication module of an augmented reality environment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The present disclosure describes orthopaedic surgical systems and methods for providing augmented reality visualization during surgical procedures, such as arthroplasty procedures. The disclosed surgical systems and methods utilize augmented reality navigation and visualization techniques to transfer aspects of pre-operative surgical planning to the patient's intra-operative anatomy.

[0011] In some implementations, the presently disclosed orthopaedic surgical systems and methods may be utilized to achieve precise alignment of surgical positioning objects, such as guide pins, to guide a reaming procedure between a preoperative surgical plan and intraoperative anatomy associated with the actual surgical site. Augmented reality may be utilized to achieve visualization of both the entry point and drill trajectory of the surgical positioning object in a manner that avoids occluding the intraoperative anatomy during the procedure. These and other features of the present disclosure are described in further detail below.

[0012] An augmented reality system for a surgical system according to an exemplary aspect of the present disclosure may include an augmented reality visualization device and a processor. The processor may be programmed to control the augmented reality visualization device to provide an augmented reality environment relative to a patient's anatomy. The processor may be further programmed to enable a user to interface with the augmented reality environment to intraoperatively achieve a desired entry point and a desired trajectory of a surgical positioning object relative to the patient's anatomy.

[0013] In a further implementation, the processor is further programmed to control the augmented reality visualization device to virtually show the desired entry point on the virtual object.

[0014] In a further implementation, the virtual object includes a virtual crosshair.

[0015] In a further implementation, the processor is further programmed to control an augmented reality visualization device to virtually show the desired trajectory with a direction indicator or angle difference indicator.

[0016] In a further implementation, the direction indicator or angle difference indicator is overlaid on a trajectory marker connected to the surgical positioning object.

[0017] In a further implementation, the processor is further programmed to control the augmented reality visualization device to perform a registration process to register the virtual bone model to the patient's anatomical structure within the augmented reality environment.

[0018] In a further implementation, the processor is further configured to control the augmented reality visualization device to present a plurality of registration reference points within the augmented reality environment, each of the plurality of registration reference points visually indicating where a user should physically contact the patient's anatomy to initialize an approximation of the virtual bone model to the patient's anatomy.

[0019] In a further implementation, the processor is further programmed to control the augmented reality visualization device to visually indicate the accuracy between the actual trajectory and the desired trajectory of the surgical positioning object within the augmented reality environment.

[0020] In a further implementation, the processor is further programmed to control the augmented reality visualization device to visually indicate the accuracy by changing the color of a virtual indicator presented within the augmented reality environment.

[0021] In a further implementation, the surgical positioning object is a surgical guide pin.

[0022] A surgical method according to an exemplary aspect of the present disclosure may include intraoperatively defining a desired entry point and a desired trajectory of a surgical positioning object within an augmented reality environment generated by an augmented reality system, wherein a visual indication of accuracy between the actual trajectory of the surgical positioning object within the augmented reality environment and the desired trajectory may be provided.

[0023] In a further implementation, intraoperatively defining the desired entry point includes presenting a virtual crosshair within the augmented reality environment.

[0024] In a further implementation, defining the desired trajectory intraoperatively includes presenting a virtual trajectory in an augmented reality environment.

[0025] In a further implementation, providing a visual indication of accuracy includes presenting a directional indicator within the augmented reality environment.

[0026] In a further embodiment, providing a visual indication of accuracy includes presenting an angle difference indicator within the augmented reality environment.

[0027] In a further embodiment, the visual indication of accuracy is virtually overlaid on a trajectory marker connected to the surgical positioning object.

[0028] In a further implementation, the trajectory marker is configured to digitize the actual trajectory of the surgical positioning object.

[0029] In a further implementation, the method includes registering the virtual bone model to the patient's anatomy in the augmented reality environment prior to intraoperatively defining the desired entry point and the desired trajectory.

[0030] In a further implementation, the surgical positioning object is a surgical guide pin.

[0031] A surgical method according to another exemplary aspect of the present disclosure may include positioning a tip of a surgical guide pin at a location on a bone surface of an anatomical structure indicated by virtual crosshairs, preparing an indentation at the location, pivoting the surgical guide pin about a pivot point established by the indentation, and aligning an actual trajectory of the surgical guide pin with a virtual trajectory of the surgical guide pin within an augmented reality environment.

[0032] 1 illustrates an exemplary orthopaedic surgical system 10 (hereinafter "system 10") according to an exemplary embodiment of the present disclosure. System 10 may be used to create, edit, review, and / or execute surgical plans, such as, for example, surgical plans for performing arthroplasty to repair a joint. The teachings of the present disclosure are not intended to be limited to any particular joint of the human musculoskeletal system and, therefore, are applicable to the shoulder, knee, hip, ankle, wrist, etc.

[0033] System 10 may include, among other subsystems, a surgical planning system 12, an augmented reality (AR) system 14, a storage system 16, and a network 18. System 10 may include a greater or lesser number of subsystems within the scope of this disclosure. As discussed in more detail below, surgical planning system 12 may be configured to allow one or more users to pre-operatively create a surgical plan, and AR system 14 may be configured to allow one or more users to intra-operatively review, edit, update, verify, and / or execute the pre-operative surgical plan. In one embodiment, AR system 14 is used to transfer certain aspects of the pre-operative surgical plan to the surgical site, including, but not limited to, aspects such as desired implant size and positioning, and desired guide pin placement to guide the reaming procedure necessary to prepare the native anatomy to receive the desired implant size and positioning.

[0034] In this disclosure, the term "augmented reality" is intended to refer to the ability to provide an interactive experience in which real objects (e.g., a patient's anatomy) present in a given environment may be augmented with computer-generated sensory information across one or more sensory modalities (e.g., vision, hearing, touch, etc.). Additionally, the term "augmented reality" is intended to encompass aspects such as mixed reality, virtual reality, augmented reality, holographic projection, etc.

[0035] The surgical planning system 12 may be configured to pre-operatively plan a surgical procedure. The pre-operative planning provided by the surgical planning system 12 may include features such as, but not limited to, constructing a virtual model of the patient's anatomy, identifying landmarks within the virtual model, selecting and orienting a virtual implant within the virtual model, and identifying optimal insertion points and trajectories to guide the surgical reaming procedure within the virtual model. An exemplary surgical planning system suitable for use as the surgical planning system 12 of the system 10 is the Virtual Implant Positioning™ (VIP) system available from Arthrex, Inc.

[0036] The surgical planning system 12 may include a computing device 20 including a processor 22 operably coupled to a memory 24. The computing device 20 may be a single computer or multiple computers configured to process software instructions serially or in parallel. The computing device 20 may be configured to communicate with the AR system 14 and / or other computing devices via a network 18.

[0037] The processor 22 may be a custom-made or commercially available processor, a central processing unit (CPU), or generally any device for executing software instructions. The memory 24 may include any one or combination of volatile and / or non-volatile memory elements. The processor 22 may be operatively coupled to the memory 24 and configured to execute one or more programs stored in the memory 24 based on various inputs received from other devices or data sources.

[0038] In one embodiment, the processor 22 of the computing device 20 may be operable to access, locally and / or remotely execute, a planning environment 26 for creating, editing, executing, and / or reviewing one or more surgical plans 34 during pre-operative, intra-operative, and / or post-operative phases of a surgical procedure. The planning environment 26 may be a standalone software package or may be incorporated into another surgical tool. The planning environment 26 may, for example, provide a display or visualization of one or more bone models 30 and associated images, and one or more implant models 32 and associated images, via one or more graphical user interfaces (GUIs). Each bone model 30, implant model 32, and associated images and other information may be stored in one or more files or records according to a specified data structure.

[0039] The planning environment 26 may include various modules for performing desired planning functions. In one embodiment, the planning environment 26 includes a data module for accessing, acquiring, and / or storing data related to the surgical plan 34, a display module for representing the data (e.g., in a GUI), a spatial module for modifying the data represented by the display module, and a comparison module for determining one or more relationships, for example, between a selected bone model and a selected implant model. However, a greater or fewer number of modules may be utilized, and / or one or more of the modules may be combined to provide the disclosed functionality.

[0040] Storage system 16 may be configured to store or otherwise provide data from / to surgical planning system 12 and AR system 14. Storage system 16 may be, for example, a storage area network device (SAN) configured to communicate with surgical planning system 12 and AR system 14 via network 18. Although shown as a separate device, storage system 16 may be incorporated within or directly coupled to computing device 20 of surgical planning system 12. Storage system 16 may be configured to store one or more of computer software instructions, data, database files, configuration information, etc.

[0041] In one embodiment, surgical planning system 12 includes a client-server architecture configured to execute computer software on computing device 20, which is accessible using either a thin client application or a web browser executed on computing device 20. Computing device 20 may load computer software instructions from local storage or storage system 16 into memory 24 and execute the computer software using processor 22.

[0042] The system 10 may further include one or more databases 28. The databases 28 may be stored in a central location, such as the storage system 16. Each database 28 may be a relational database configured to associate one or more bone models 30, one or more implant models 32, and one or more transmission models 33 with each other and / or with surgical plans 34. Each surgical plan 34 may be associated with a respective patient. Each bone model 30, implant model 32, transmission model 33, and surgical plan 34 may be assigned a unique identifier or database entry or record within the database 28. The database 28 may be configured to store data corresponding to the bone models 30, implant models 32, transmission models 33, and surgical plans 34 in one or more database records or entries, and / or may be configured to link or otherwise associate one or more files corresponding to each respective bone model 30, implant model 32, transmission model 33, and surgical plan 34. The bone models 30 stored in the database 28 may correspond to the anatomy of each patient from previous surgical cases and may be classified into one or more predetermined categories, such as gender, age, race, defect category, type of procedure, etc.

[0043] Each bone model 30 may include information obtained from one or more medical devices or tools, such as a computed tomography (CT), magnetic resonance imaging (MRI) machine, and / or an X-ray machine that obtains one or more images of a patient. The bone model 30 may include one or more digital images and / or coordinate information related to the patient's anatomy that may be obtained or derived from a medical device.

[0044] Each implant model 32 may include coordinate information associated with a given implant design. The planning environment 26 may incorporate and / or interface with one or more modeling packages, such as computer-aided design (CAD) packages, to render the models 30, 32 as two-dimensional (2D) and / or three-dimensional (3D) volumes or constructs. A given design may correspond to one or more components. The implant models 32 may correspond to implants and components of various shapes and sizes. Each implant may include one or more components that may be placed at a surgical site, including screws, anchors, grafts, etc. Each implant model 32 may correspond to a single component or may include two or more components that may be configured to establish an assembly. Each bone model 30, implant model 32, and transfer model 33 may correspond to 2D and / or 3D geometric shapes and may be utilized to generate wireframe, mesh, and / or solid structures in a graphical display.

[0045] Each transfer model 33 may correspond to various instruments and devices used to implement each surgical plan 34, including preparing the surgical site and securing one or more implants to bone or other tissue to restore function to the respective joint. Each transfer model 33 may be associated with a respective surgical instrument or device (e.g., guide pin, transfer guide, etc.) and / or a respective implant model 32. Each transfer model 33 may include coordinate information associated with a given instrument design.

[0046] The surgical plan 34 may be associated with one or more surgical positioning objects, such as guide pins (e.g., guide wires or Kirschner wires) sized to be anchored within tissue to position and orient various instruments, devices, and / or implants. In some implementations, the surgical positioning objects are used to guide reaming procedures necessary to prepare the patient's anatomy to receive a desired implant. The exemplary transmission model 33 may be configured to preoperatively establish virtual entry or insertion positions and virtual trajectory axes of one or more surgical positioning objects relative to one or more bone models 30. The virtual positions may be associated with specified insertion points and trajectories of the surgical positioning objects relative to the patient's anatomy (as represented by the bone models 30). The virtual trajectory axes may extend through the virtual positions and may be associated with specified orientations of the surgical positioning objects relative to the patient's anatomy for any given surgical plan 34.

[0047] The planning environment 26 may be configured to set virtual positions and / or virtual axes of the surgical positioning objects in response to the placement of each implant model 32 relative to the bone model 30 and associated patient anatomy. The virtual positions and / or virtual axes may be automatically set and / or adjusted based on the position and orientation of the selected implant model 32 relative to the selected bone model 30 and / or in response to user interaction with the planning environment 26.

[0048] Each surgical plan 34 may be associated with one or more of the bone model 30, the implant model 32, and the transfer model 33. The surgical plan 34 may include one or more modifications to the bone model 30, information related to the desired position of the implant model 32 relative to the original and / or modified bone model 30, and information related to the desired position of the surgical positioning object of the transfer model 33 relative to the original and / or modified bone model 30 (e.g., desired entry point and trajectory). The surgical plan 34 may include coordinate information related to the modified bone model and the relative position of the implant model 32 and / or the transfer model 33 in a predetermined data structure. Modifications to each bone model 30 and the surgical plan 34 may be stored in the database 28 automatically and / or in response to user interaction with the surgical planning system 12.

[0049] One or more surgeons and other users may be presented with the planning environment 26 via the computing device 20 or another computer operably linked to the surgical planning system 12. The users may simultaneously access each bone model 30, implant model 32, transfer model 33, and surgical plan 34 stored in the database 28. Each user may interact with the planning environment 26 to create, view, and / or modify various aspects of the surgical plan 34. The computing device 20 may be configured to store local instances of the bone models 30, implant models 32, transfer models 33, and / or surgical plan 34, which may be synchronized with the database 28 in real time or periodically.

[0050] The AR system 14 may enable one or more users to intraoperatively review, edit, update, verify, and / or execute a given patient's surgical plan 34. In one embodiment, the AR system 14 provides one or more users with an interactive surgical experience in which real objects present within a medical facility (e.g., a hospital, a surgical center, etc.), and more specifically, within an operating room 36 of the medical facility, are augmented with computer-generated sensory information across one or more sensory modalities. Thus, when using the AR system 14 before or during a surgical procedure, a user may perceive and interact with images that include both real and virtual objects.

[0051] The AR system 14 may include, among other things, a visualization device 38, a processor 40, and a memory 42 operably coupled to the processor 40. The visualization device 38 may be configured to communicate with the surgical planning system 12, the storage system 16, and / or other AR visualization devices over the network 18. The processor 40 and the memory 42 may be provided on or within the visualization device 38, such as in a computing device operably connected to the visualization device 38, or may be separate from the visualization device 38.

[0052] In one embodiment, visualization device 38 is a head-mounted or head-up display that can be worn on a user's head. However, other types of visualization devices are also contemplated within the scope of this disclosure. An exemplary visualization device suitable for use in AR system 14 of system 10 is the Microsoft HOLOLENS™ headset available from Microsoft Corporation. In other embodiments, AR system 14 can include multiple different visualization devices that can be used together to provide an interactive surgical experience.

[0053] The processor 40 of the AR system 14 may be a custom-made or commercially available processor, central processing unit (CPU), or generally any device for executing software instructions. The memory 42 may include any one or combination of volatile and / or non-volatile memory elements. The processor 40 may be operatively coupled to the memory 42 and may be programmed to execute one or more programs stored in the memory 42 based on various inputs received from other devices or sources. For example, the processor 40 may be programmed to execute various software instructions stored on the memory 42 to provide an interactive surgical experience. In one embodiment, as discussed in more detail below, the processor 40 is programmed to control the visualization device 38 to present one or more AR environments to the user. The AR environments may include various user interfaces, menus, virtual objects, etc. for transferring aspects of the pre-operative surgical plan 34 to the intra-operative anatomy.

[0054] The processor 40 may be further programmed to selectively access the associated bone models 30, implant models 32, transfer models 33, and surgical plans 34 for a particular patient from the database 28 of the storage system 16. In some embodiments, certain aspects associated with the bone models 30, implant models 32, and / or surgical plans 34 for a particular patient may be stored directly in the memory 42, which may be synchronized with the database 28 in real time or periodically, and executed by the processor 40. In other embodiments, certain aspects and functions associated with the surgical planning system 12 may be stored on the memory 42 and executed by the processor 40.

[0055] The visualization device 38 may additionally include a sensor system 44 including multiple sensors (e.g., image sensors, optical sensors, depth sensors, motion sensors, etc.). The sensor system 44 may be configured to collect data that may be processed by the processor 40 to present, position, move, and / or adjust virtual objects in the AR environment relative to the actual environment of the operating room 36. In one embodiment, the sensor system 44 may detect hand gestures, audible commands, etc., that may be processed by the processor 40 to interact with the virtual objects projected by the visualization device 38.

[0056] In one embodiment, sensor system 44 includes both a time-of-flight camera 45 and a visible light stereo camera 47. Time-of-flight camera 45 and visible light stereo camera 47 may each be operably coupled to visualization device 38. Image data from time-of-flight camera 45 and visible light stereo camera 47 may be processed by processor 40 to perform an inside-out registration process that does not require the use of fiducial markers placed on the patient's anatomy. The inside-out registration process may include performing a 3D reconstruction of the captured images and then registering the pre-operative image data to the patient's anatomy.

[0057] In one embodiment, the processor 40 of the AR system 14 can be programmed to implement a stereo transformer model to provide stereo reconstruction of captured images. The stereo transformer model can utilize both self-attention within a single image and cross-attention between multiple images to identify pixel correspondences between captured images. The attention mechanism of the stereo transformer model can enable the model to reduce ambiguity in feature correspondences on textureless surfaces by attaching to identifiable features that are close to points of interest. Thus, the stereo transformer model can generate reasonably dense reconstructions even on relatively textureless bone surfaces.

[0058] 2, visualization device 38 of AR system 14 is configured to provide an AR environment 48 that can be worn by a user 46 (e.g., a surgeon or other medical personnel) and overlaid on real objects 52 (e.g., a patient's anatomy, an operating table, etc.) located within operating room 36. Thus, visualization device 38 allows user 46 to intraoperatively visualize both virtual and real objects within operating room 36.

[0059] The AR environment 48 may be projected as a holographic image onto the screen 50 of the visualization device 38 and may present, among other things, one or more user interfaces including virtual details associated with one or more surgical plans 34 that were pre-operatively planned using the surgical planning system 12. Once projected, the AR environment 48 is visually perceptible by the user 46 in the foreground of the operating room 36, with real objects 52 appearing in the foreground of the projected virtual image. As discussed further below, the user 46 may visualize and interact with the AR environment 48 in order to execute the surgical plan 34 for a given patient during a live surgical procedure.

[0060] The AR environment 48 generated by the AR system 14 can be utilized to transfer certain aspects of the pre-operative surgical plan 34 to the intra-operative anatomy during an orthopaedic surgical procedure. For example, as described in further detail below, a user can utilize the AR environment 48 to perform a targeting process for positioning a surgical positioning object (e.g., a guide pin 80) within a respective patient's anatomy 56. In some implementations, this targeting process can include indicating a desired entry point P and a desired trajectory T of the guide pin 80 within the AR environment 48. FIG. 3 visually illustrates such a targeting process for the guide pin 80. Breaking the targeting process down into separate steps allows metrics of entry point location accuracy (in millimeters) and trajectory rotation accuracy (in degrees) to be separated from one another and used to evaluate drilling performance during an orthopaedic surgical procedure.

[0061] The first step of positioning guide pin 80 at a desired entry point P in anatomical structure 56 may be referred to as establishing a desired translational aspect of guide pin 80. The translation distance d may refer to the distance (e.g., in mm) between the desired entry point P of guide pin 80 and the actual location P2 of the tip of guide pin 80.

[0062] The second step of establishing the desired trajectory T of the guide pin 80 may be referred to as establishing the rotational trajectory aspect of the guide pin 80. The rotation may refer to the amount of rotation Θ (e.g., in degrees) between the desired trajectory T and the actual trajectory T2 of the guide pin 80.

[0063] 4, with continued reference to FIGS. 1-3, generally illustrates an exemplary surgical method 100 that may be performed using the system 10 for planning and executing orthopaedic surgical procedures. The surgical method 100 may be utilized pre-operatively and intra-operatively to create, edit, and execute respective surgical plans 34. In one embodiment, the surgical method 100 is utilized to perform arthroplasty to restore function to a shoulder joint or any other joint.

[0064] Although surgical method 100 is described herein with respect to repairing glenoid defects during shoulder arthroplasty, it should be understood that surgical method 100 may be utilized elsewhere on a patient and for other orthopedic surgical procedures. Accordingly, fewer or additional steps than those listed below may be performed within the scope of this disclosure. Furthermore, the order of the listed steps is not intended to limit this disclosure.

[0065] Orthopaedic surgical procedures may be pre-operatively planned using the surgical planning system 12 of the system 10 at block 102. Pre-operative planning may include acquiring images of the patient's anatomy, constructing a virtual model of the patient's anatomy, identifying landmarks within the virtual model, selecting and orienting virtual implants within the virtual model, identifying and marking desired entry points P and desired trajectories T of guide pins 80 to guide the reaming procedure for reaming into the anatomy 56, etc., resulting in the creation of a surgical plan 34 for each patient.

[0066] A user may use the visualization device 38 of the AR system 14 to intraoperatively generate an AR environment 48 to augment real-world objects present in the operating room 36 of the medical facility with computer-generated sensory information across one or more sensory modalities in block 104. An exemplary AR environment 48 is illustrated in FIG. 5 . The AR environment 48 may include various user interface modules or menus designed to present a multitude of information to the user during a given surgical procedure. While a particular arrangement of user interface modules is shown in the figures of this disclosure, other arrangements are also contemplated within the scope of this disclosure. Accordingly, the particular size, positioning, and overall arrangement of the AR modules shown herein are not intended to limit this disclosure.

[0067] During step 104 of method 100, the user may register the virtual bone model 54 to the anatomical structure 56. In one embodiment, the AR environment 48 includes a registration module that allows the user to overlay the virtual bone model 54 (e.g., derived from the surgical plan 34) on the patient's actual anatomical structure 56. The registration module may be configured to allow the user to interact with the AR environment 48 before and / or during the surgical procedure. The user may interact with selectable buttons, menus, widgets, etc., using, for example, hand gestures or audible commands. For example, the user may use their hand 58 to interact with a cursor 59 in the AR environment 48 to position the virtual bone model 54 at a desired location relative to the anatomical structure 56.

[0068] Once the desired position is achieved, the user may perform a hand gesture 60 to cause a registration menu 62 to be presented within the AR environment 48 (see FIG. 6). The user may confirm the completion of the overlay by pressing a confirmation button 64 on the registration menu 62 or by using an audible command.

[0069] In other implementations, the registration module of the AR environment 48 may be configured to allow a user (or a group of users with operatively connected visualization devices) to register the virtual bone model 54 to the anatomical structure 56 using a multi-stage registration process. An initialization step of the multi-stage registration process may be performed first to approximate the position and orientation of the virtual bone model 54 to the anatomical structure 56. During the initialization step, the system 10 may present multiple registration reference points 85 on the virtual bone model 54 (see FIG. 7 ). Although three registration reference points 85-1 (e.g., a superior reference point), 85-2 (e.g., an inferior reference point), and 85-3 (e.g., an anterior / posterior reference point) are shown, more or fewer registration reference points 85 may be provided for the virtual bone model 54 in the AR environment 48. Furthermore, one skilled in the art will understand that the registration reference points 85 may be presented anywhere on the virtual bone model 54.

[0070] The registration reference points 85-1, 85-2, and 85-3 may visually indicate where the user should physically touch the anatomical structure 56 to initialize the approximation of the virtual bone model 54 to the anatomical structure 56. The registration reference points 85-1, 85-2, and 85-3 essentially define a reference plane in the physical space defined by the patient anatomy for orienting and positioning the virtual bone model 54 relative to the anatomical structure 56. As shown schematically, the user may physically touch the anatomical structure 56 with multiple contact points 87-1, 87-2, and 87-3 (see FIG. 7 ) that correspond as closely as possible to the locations indicated by the registration reference points 85-1, 85-2, and 85-3. The contact points 87-1, 87-2, and 87-3 may be contacted using fingers 99 of the user's hand 58, a pointer tool, or some other suitable object. Next, the system 10 may complete an initialization step by roughly aligning the silhouette of the virtual bone model 54 to the corresponding features of the anatomical structure 56 (see FIG. 8).

[0071] A 3D scanning step of the multi-step registration process may then be performed to capture, segment, and classify an anatomical volume using a 3D point cloud mesh of the anatomical structure 56 associated with the patient site. During the initialization step, the system 10 may analyze the anatomical structure 56 from various angles to three-dimensionally reconstruct the anatomical structure 56 and improve the positional accuracy of the virtual bone model 54 relative to the anatomical structure 56. The system 10 may present a progress indicator 89 within the AR environment 48 as the 3D scan is performed (see FIG. 9 ). The progress indicator 89 may be overlaid on the anatomical structure 56 and configured to visually indicate the progress of the 3D scanning operation to the user. Upon completion of the 3D scanning step, the system 10 may present a graphic 91 (e.g., a check mark) within the AR environment 48 adjacent to the progress indicator 89 to visually indicate to the user that the 3D scan is complete (see FIG. 10 ).

[0072] The initialization and 3D scanning operations can produce a registration of the virtual bone model 54 to the anatomical structure 56. After the initialization, 3D scanning, and registration are complete, the multi-stage registration process of method 100 can continuously analyze the camera feed from the visualization device 38 to maintain accurate blending (spatial anchoring) of the virtual structures based at least on the movement and pose of the visualization device 38.

[0073] After registering the virtual bone model 54 to the anatomy 56, the user may utilize the AR environment 48 to transfer certain aspects of the pre-operative surgical plan 34 to the intra-operative anatomy 56. For example, as described in further detail below, the user may utilize the AR environment 48 to perform a targeting process to precisely position a surgical positioning object (e.g., a guide pin 80) within the anatomy 56.

[0074] 4 , the user may next initiate a guide pin placement module of the AR environment 48 at block 106 of the method 100. The guide pin placement module may be used to precisely set the entry location and trajectory of a guide pin 80 that is placed within the patient's anatomy 56 (e.g., the glenoid) to guide the placement of an implant (e.g., a glenoid baseplate). The guide pin placement module of the AR environment 48 may be initialized by activating a guide pin placement button 66, which may be presented within another menu 68 of the AR environment 48 (see, e.g., FIG. 11 ).

[0075] A virtual crosshair 70 (or some other visual indicator) may be displayed on the anatomical structure 56 within the AR environment 48 at block 108 (see FIG. 12 ). The location of the virtual crosshair 70 may be derived from information stored in the pre-operative surgical plan 34. The virtual crosshair 70 may identify the desired entry point P of the guide pin 80, thus providing visualization of the optimal insertion location for the guide pin 80.

[0076] Next, at block 110, the user may use the drill tip of the guide pin 80 to prepare a shallow depression 74 (e.g., a dent) in a bone surface 76 (e.g., a glenoid surface) of the anatomical structure 56 at the location of the virtual crosshairs 70 (see FIG. 13 ). The shallow depression 74 may act as a pivot point for adjusting the rotational position, and therefore the trajectory, of the guide pin 80 relative to the bone surface 76. The guide pin 80 may be held by a powered surgical instrument, such as a surgical drill 78 (see FIG. 13 ). The surgical drill 78 may be configured to rotate the guide pin 80 to drill the shallow depression 74 in the bone surface 76.

[0077] In block 112, the user may align the trajectory of the guide pin 80 with a virtual trajectory 82, which may be presented within the AR environment 48. The virtual trajectory 82 may be represented by a dot or an axis, which is presented within the AR environment 48 and is intended to provide a visual indication of the desired trajectory T of the guide pin 80.

[0078] The alignment step of block 112 of method 100 may be accomplished by using a trajectory marker 84, which is visually shown in Figure 14 and may be connected to the guide pin 80 to track the central axis of the guide pin 80. The trajectory marker 84 may be utilized, for example, to digitize the trajectory of the central axis of the guide pin 80.

[0079] In one embodiment, the trajectory marker 84 includes a self-calibrating slide-on-disc design that includes an inner diameter that is slightly larger than the outer diameter of the guide pin 80. The trajectory marker 84 may be slid onto a proximal portion 86 of the guide pin 80 that protrudes proximally from a rear face 88 of the surgical drill 78. In one embodiment, the guide pin 80 extends completely through the housing 79 of the surgical drill 78.

[0080] The user may align the trajectory of the guide pin 80 with the virtual trajectory 82 by pivoting the guide pin 80 about the pivot point established by the shallow indentation 74. As the guide pin 80 moves about the pivot point, various visual indicators may be presented to the user within the AR environment 48 to provide targeting guidance for aligning the trajectory of the guide pin 80 with the virtual trajectory 82. For example, a direction indicator 90 may be provided to visually indicate the direction in which the user needs to pivot the guide pin 80 to move near the desired trajectory indicated by the virtual trajectory 82. An angle difference indicator 92 may further be provided as a visual reference within the AR environment 48 to indicate the angle difference between the actual trajectory of the guide pin 80 and the virtual trajectory 82.

[0081] In some implementations, an indicator ring 94 overlaying the trajectory marker 84 may be presented within the AR environment 48 (see, e.g., FIG. 14 ). The indicator ring 94 may be configured to automatically change from a first color (e.g., red or amber) to a second color (e.g., green) when the trajectory of the guide pin 80 moves from a poorly aligned position (e.g., an error of more than about 3 degrees) to a properly aligned position (e.g., an error of less than about 3 degrees) with respect to the virtual trajectory 82.

[0082] After achieving the desired trajectory through the assistance provided by the trajectory markers 84, the guide pin 80 may be drilled into the anatomy 56 at block 114. A subsequent reaming process may then be performed at block 116 to prepare the anatomy 56 for receiving the desired arthroplasty implant (e.g., a glenoid baseplate).

[0083] The exemplary surgical method 100 described above contemplates the use of guide pins to prepare a bone or joint to receive an implant during arthroplasty, however, the present disclosure is not limited to positioning guide pins and can extend to the positioning of any surgical positioning object that can be used to transfer aspects of the preoperative surgical plan to the intraoperative anatomy.

[0084] The exemplary surgical systems and methods of the present disclosure advantageously provide improved intraoperative guidance for transferring preoperative surgical plans to intraoperative anatomy during orthopaedic surgical procedures. The intraoperative guidance may be presented within one or more augmented reality environments that may be overlaid on the in situ anatomy without obstructing the user's view of the natural anatomy. Thus, improved guidance during transfer provides improved surgical outcomes.

[0085] Although different non-limiting embodiments are illustrated as having particular components or steps, embodiments of the present disclosure are not limited to those particular combinations. Some of the components or features from any of the non-limiting embodiments can be used in combination with features or components from any of the other non-limiting embodiments.

[0086] It should be understood that like reference numerals identify corresponding or similar elements throughout the several views. While particular component arrangements are disclosed and illustrated in these exemplary embodiments, it should be further understood that other arrangements can also benefit from the teachings of the present disclosure.

[0087] The foregoing description is illustrative and is not to be construed in any limiting sense. Those skilled in the art will recognize that certain modifications may fall within the scope of the present disclosure. For these reasons, the following claims should be studied to determine the true scope and content of the present disclosure. [Explanation of symbols]

[0088] 10 Orthopedic Surgery Systems 12 Surgical Planning System 14 Augmented Reality (AR) Systems 16 Memory Systems 18 Network 20 Computing Devices 22 processors 24 memory 26 Planning Environment 28 databases 30 bone models 32 Implant Model 33 Transmission Model 34 Preoperative Surgical Planning 36 Operating room 38 Visualization Devices 40 processors 42 memory 44 Sensor System 45 Time-of-Flight Camera 46 users 47 Visible light stereo camera 48 AR environment 50 screens 52 objects 54 Virtual Bone Model 56 Anatomical Structure 58 moves 59 Cursor 62 Registration Menu 64 Confirm button 66 Guide pin placement button 68 Menu 70 Virtual Crosshair 76 Bone surface 78 Surgical Drills 79 Housing 80 guide pin 82 Virtual Orbit 84 Orbital Marker 85 Registration Reference Point 85-1 Registration reference point 85-2 Registration Reference Point 85-3 Registration reference point 86 Proximal part 87-1 Contact point 87-2 Contact point 88 Rear 89 Progress Indicator 90 Directional Indicator 91 Graphics 92 Angle Difference Indicator 94 Indicator Ring 99 fingers 100 Surgical Methods

Claims

1. 1. An augmented reality system for a surgical system, comprising: an augmented reality visualization device; Controlling the augmented reality visualization device providing an augmented reality environment for a patient's anatomy; enabling a user to interface with the augmented reality environment to intraoperatively achieve a desired entry point and a desired trajectory of a surgical positioning object relative to the patient's anatomy; controlling the augmented reality visualization device to virtually show the desired trajectory with a direction indicator or an angle difference indicator; a processor programmed to: Equipped with The direction indicator or the angular difference indicator is overlaid on a trajectory marker connected to the surgical positioning object. Augmented reality system.

2. The augmented reality system of claim 1 , wherein the processor is further programmed to control the augmented reality visualization device to virtually show the desired entry point with a virtual object.

3. The augmented reality system of claim 2 , wherein the virtual object comprises a virtual crosshair.

4. 10. The augmented reality system of claim 1, wherein the processor is further programmed to control the augmented reality visualization device to perform a registration process for registering a virtual bone model to the patient's anatomical structure within the augmented reality environment.

5. 5. The augmented reality system of claim 4, wherein the processor is further configured to control the augmented reality visualization device to present a plurality of registration reference points within the augmented reality environment, each of the plurality of registration reference points visually indicating a location where the user should physically touch the patient's anatomy to initialize an approximation of the virtual bone model to the patient's anatomy.

6. 10. The augmented reality system of claim 1, wherein the processor is further programmed to control the augmented reality visualization device to visually indicate accuracy between an actual trajectory of the surgical positioning object and the desired trajectory within the augmented reality environment.

7. 7. The augmented reality system of claim 6, wherein the processor is further programmed to control the augmented reality visualization device to visually indicate the accuracy by changing a color of a virtual indicator presented within the augmented reality environment.

8. The augmented reality system of claim 1 , wherein the surgical positioning object is a surgical guide pin.

9. 1. A computer-implemented method for a surgical procedure, comprising: providing an augmented reality environment for the patient's anatomy; a user interfacing an augmented reality visualization device with an augmented reality environment to intraoperatively achieve a desired entry point and a desired trajectory of a surgical positioning object relative to the patient's anatomy; Intraoperatively defining a desired entry point and a desired trajectory of a surgical positioning object within an augmented reality environment generated by an augmented reality system; controlling the augmented reality visualization device to virtually show the desired trajectory with a direction indicator or an angle difference indicator. Including, The direction indicator or the angular difference indicator is overlaid on a trajectory marker connected to the surgical positioning object. method.

10. The method of claim 9 , wherein intra-operatively defining the desired entry point comprises presenting a virtual crosshair within the augmented reality environment.

11. The method of claim 9 , wherein intra-operatively defining the desired trajectory comprises presenting a virtual trajectory within the augmented reality environment.

12. The method of claim 9 , wherein virtually indicating the desired trajectory comprises presenting a directional indicator within the augmented reality environment.

13. The method of claim 9 , wherein virtually indicating the desired trajectory comprises presenting an angular difference indicator within the augmented reality environment.

14. The method of claim 9 , wherein the trajectory marker is configured to digitize an actual trajectory of the surgical positioning object.

15. The method of claim 9 , comprising registering a virtual bone model to the patient's anatomy in the augmented reality environment prior to intraoperatively defining the desired entry point and the desired trajectory.

16. The method of claim 9 , wherein the surgical positioning object is a surgical guide pin.

Images