Surgical navigation system

Abstract

The present disclosure relates to a surgical navigation system for aligning a surgical instrument and methods of using the same, wherein the surgical navigation system can include a head-mounted display including a lens. The surgical navigation system can further include a tracking unit, wherein the tracking unit can be configured to track a patient tracker and / or a surgical instrument. Patient data can be registered to the patient tracker. The surgical instrument can define an instrument axis. The surgical navigation system can be configured to plan one or more trajectories based on the patient data. The head-mounted display can be configured to display an augmented reality visualization on the lens of the head-mounted display, including an augmented reality position alignment visualization and / or an augmented reality angle alignment visualization related to the surgical instrument.

Description

[0001] This application is a divisional application of the invention patent application filed on May 4, 2018, with application number "201880029762.0" and invention title "Surgical Navigation System".

[0002] Related applications

[0003] This application claims priority and benefit to European Patent Application No. EP17169700.6, filed on 5 May 2017, the entire contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure generally relates to surgical navigation systems for supporting surgical interventions. More specifically, but not exclusively, this disclosure generally relates to a holographic surgical navigation system. Background Technology

[0005] In the medical field, computer-aided surgery, including surgical navigation systems, is on the rise. During surgical interventions, surgical navigation systems can be used in conjunction with preoperative images and / or patient data to support surgeons in performing medical procedures. For this purpose, image-guided surgical navigation systems are used for open and minimally invasive surgical interventions, such as spinal, joint, and / or neurosurgical procedures. The purpose of such surgical navigation systems is to determine the position of surgical instruments used by the surgeon, which can be shown or visualized in preoperative images and / or patient data corresponding to the patient's anatomy. Continuous monitoring of the position and / or orientation (i.e., posture) of the patient and / or surgical instruments (so-called navigation data) is essential to providing an accurate spatial representation of the surgical instruments relative to the patient.

[0006] Surgical navigation systems can also visualize the location and / or alignment of medical devices or implants for surgeons, such as the location and alignment of pedicle screws used for multi-vertebral fixation in spinal surgery scenarios. During the procedure, the surgical navigation system can provide surgeons with visualizations that allow them to see an overlay of the precise locations of surgical instruments and / or medical devices / implants projected onto an image or visualization of the patient.

[0007] Given the existing solutions, it's important to note that known surgical navigation solutions do not adequately address the guidance of surgical navigation systems and the alignment of surgical instruments. Another drawback of existing solutions is the separation of visualization from the surgical site, which forces surgeons to lose focus.

[0008] Therefore, there is a need for novel surgical navigation systems that address these shortcomings. Summary of the Invention

[0009] In exemplary configurations of surgical navigation systems and methods for aligning surgical instruments, the surgical navigation system can be configured to provide augmented reality visualizations during medical interventions. Generally, the surgical navigation system may include a tracking unit. The tracking unit may include one or more position sensors. The surgical navigation system may further include a head-mounted display, a patient tracker, and surgical instruments, each of which may include one or more tracking elements or markers configured to be detected by the one or more position sensors of the tracking unit. The tracking unit may be configured to continuously track the position and / or orientation (pose) of the head-mounted display, patient tracker, and surgical instruments in a local or common coordinate system. The surgical navigation system may be further configured to register the position and / or orientation of the head-mounted display, patient tracker, and surgical instruments with patient data to generate and display augmented reality visualizations on a lens or screen of the head-mounted display. The augmented reality visualization may include surgical instrument and / or target trajectory axes superimposed on patient data, wherein the patient data may include preoperative images of the patient's anatomy and / or the target object. The target object may include a planned surgical path, a planned location for an implant or medical device, and / or the patient's anatomical features.

[0010] Augmented reality visualizations can include positional alignment visualizations and / or angular alignment visualizations. Navigation systems can be configured to display positional alignment visualizations and / or angular alignment visualizations on a head-mounted display, allowing visualizations to be overlaid on a real patient from a surgeon's perspective to create 3D or holographic visualizations.

[0011] In another example configuration of the augmented reality system, the augmented reality system may include a surgical navigation system, a patient tracker that can be tracked by the surgical navigation system, and surgical instruments that can be tracked by the surgical navigation system. The augmented reality system may further include a head-mounted display comprising lenses and a controller configured to visualize augmented reality position alignment on the lenses as two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of the distance vector from a target point on the target trajectory axis to the surgical instruments.

[0012] In another example configuration of the augmented reality system, the augmented reality system may include a head-mounted display containing lenses. The augmented reality system may also include a surgical navigation system including a tracking unit configured to track multiple objects positioned within a defined area or coordinate system (e.g., the surgical area). The augmented reality system may also include a patient tracker registered to patient data and trackable by the surgical navigation system, and surgical instruments having instrument trackers trackable by the surgical navigation system. The instrument trackers may be configured to define the instrument axes of the surgical instruments. The augmented reality system may also include a controller configured to generate augmented reality positional alignment visualization as two axis-aligned deviation vectors displayed on the lenses of the head-mounted display, the two axis-aligned deviation vectors comprising a decomposition of distance vectors from a target point on the target trajectory axis to a point on the surgical instrument.

[0013] In another example configuration of the augmented reality system, the surgical navigation system may include a patient tracker and a surgical instrument tracker. The surgical navigation system may be configured to plan a target trajectory axis based on patient data and to align the tip of the surgical instrument with the target trajectory axis. The surgical navigation system may further include a head-mounted display including a head-mounted display tracker, wherein the head-mounted display is configured to display an augmented reality position alignment visualization including two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of distance vectors from a point on the target trajectory axis to the tip of the surgical instrument. The head-mounted display may be further configured to display an augmented reality angle alignment visualization including a deviation angle, the deviation angle representing the angle between a first direction vector representing the instrument axis and a second direction vector representing the target trajectory axis.

[0014] An exemplary method for displaying surgical navigation information may include a head-mounted display, the surgical navigation system including a surgical instrument with a tip, given a surgical plan including a target trajectory axis. The method may include the step of displaying on the head-mounted display an augmented reality position alignment visualization including two axis-aligned deviation vectors, the two axis-aligned deviation vectors including a first vector and a second vector, wherein the first vector and the second vector represent decompositions of distance vectors from a point on the target trajectory axis to the tip of the surgical instrument. The method may further include the step of updating the augmented reality position alignment visualization displayed on the head-mounted display to indicate the relative position of the surgical instrument with respect to the target trajectory axis from the perspective of the head-mounted display.

[0015] Another exemplary method for aligning surgical instruments may include a surgical navigation system comprising a tracking unit configured to track the positions of a head-mounted display, a patient tracker, and a surgical instrument tracker. The surgical navigation system may be further configured to plan a target trajectory axis based on patient data registered to the patient tracker. The method for aligning surgical instruments may include the steps of: displaying an augmented reality positional alignment visualization on the head-mounted display as two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of distance vectors from a point on the target trajectory axis to the tip of the surgical instrument. The method may further include the step of: displaying an augmented reality angular alignment visualization on the head-mounted display including a deviation angle, the deviation angle showing the angle between a first direction vector of the surgical instrument's axis and a second direction vector of the target trajectory axis. The method may further include the step of: continuously updating the augmented reality positional alignment visualization and / or the augmented reality angular alignment visualization displayed on the head-mounted display based on the tracking unit to indicate the relative position of the surgical instrument with respect to the target trajectory axis from the perspective of the head-mounted display.

[0016] The advantages of the proposed equipment and methods are that they make medical interventions more effective, safer, and more precise. Attached Figure Description

[0017] The advantages of this disclosure will be readily understood when taken in conjunction with the following detailed description, in which:

[0018] Figure 1A This is a perspective view of the first configuration of a surgical navigation system used by surgeons, which includes a head-mounted display and a surgical tracking unit.

[0019] Figure 1B It is used for Figure 1A A schematic diagram of the control system of the surgical navigation system.

[0020] Figure 2 yes Figure 1A A perspective view of the patient tracker of the surgical tracking unit, which is coupled to the patient near the region of interest.

[0021] Figure 3 It is projected on Figure 1A A schematic diagram of a first configuration of augmented reality visualization on the lens of a head-mounted display, wherein the augmented reality visualization projected onto the lens includes virtual images superimposed on live features indicated by dashed lines.

[0022] Figure 4 It is projected on Figure 1A A schematic diagram of a second configuration of augmented reality visualization on the lens of a head-mounted display, wherein the augmented reality visualization projected onto the lens includes virtual images superimposed on live features indicated by dashed lines.

[0023] Figure 5 It is an enhanced graph of an exemplary augmented reality visualization depicted on a head-mounted display during a surgical procedure that includes surgical instruments.

[0024] Figure 6 This is a second exemplary augmented reality visualization of an enhanced image that a user observes on a head-mounted display during a surgical procedure that includes a patient tracker and preoperative image slices.

[0025] Figure 7 It is projected on Figure 1A A schematic diagram of a third configuration of augmented reality visualization on the lens of a head-mounted display, wherein the augmented reality visualization projected onto the lens includes a virtual image of preoperative data shown in the display window.

[0026] Figure 8 It is projected on Figure 1A A schematic diagram of a fourth configuration of augmented reality visualization on the lens of a head-mounted display, wherein the augmented reality visualization projected onto the lens includes virtual images of preoperative data. Detailed Implementation

[0027] Figure 1A and 1B An exemplary configuration of a surgical navigation system 20 is shown, which may include a tracking unit 10 and a head-mounted display 30. Surgeons can utilize the surgical navigation system 20 to assist in performing medical procedures, such as inserting pedicle screws as part of a multi-vertebra fixation procedure or removing a brain tumor.

[0028] The surgical navigation system 20 may include a navigation interface comprising one or more user input devices I and one or more displays 22. The user input devices I may be configured to allow surgeons to input or type patient data. Patient data may include patient images, such as preoperative images of the patient's anatomy. These images may be based on MRI, radiographic, or computed tomography (CT) scans of the patient's anatomy. Patient data may also include other information relating to the type of medical procedure being performed, the patient's anatomical features, the patient's specific medical condition, and / or the operational settings of the surgical navigation setup. For example, during spinal surgery, the surgeon may input information related to a specific vertebra being operated on via the user input device I. The surgeon may also input various anatomical dimensions related to the vertebra and / or the size and shape of medical devices or implants to be inserted during the medical procedure.

[0029] The display 22 of the surgical navigation system 20 can be configured to display various prompts or data input boxes. For example, the display 22 can be configured to display text boxes or prompts that allow the surgeon to manually enter or select the type of surgical procedure to be performed. The display 22 can also be configured to display patient data, such as preoperative images or scans. As mentioned above, preoperative images can be MRI scans, radiographic scans, or computed tomography (CT) scans based on the patient's anatomy. Preoperative images can be uploaded to the surgical navigation system and displayed on the display 22. The display 22 can be further configured to display a surgical plan for a medical procedure overlaid on the patient data or images. The surgical plan may include a surgical path for performing the medical procedure or a planned trajectory or orientation of a medical device during the medical procedure. The surgical plan may also include the location and / or orientation of an implant or medical device to be inserted during the medical procedure overlaid on the patient data or images.

[0030] The surgical navigation system 20 may further include a navigation controller 80. The navigation controller 80 may be a personal computer or a laptop computer. The navigation controller 80 may communicate with a user input device 1, a display 22, a central processing unit (CPU) and / or other processors, a memory (not shown), and a storage device (not shown). The navigation controller 80 may further include software and / or operating instructions relating to the operation of the surgical navigation system 20. This software and / or operating instructions may include a planning system configured to find the precise position and / or angular alignment of the surgical instrument 50 relative to the patient 60. The navigation controller 80 may communicate wirelessly with a head-mounted display. Therefore, the head-mounted display 30 may include a wireless or wired transceiver.

[0031] The surgical navigation system 20 may further include a tracking unit 10. The tracking unit 10 may also be referred to as a tracking system or a camera unit. The tracking unit 10 may include a housing 12 that includes a casing housing one or more position sensors 14. The position sensors may include cameras, such as charge-coupled device (CCD) cameras, CMOS cameras and / or optical imaging cameras, electromagnetic sensors, magnetoresistive sensors, radio frequency sensors, or any other sensors suitable for adequately sensing the position of navigation markers. In some configurations, at least two position sensors 14 may be employed, preferably three or four. For example, the position sensors 14 may be separate CCDs. In one configuration, three one-dimensional CCDs are employed. Two-dimensional or three-dimensional sensors may also be employed. It should be understood that in other configurations, separate tracking units 10 may also be arranged around the operating room, each tracking unit 10 having a separate CCD, or two or more CCDs. The CCDs detect optical signals, such as infrared (IR) signals.

[0032] The housing 12 of the tracking unit 10 can be mounted on an adjustable bracket or arm to allow for repositioning of the position sensor 14. For example, the adjustable bracket or arm can be configured to allow for repositioning of the position sensor 14 to ideally provide the best view of the surgical field of vision without obstructions.

[0033] Tracking unit 10 may include a sensor controller (not shown) that communicates with position sensor 14 (such as optical sensor 14) and is configured to receive signals from optical sensor 14. The sensor controller may communicate with navigation controller 80 via a wired and / or wireless connection. One such connection may be an RS-232 communication standard or an IEEE 1394 interface, both of which are serial bus interface standards for high-speed communication and synchronous real-time data transmission. The connection may also use company-specific protocols or network protocols such as UDP or TCP. In other embodiments, optical sensor 14 may be configured to communicate directly with navigation controller 80.

[0034] The tracking unit 10, which communicates with the surgical navigation system 20 via the navigation controller 80, can be used to determine the relative positions of the head-mounted display 30, surgical instruments 50, and patient 60 or region of interest 62. Using the relative positions of the head-mounted display 30, one or more surgical instruments 50, and patient 60, the navigation controller 80 of the surgical navigation system 20, registered with the patient 60 and the head-mounted display 30, is able to calculate augmented reality (AR) visualizations that can be displayed on the head-mounted display 30.

[0035] Surgical instrument 50 may include one or more instrument markers 52 configured to be detectable by position sensor 14 of tracking unit 10. Surgical instrument 50 may be configured to include passive tracking elements or instrument markers 52 (e.g., reflectors) for transmitting optical signals (e.g., reflecting light emitted from tracking unit 10) to position sensor 14. Alternatively, instrument markers 52 may include a radiopaque material that is identified and tracked by position sensor 14. In other configurations, active tracking markers may be employed. Active tracking markers may emit light, for example, light from a light-emitting diode, such as infrared light. Both active and passive configurations are possible. Instrument markers 52 may be arranged in a defined or known position and orientation relative to other instrument markers 52 to allow surgical navigation system 20 to determine the position and orientation (posture) of surgical instrument 50. For example, instrument markers 52 may be registered to surgical instrument 50 to allow surgical navigation system 20 to determine the position and / or orientation of the tip 54 or tool portion of surgical instrument 50 within a defined space, such as the surgical area.

[0036] The surgical navigation system 20 may further include a patient tracker 40, wherein the patient tracker 40 can be configured to position the patient 60 within a space such as the surgical area. The patient tracker 40 may include an attachment member 44 configured to secure the patient tracker 40 to the patient 60. The attachment member 44 may include a clamp, adhesive, strap, threaded fastener, or other similar attachment means. For example, the attachment member 44 may include a clamp configured to secure to the patient 60. This may include using the clamp to secure the patient tracker 40 to the vertebrae of the patient 60 adjacent to the region of interest 62. This may allow the tracking unit 10 to determine the position and / or orientation of the patient's spine during spinal surgery. Alternatively, the attachment 44 may include a strap configured to wrap around the patient tracker and secure it to the patient's head. This may allow the tracking system to determine the position and / or orientation of the patient's head during neurosurgical procedures.

[0037] The patient tracker 40 may further include one or more patient markers 42 configured to be detectable by the position sensor 14 of the tracking unit 10. The patient tracker 40 may be configured to include passive tracking elements or patient markers 42 (e.g., reflectors) for transmitting optical signals (e.g., reflecting light emitted from the tracking unit 10) to the position sensor 14. Optionally, the patient markers 42 may comprise a radiopaque material capable of being identified and tracked by the position sensor 14. The patient markers 42 may be arranged in a position and orientation defined or known relative to other patient markers 42 to allow the surgical navigation system 20 to determine the position and orientation (posture) of the patient 60 and / or the region of interest 62. In other configurations, active tracking markers may be employed. Active markers may be light emitted, such as by a light-emitting diode, or infrared light. Both active and passive configurations are possible. The patient markers 42 may be arranged in a position and orientation defined or known relative to other patient markers 42 to allow the surgical navigation system 20 to determine the position and orientation (posture) of the patient 60 and / or the region of interest 62.

[0038] Reference Figure 1A and Figure 1B As an addition to or alternative to one or more displays 22, a head-mounted display (HMD) 30 may be employed to enhance visualization before, during, and / or after surgery. The HMD 30 can be used to visualize the same objects visualized on the displays 22, and can also be used to visualize other objects, features, instructions, warnings, etc. The HMD 30 can be used to aid in locating and / or visualizing target objects relevant to the medical procedure. The HMD 30 can also be used to visualize instructions and / or warnings, among other uses, as further described below.

[0039] HMD 30 may be provided by Microsoft Corporation. Because it overlays augmented reality visualizations or computer-generated images onto the real world, it is referred to as a mixed or augmented reality HMD. It should be understood that any reference to augmented reality includes mixed reality. Therefore, in the configuration described herein, HMD 30 provides a computational holographic display. Other types of mixed / augmented reality HMDs can also be used, such as those that overlay computer-generated images onto real-world video images. HMD 30 may include a cathode ray tube display, a liquid crystal display, a liquid crystal on silicon display, or an organic light-emitting diode display. HMD 30 may include perspective techniques similar to those described herein, including diffractive waveguides, holographic waveguides, polarization waveguides, reflection waveguides, or switchable waveguides.

[0040] The HMD 30 includes a head-mounted structure 32, which may be in the form of eyeglasses, and may include an additional headband 38 or support to hold the HMD 30 on the user's head. In other embodiments, the HMD 30 may be integrated into a helmet or other structure worn on the user's head, neck, and / or shoulders.

[0041] The HMD 30 features a shield 32 and a lens / waveguide 36 structure. When the HMD 30 is placed on a user's head, the lens / waveguide 36 structure is configured to be positioned in front of the user's eyes. The waveguide transmits augmented reality visualizations or images (also known as computer-generated images, virtual images, or holograms) to the user's eyes, while the real image can be seen through the lens / waveguide 36 (which is transparent), allowing the user to see a blend of real and virtual objects in augmented reality.

[0042] Reference Figure 1B HMD 30 may also include a head-mounted display controller (HMD controller) 180 configured to communicate with the navigation controller 80 of the surgical navigation system 20. HMD controller 180 may include an image generator configured to generate augmented reality visualizations and transmit those visualizations to the user via the lens / waveguide 36 structure. HMD controller 180 may control the transmission of augmented reality visualizations to the lens / waveguide 36 structure of HMD 30. HMD controller 180 may be a separate computer located remotely from the HMD 30. Alternatively, HMD controller 180 may be integrated into the head-mounted structure 32 of HMD 30. HMD controller 180 may be a laptop computer, desktop computer, microcontroller, etc., having memory, one or more processors (e.g., a multi-core processor), input device 1, output device (a fixed display other than HMD 30), storage capacity device, etc.

[0043] HMD 30 includes one or more head-mounted display markers or HMD markers 34 configured to be detectable by position sensor 14 of tracking unit 10. HMD 30 may be configured to include passive tracking elements or HMD markers 34 (e.g., reflectors) for transmitting optical signals (e.g., reflecting light emitted from tracking unit 10) to position sensor 14. Alternatively, HMD markers 34 may comprise a radiopaque material capable of being detected and tracked by position sensor 14. In other configurations, active tracking markers may be employed. Active markers may emit light, such as infrared light, for example, from a light-emitting diode. Both active and passive configurations are possible. HMD markers 34 may be arranged in a defined or known position and orientation relative to other HMD markers 34 to allow surgical navigation system 20 to determine the position and orientation (posture) of HMD 30 within a defined area, such as the surgical area.

[0044] HMD 30 may also include a photo and / or video camera 170 that communicates with HMD controller 180. Camera 170 can be used to acquire photo or video images through HMD 30, which is useful for identifying objects or tags attached to objects, as will be described further below.

[0045] HMD 30 may further include an inertial measurement unit (IMU) 176 communicating with HMD controller 180. IMU 176 may include one or more 3-D accelerometers, 3-D gyroscopes, and other sensors to help determine the position and / or orientation of HMD 30 in the HMD coordinate system or to aid in tracking relative to other coordinate systems. HMD 30 may also include an infrared motion sensor 178 to recognize gesture commands from a user. Other types of gesture sensors are also contemplated. Infrared motion sensor 178 may be arranged to project infrared light or other light onto the front of HMD 30, enabling infrared motion sensor 178 to sense the user's hand, fingers, or other objects for the purpose of determining the user's gesture commands and accordingly controlling HMD 30, HMD controller 180, and / or navigation controller 80.

[0046] although Figure 1AThis may include only a single individual or surgeon wearing an HMD 30, but it is also anticipated that multiple HMDs 30 can be configured to communicate with the surgical navigation system 20. For example, the entire surgical team may wear HMDs 30. In some configurations, such as where a video camera 170 is integrated into the HMD 30 to provide point-of-view (POV) video, a computer-based surgical environment model can provide participant-specific mixed / augmented reality assistance through POV video stream analysis or simpler context-aware mechanisms (e.g., sensory-based feedback, heuristics based on input from isolated sensors, etc.) to facilitate the current task performed by that individual participant or to help them prepare for further contributions to the surgery or medical procedure.

[0047] In one configuration, two or more participants and their HMD 30s can be linked together, integrated with contextual information about the medical procedure. Participants can link by sharing their current POV or, more inherently, by sharing any object of interest that a participant is addressing at any given point in time. In this configuration, when a first participant directs his / her personal POV to a second participant's POV, the first participant can enhance his / her personal assessment of the second participant's situation through a mixed / augmented reality-assisted display. When the first participant becomes aware of opportunities to optimize or modify the planned medical procedure, appropriate interactions with different participants or the surgical environment will occur. This interaction can be performed directly by the first participant or facilitated by mixed / augmented reality assistance or other computer-assisted tools, which can be automatically generated or created by the first participant to support the second participant's actions.

[0048] Reference Figure 2 This illustrates an example representation of augmented reality visualization of the region of interest 62 perceived by the user through lens 36 of the HMD 30. (Example...) Figure 2 As shown, the patient tracker 40 can be attached or fixed to the patient 60 via the attachment member 44. This allows the tracking unit 10 to identify the position and / or orientation of the patient 60 and / or the region of interest 62 relative to the HMD 30 in order to generate and orient an appropriate augmented reality visualization on the lens 36, so that the virtual portion of the image can be correctly superimposed on the real portion of the patient 60 when viewed through the lens 36. Although in Figure 1A and 2 Only a single patient tracker 40 is shown, but it is envisioned that other patient trackers 40 can be connected to or attached to the patient 60. Using other patient trackers 40 can allow tracking of the position and / or movement of multiple portions of the patient 60 and / or the region of interest 62. The use of multiple patient trackers 40 can also improve the accuracy of tracking the position and / or movement of the patient 60 and / or the region of interest 62.

[0049] The augmented reality visualization of region of interest 62 may include both augmented reality boundary visualization and / or a combined boundary with augmented reality text labels. For example, as Figure 2 As shown, the augmented reality boundary visualization is displayed as a cube surrounding a single vertebra when projected or displayed on lens 36 of the HMD 30, as seen by the user when observing the actual spine of patient 60 through lens 36. The augmented reality boundary visualization can also be recognized and presented to the user on lens 36 of the HMD 30 via augmented reality text labels such as L1. For example, as... Figure 2 As shown, "L1" is included in the augmented reality visualization projected onto lens 36 of the HMD 30, used to identify specific vertebrae surrounded by the augmented reality boundary visualization. Although in Figure 2 Only a single augmented reality text label is shown, but it should be understood that multiple other augmented reality text labels may be included as part of the augmented reality visualization to mark and / or identify other features or objects for the HMD 30 user.

[0050] Reference Figure 3 This illustrates an additional configuration that allows the user to perceive augmented reality visualizations through lens 36 of the HMD 30. Figure 3 In the example configuration shown, the user is viewing a portion of the patient's head 60 through lens 36 of the HMD 30. For example, Figure 3 This can represent an augmented reality visualization as seen by a surgeon performing neurosurgery (e.g., tumor removal). Within lens 36, items are displayed virtually, indicated by dashed lines, representing real-world features that the user can see. This could include a portion of the patient 60's head, as well as incision sites or regions of interest 62. The user may also be able to see a portion of the patient tracker 40, attachment members 44, and / or patient markers 42 through lens 36 of the HMD 30.

[0051] HMD 30 users can also observe many augmented reality visualizations, which in Figure 3 The area within lens 36 is shown using solid lines. Such augmented reality visualization can include target objects representing single dot-like markers associated with medical procedures. For example, as... Figure 3 As shown, target object 130 may represent an item, such as a tumor or organ, that will be operated on within the patient 60 during a medical procedure. Alternatively, target object 230 may represent a medical device or implant to be inserted into the patient 60 during a medical procedure, which will be discussed in detail below.

[0052] Augmented reality visualization may also include a target trajectory axis 210. The target trajectory axis 210 may represent a planned or anticipated surgical path. For example, the target trajectory axis 210 may represent the optimal or preferred angle or direction for aligning and / or inserting surgical instruments 50 during a medical procedure. The target trajectory axis 210 may be defined by a solid or discontinuous line connecting the entry or insertion points of the target objects 130, 230 and the adjacent region of interest 62.

[0053] Augmented reality visualization may also include an augmented reality window, indicator, or text box 310. The augmented reality window 310 may include a window displayed on the lens 36 of the HMD 30. The augmented reality window 310 may be configured to display patient data, such as preoperative images or scans. The augmented reality window 310 may also be configured to display images of the planned surgical procedure, including identifying any key items the user might want to know during the medical procedure, such as nerves, organs, or similar elements. For example, when inserting a screw into the vertebra of patient 60, the image may include nerve endings to be avoided. The augmented reality window 310 may also include text or markings related to the medical procedure. For example, the augmented reality window 310 may include annotations related to the medical procedure, such as facts specific to the patient 60's medical condition. Optionally, the augmented reality window 310 may also include informational text related to the distance of the surgical instrument 50 from patient 60, region of interest 62, target objects 130, 230, and / or target trajectory axis 210.

[0054] The augmented reality visualization may also include one or more control buttons 320, 322, 324 displayed on the lens 36 of the HMD 30. The one or more control buttons 320, 322, 324 may be configured to allow a user to operate the augmented reality visualization displayed on the lens 36. For example, as described above, a user may use hand and / or facial gestures or movements to select or activate one of the control buttons 320, 322, 324. The control buttons 320, 322, 324 may be configured to adjust the contrast, transparency, and / or color of the augmented reality visualization on the lens 36. The control buttons 320, 322, 324 may also be used to operate on or enhance the information displayed on the lens 36. For example, the control buttons 320, 322, 324 may be configured to zoom in and / or zoom out on patient data displayed in the augmented reality window 310. This may include zooming in on a preoperative image or scan of the patient 60 to allow a user to better visualize areas close to the target objects 130, 230. The control buttons 320, 322, 324 may also be configured to rotate or move the image of the patient data. This may include moving the position and / or location of the augmented reality window 310 displayed on the lens 36 to avoid obstructing or interfering with the user's viewing of the augmented reality visualization on the patient 60.

[0055] Figure 4 Another configuration includes augmented reality visualization perceived by the user through lens 36 of HMD 30, including surgical instrument 50. Similar to the example above, augmented reality visualization may include target trajectory axis 210, target object 130, augmented reality window 310 and / or control buttons 320, 322, 324. However, augmented reality visualization may also include instrument axis 240. Instrument axis 240 may be defined by a line originating from tip 54 of surgical instrument 50 and extending from tip 54 along the normal axis of surgical instrument 50, wherein the normal axis substantially bisects surgical instrument 50. For example, as Figure 4 As shown, the instrument axis 240 is defined by the following line: this line originates at the tip 54 of the surgical instrument 50 and extends generally parallel to the longitudinal axis that divides the surgical instrument 50 in two. The instrument axis 54 may also protrude from the end or tip 54 of the surgical instrument 50.

[0056] Figure 4 The illustrated augmented reality visualization also includes an augmented reality position alignment visualization 220. The augmented reality position alignment visualization 220 includes two axis-aligned deviation vectors 222 and 224, which comprise a decomposition of the distance vector from a point on the target trajectory axis 210 to the tip 54 of the surgical instrument 50 or other parts of the surgical instrument 50. Axial alignment can mean that the lines representing the deviation vectors 222 and 224 are oriented parallel to one of the three principal axes of the reference coordinate system. Typically, the first deviation vector 222 may be oriented parallel to a first axis of the reference coordinate system, and the second deviation vector 224 may be oriented parallel to a second axis of the reference coordinate system. For example, the two axis-aligned deviation vectors 222 and 224 can mean that the line representing the first deviation vector 222 is oriented parallel to the x-axis, while the line representing the second deviation vector 224 is oriented parallel to the y-axis of the reference coordinate system. The reference coordinate system can be defined relative to HMD 30, patient tracker 40, surgical instrument 50, tracking unit 10, the user's line of sight, or another point within the surgical field.

[0057] The lines representing the first deviation vector 222 and the second deviation vector 224 can be configured to intersect at the origin, wherein the first deviation vector 222 and the second deviation vector 224 are positioned substantially perpendicular to each other. The origin of the first deviation vector 222 and the second deviation vector 224 can be positioned and / or set on or along the target trajectory axis 210. Optionally, the origin of the first deviation vector 222 and the second deviation vector 224 can be positioned and / or set near the tip 54 of the surgical instrument 50. In yet another configuration, the origin of the first deviation vector 222 and the second deviation vector 224 can be positioned and / or set within the augmented reality window 310 or float on the lens 36 of the head-mounted display 30.

[0058] The first deviation vector 222 can be defined by the lateral and / or horizontal position of the tip 54 of the surgical instrument 50 relative to the target trajectory axis 210 and / or the target object 130. The second deviation vector 224 can be defined by the longitudinal and / or vertical position of the tip 54 of the surgical instrument 50 relative to the target trajectory axis 210 and / or the target object 130. The magnitude and / or length of the first deviation vector 222 and / or the second deviation vector 224 can indicate the distance or deviation of the surgical instrument 50 relative to the target trajectory axis 210 in the direction of the respective deviation vector 222 or 224. For example, the longer the line representing the first deviation vector 222 or the second deviation vector 224, the farther the surgical instrument 50 may be positioned from the target trajectory axis 210. Alternatively, the shorter the line representing the first deviation vector 222 or the second deviation vector 224, the closer the surgical instrument 50 is positioned to the target trajectory axis 210. It can also be envisioned that the absence of a line representing the first deviation vector 222 or the second deviation vector 224 indicates that the surgical instrument 50 is correctly positioned and / or aligned in the direction corresponding to the missing deviation vectors 222, 224. The two axis-aligned deviation vectors 222, 224 can help the user correctly position and / or align the surgical instrument 50 relative to the target trajectory axis 210 and / or the target object 130. For example, the two axis-aligned deviation vectors 222, 224, including augmented reality position alignment visualization 220, are configured to inform the user of the relative position of the surgical instrument 50 relative to the target trajectory axis 210.

[0059] In each of the various configurations of augmented reality visualization, the visualization can be scaled to allow the user to see additional details. This scaling also allows the user to see smaller discrepancies. (See reference...) Figure 4The two axis-aligned deviation vectors 222 and 224 of the augmented reality position alignment visualization 220 can be sized and / or scaled to allow the user to see smaller deviations. For example, the lengths of the first deviation vector 222 and / or the second deviation vector 224 can be scaled by a factor of K, where a small deviation of two millimeters between the target trajectory axis 210 and the tip 54 of the surgical instrument 50 can be represented by a one-inch line representing the first deviation vector 222 and / or the second vector 224 displayed on the lens 36.

[0060] In another example, such as Figure 4 The decomposition of the distance vector into the two axis-aligned deviation vectors 222 and 224 shown can be based on two eigenvectors derived from: the two principal patient axes that form the largest angle with the target trajectory axis 210 among the three principal patient axes, or the collimation axis projected onto a plane perpendicular to the target trajectory axis 210 and attached to the point on the target trajectory axis 210 closest to the tip 54 of the surgical instrument 54, and a perpendicular vector in the same plane as the projected collimation axis. The decomposition of the distance vector into the two axis-aligned deviation vectors, including the augmented reality position alignment visualization 220, can be calculated based on the two eigenvectors. For example, these two eigenvectors can be based on the two principal patient axes that form the largest angle with the target trajectory axis 210 among the three principal patient axes of the patient coordinate system. The principal patient axes of the patient coordinate system can be derived from patient data, such as a 3D image dataset previously registered to the patient tracker 40. Alternatively, these two eigenvectors can be based on a reference coordinate system defined by the user's line of sight, which, for the surgeon, would increase the distinguishability of the first deviation vector 222 and / or the second deviation vector 224 as part of the augmented reality position alignment visualization 220. This may be helpful if the user's viewing direction is approximately perpendicular to the target trajectory axis 210. The reference coordinate system can be derived from a plane perpendicular to the target trajectory axis 210 and attached to the point on the target trajectory axis 210 closest to the tip 54 of the surgical instrument 50, and from the surgeon's line of sight. It is also conceivable that these two eigenvectors could be based on a combination of the primary patient axis of the patient coordinate system and the reference coordinate system defined by the user's line of sight.

[0061] In one configuration, the distance between the surgical instrument 50 and the target trajectory axis 210 is calculated based on the distance from the tip 54 of the surgical instrument 50 to at least one location selected from the group consisting of: a line segment connecting the target objects 130, 230 and the entry point of the target trajectory axis 210, a line connecting the target objects 130, 230 and the entry point of the target trajectory axis 210, and the entry point of the target objects 130, 230 or the target trajectory axis 210.

[0062] Reference Figure 5 This illustrates another configuration of augmented reality visualization from the angle viewed by the user through lens 36 of the HMD 30. As mentioned above, regarding Figure 4 Augmented reality visualization may include augmented reality position alignment visualization 220. Augmented reality visualization may also include augmented reality angle alignment visualization 200. Augmented reality angle alignment visualization 200 may include a deviation angle 206, which represents the angle between a first angle vector 204 representing an offset from and parallel to the instrument axis 240 and a second angle vector 202 representing the target trajectory axis 210. For example, augmented reality angle alignment visualization 200 is shown as a first angle vector 204 representing an axis offset from and parallel to the instrument axis 54 relative to the surgical instrument 50 and a second angle vector 202 representing the target trajectory axis 210. The first angle vector 204 and the second angle vector 202 may be connected by an arc representing the deviation angle 206 between the first angle vector 204 and the second angle vector 202. This may represent the deviation angle 206 between the target trajectory axis 210 and the instrument axis 54. As described above, the target trajectory axis 210 may be defined by a line connecting the target objects 130, 230 and an entry or insertion point near the region of interest 62. For example, as Figure 5 As shown, target object 230 may include an image of a screw superimposed on the patient at the planned location. This may include a superimposed image of the screw 230 at its location in the vertebrae of the patient 60 to be inserted. As described above, the augmented reality visualization may also include an augmented reality window 310, wherein the augmented reality window 310 includes labels, text, or similar markings for identifying a deviation angle 206 between the first angle vector 204 and the second angle vector 202. For example, the augmented reality window 310 may be positioned close to the augmented reality angle alignment visualization 200 and configured to display a text label identifying the deviation angle 206. The text label displayed in the augmented reality window 310 may include “30 degrees,” “1 radian,” or a similar angular measurement.

[0063] In another example, the visualization of the deviation angle 206 includes: augmented reality visualization of the position correction of the instrument axis 240, the target trajectory axis 210, and the arc connecting the proximal ends of both the instrument axis 240 and the target trajectory axis 210, or axonometry of the position correction augmented reality visualization. Axonometry is a graphical process belonging to descriptive geometry that generates a planar image of a 3D object. The term "axonometry" can be defined as "measuring along an axis" and can indicate that the size and scale of the coordinate axis are important. The result of the axonometry process is a parallel projection of the object at a uniform scale.

[0064] exist Figure 5 In the example configuration shown, target object 230 includes an augmented reality visualization of the planned location of a screw to be inserted into the vertebrae of patient 60, and a second angle vector 202 represents the orientation of the target trajectory axis 210 for inserting the screw into the identified location. The augmented reality visualization of target object 230 enhances spatial awareness when the actual surgical target is hidden from the surgeon's line of sight. Furthermore, an arc representing the deviation angle 206 connects the first angle vector 204 (representing the instrument axis 240 of surgical instrument 50) relative to target object 230 and the second angle vector 202 (representing the target trajectory axis 210), providing the user with a visual cue to approximate the angular deviation 206 between the first and second angle vectors 204 and 202.

[0065] The augmented reality angle alignment visualization 200 can also be scaled to allow users to more easily see smaller deviations in the deviation angle 206. For example, as Figure 5 As shown, the length of the line representing the first angle vector 204 and / or the length of the line representing the second angle vector 202 can be scaled by a factor of K to enlarge the length of the lines representing the first angle vector 204 and the second angle vector 202, allowing the user to more easily see smaller deviations in the deviation angle 206 represented by arcs. The scaling of the length of the line representing the first angle vector 204 and / or the line representing the second angle vector 202 is typically chosen such that the visualized length of the first angle vector 204 and / or the length of the line representing the second angle vector 202 corresponds to a few centimeters. In the case of additional visualization of target objects 130, 230 such as pedicle screws, the length of the lines representing the first angle vector 204 and / or the second angle vector 202 can also depend on the length / size of the target object 130, 230. Scaling can also include scaling the lines representing the first angle vector 204 and / or the second angle vector 202 to increase the length of the lines representing the first angle vector 204 and / or the second angle vector 202. Optionally, this may also include scaling the lines representing the first angle vector 204 and / or the second angle vector 202 to reduce the length of the lines representing the first angle vector 204 and / or the second angle vector 202, thereby reducing the size of the augmented reality angle alignment visualization 200 on the lens 36. This can prevent the augmented reality angle alignment visualization 200 from obstructing or interfering with the user's observation.

[0066] In another example, the decomposition of the deviation angle 206 can be scaled relative to two eigenvectors derived from: the two principal patient axes that form the largest angle with the target trajectory axis 210 among the three principal patient axes, or the collimation axis projected onto a plane perpendicular to the target trajectory axis 210 and attached to the point on the target trajectory axis 210 closest to the tip 54 of the surgical instrument 54, and a perpendicular vector in the same plane as the projected collimation axis. As mentioned above, these two eigenvectors can be based on the two principal patient axes that form the largest angle with the target trajectory axis 210 among the three principal patient axes of the patient coordinate system. Alternatively, these two eigenvectors can be based on a reference coordinate system defined by the user's line of sight, intended for the surgeon to increase the distinguishability of the first deviation vector 222 and / or the second deviation vector 224 as part of the augmented reality position alignment visualization 220. This may be helpful if the user's viewing direction is approximately perpendicular to the target trajectory axis 210.

[0067] Furthermore, it should be understood that the various configurations of the aforementioned augmented reality visualizations may include highlighting and / or color schemes to allow users to distinguish different types of augmented reality visualizations. For example, augmented reality position alignment visualization 220 may be displayed on lens 36 of HMD 30 in a first color, and augmented reality angle alignment visualization 200 may be displayed on lens 36 of HMD 30 in a second color. The first and second colors can be selected to distinguish them from each other. The distinguishable and / or different colors for the various features or elements used in augmented reality visualizations can be selected from the base colors of the trajectory by arranging alternating colors equidistantly in a chromaticity space around a white point and selecting their dominant wavelengths by extrapolation. These colors can be selected from the following: high-intensity complementary colors selected from the group consisting of yellow, pink, green, and cyan. (See again) Figure 5 In a non-limiting example, the first deviation vector 222 and the second deviation vector 224 of the augmented reality position alignment visualization 220 can be displayed as purple lines on the lens 36 of the HMD 30. In contrast, the first angle vector 204 and the second angle vector 202 of the augmented reality angle alignment visualization 200 can be displayed as blue or cyan lines on the lens 36 of the HMD 30. Similarly, different colors can be used to distinguish the target trajectory axis 210 from other visualizations. For example, the target trajectory axis 210 can be displayed as a yellow line on the lens 36 of the HMD 30. It should be understood that any combination of colors can be used for each of the various augmented reality visualizations to distinguish them from one another.

[0068] The augmented reality visualizations described above can also be differentiated by varying the transparency and / or opacity of each augmented reality visualization, as shown on the lens 36 of the HMD 30. For example, augmented reality visualizations can be configured such that augmented reality position alignment visualization 220 can be displayed as an opaque line on the lens 36 of the HMD 30, and augmented reality angle alignment visualization 200 can be displayed as a line that is at least partially transparent or completely invisible on the lens 36 of the HMD 30. In another example, where the augmented reality visualization includes multiple tracks and / or axes, all of which, except for the one at the minimum distance from the instrument axis 240 and / or tip 54 of the surgical instrument 50, can be displayed with increased transparency. From the user's perspective, this eliminates unnecessary or less important augmented reality visualizations.

[0069] Similarly, the type of line can be used to distinguish the various augmented reality visualizations described above. See again... Figure 5 In a non-limiting example, the first deviation vector 222 and the second deviation vector 224 of the augmented reality position alignment visualization 220 can be displayed as solid lines with defined linewidths on the lens 36 of the HMD 30. In contrast, the first angle vector 204 and the second angle vector 202 of the augmented reality angle alignment visualization 200 can be displayed as solid lines with linewidths different from those of the augmented reality position alignment visualization 220 on the lens 36 of the HMD 30. Similarly, different line types can be used to distinguish the target trajectory axis 210 from other visualization areas. For example, the target trajectory axis 210 can be displayed as a dashed line, a dotted line, or a similarly different line type on the lens 36 of the HMD 30. It should be understood that any combination of line types can be used in each of the various augmented reality visualizations to distinguish them from one another.

[0070] Figure 6 An example configuration of augmented reality visualization is shown, comprising only the virtual image perceived by the user through lens 36 of the HMD 30. (Example shown) Figure 6 As shown, augmented reality visualization can include virtual images of slices of patient image data, which, when viewed through lens 36, are displayed above the actual surgical site or region of interest 62 in a fixed-position floating box 300. Similar to the example above, the user can still view the actual features of the patient 60, such as the patient 60's spine, through lens 36 of the HMD 30. However, additional information or images, such as patient data, can be displayed in the floating box 300 on lens 36. For example, as... Figure 6As shown, augmented reality visualization can include sliced ​​images of patient data, such as two-dimensional images of specific vertebrae. Axes or coordinate systems can be overlaid on the slices of patient data depicted in the floating box 300. The axes or coordinate systems can be configured to represent the position and / or orientation of the surgical instrument 50 relative to the patient data depicted in the floating box 300.

[0071] Reference Figure 7 In another example configuration of augmented reality visualization, the augmented reality visualization may include a virtual image displayed on the lens 36 of the HMD30 and in the display window 330. This differs from some existing example configurations of augmented reality visualization described above. Figure 7 The augmented reality visualization shown is an example of an augmented reality visualization that only includes a virtual image displayed on lens 36. For example, the augmented reality visualization does not require the user of HMD 30 to be within the line of sight of patient 60 and / or area of ​​interest 62. This augmented reality visualization can be displayed on lens 36 to allow the user to view an image or depiction of the surgical plan superimposed on a virtual image or model of patient 260. Similar to the floating frame 300 described above, display window 330 can be configured to depict patient data, such as images or patient scans. Display window 330 can be further configured to depict target trajectory axis 210, instrument axis 240, and / or target object 130 superimposed on the patient data. As in a similar example described above, the user can use control buttons 320, 322, 324 to operate the display window 330 of the augmented reality visualization. For example, the user can use gestures to select one or more of control buttons 320, 322, 324 to zoom in or out for a better visual effect of target trajectory axis 210, instrument axis 240, and / or target object 130.

[0072] Figure 8An alternative configuration for augmented reality visualization is shown, which includes a virtual image and / or patient data of a portion of patient 260, as displayed on lens 36 of HMD 30. That portion of patient 260 and / or patient data displayed on lens 260 as augmented reality visualization may include a target trajectory axis 210, a region of interest 262, and / or a target object 130. Augmented reality visualization may also include depictions of additional anatomical features and / or planned surgical paths overlaid on patient 260 and / or patient data. For example, augmented reality visualization may include a virtual 3D image of patient 260 displayed on lens 36. This allows the user to visualize the additional anatomical features and / or planned surgical paths on lens 36 of HMD 30, rather than having to shift their attention to another external display. Augmented reality visualization may also include additional control buttons 326, 328, which are configured to operate the augmented reality visualization displayed on lens 36. For example, one of control buttons 326, 328 may be configured to zoom in and / or zoom out. Additionally, one of the control buttons 326 and 328 can be configured to rotate the augmented reality visualization to allow the user to observe or view the augmented reality visualization from different angles or perspectives. When the augmented reality visualization is a virtual 3D image of the patient 260, the control buttons 326 and 328 can be configured to rotate the virtual 3D model of the patient 260 to allow the user to view the surgical plan and / or the target trajectory axis 210 from multiple angles.

[0073] A method for aligning a surgical instrument 50 using a surgical navigation system 20 including the aforementioned head-mounted display 30 may include planning a target trajectory axis 210 based on patient data registered to a patient tracker 40. The method may further include the step of displaying an augmented reality position alignment visualization 220 on the head-mounted display 30 as two axis-aligned deviation vectors 222, 224, which include a decomposition of the distance vector from a point on the target trajectory axis 210 to the tip 54 of the surgical instrument 50. For example, the method may include displaying a first deviation vector 222 and a second deviation vector 224 on a lens 36 of the head-mounted display 30. The first deviation vector 222 and / or the second deviation vector 224 may be displayed as solid lines and / or dashed lines. The first deviation vector 222 and the second deviation vector 224 may also be displayed as arrows.

[0074] The method may further include the step of displaying an augmented reality angle alignment visualization 200 on a head-mounted display 30, including a deviation angle 206, which shows the angle between a first angle vector 204 representing the instrument axis 240 of the surgical instrument 50 and a second angle vector 202 representing the target trajectory axis 210. For example, the method may include displaying the first angle vector 204 and the second angle vector 202 as a line connected by an arc representing the deviation angle 206 between the first angle vector 204 and the second angle vector 202.

[0075] The method may further include the following steps: updating the augmented reality position alignment visualization 220 and / or augmented reality angle alignment visualization 200 displayed on the head-mounted display 30 based on measurement data from the tracking unit 10, to indicate the relative position of the surgical instrument 50 to the target trajectory axis 210 from the angle of the head-mounted display 30. The target trajectory axis 210 may be defined by a line connecting the target objects 130, 230 and the entry point. This update may be continuous.

[0076] The method of aligning the surgical instrument 50 may further include augmented reality visualization of the instrument axis 240 of the surgical instrument 50. The instrument axis 240 may be displayed as a line or outline of the surgical instrument 50. The augmented reality visualization of the instrument axis 240 displayed on the lens 36 of the head-mounted display 30 may indicate the position of the surgical instrument 50 relative to the patient 60, the region of interest 62, the target trajectory axis 210, and / or the target objects 130, 230.

[0077] The method for aligning the surgical instrument 50 may further include using different colors to display augmented reality positional alignment visualization 220 and / or augmented reality angular alignment visualization 200, allowing the user to distinguish the corresponding alignment visualizations. Different colors for augmented reality visualization can be selected from the base color of the trajectory by arranging alternating colors equidistantly in a color space around a white point and selecting their dominant wavelengths by extrapolation. For example, these colors can be selected from the following: high-intensity complementary colors selected from the group consisting of yellow, pink, green, and cyan.

[0078] The method may further include highlighting augmented reality positional alignment visualization 220 and / or augmented reality angle alignment visualization 200 to the user based on the distance from the surgical instrument 50 to the target trajectory axis 210. For example, the color and / or line width of the line representing augmented reality positional alignment visualization 220 may be different from the color and / or line width of the line representing augmented reality angle alignment visualization 200. As described above, this can be used to allow the user to distinguish between augmented reality positional alignment visualization 220 and / or augmented reality angle alignment visualization 200 displayed on the lens 36 of the head-mounted display 30.

[0079] The step of highlighting the target trajectory axis 210 may further include the step of hiding all other trajectories except the one with the minimum distance from the user. The step of highlighting the target trajectory axis 210 may also include displaying all other trajectories except the one with the minimum distance with increased transparency. For example, when the surgical instrument 50 is correctly aligned with the direction and orientation of the first deviation vector 222 corresponding to the augmented reality position alignment visualization 220, the first deviation vector 222 may be hidden or displayed transparently on the lens 36 of the head-mounted display 30. Conversely, if the surgical instrument 50 is not aligned with the direction and orientation of the first deviation vector 222 corresponding to the augmented reality position alignment visualization 220, the first deviation vector 222 may be highlighted on the lens 36 of the head-mounted display 30 to send a signal to the user that the alignment needs to be corrected based on the target trajectory axis 210.

[0080] The method of aligning the surgical instrument 50 may further include the following steps: displaying an augmented reality position alignment visualization 220 on a head-mounted display 30 as two axis-aligned deviation vectors 222, 224, wherein the distance between the surgical instrument 50 and the target trajectory axis 210 is calculated based on the distance between the tip 54 of the surgical instrument 50 and at least one selected from the group consisting of: a line segment connecting the target objects 130, 230 and the entry point of the target trajectory axis 210, a line connecting the target objects 130, 230 and the entry point of the target trajectory axis 210, and the entry point of the target objects 130, 230 or the target trajectory axis 210.

[0081] The method of aligning the surgical instrument 50, wherein the distance vector is decomposed into two axis-aligned deviation vectors 222, 224 relative to two intrinsic vectors derived from: the two main patient axes that form the maximum angle with the target trajectory axis 210 among the three main patient axes, or the collimation axis projected onto a plane that is perpendicular to the target trajectory axis 210 and attached to the point on the target trajectory axis 210 closest to the tip 54 of the surgical instrument 54, and a perpendicular vector in the same plane as the projected collimation axis.

[0082] The method for aligning the surgical instrument 50 may further include a step of scaling the augmented reality position alignment visualization 220 to allow the user to more easily observe the position and / or small alignment deviations of the surgical instrument 50 relative to the target trajectory axis 210. For example, the lengths of the first deviation vector 222 and / or the second deviation vector 224 may be magnified to allow the user to more easily observe small deviations. Optionally, the lengths of the first deviation vector 222 and / or the second deviation vector 224 may be reduced to allow the augmented reality visualization to fit onto the lens 36 of the head-mounted display 30. The scaling of the distance vector decomposition may be limited to the absolute maximum visible length and the maximum visible length relative to the field of view of the head-mounted display 30.

[0083] The method of aligning the surgical instrument 50, wherein the decomposition of the deviation angle 206 is relative to two intrinsic vectors derived from: the two of the three principal patient axes that form the largest angle with the target trajectory axis, or the collimation axis projected onto a plane that is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip 54 of the surgical instrument 54, and a perpendicular vector in the same plane as the projected collimation axis.

[0084] A method for aligning surgical instruments 50, wherein the visualization of the deviation angle 206 includes: an instrument axis 240, a target trajectory axis 210, and a position-corrected augmented reality visualization of the arc representing the deviation angle 206 connecting the proximal ends of a first angle vector 204 representing the instrument axis 240 and a second angle vector 202 representing the target trajectory axis 210, or an axis of the position-corrected augmented reality visualization.

[0085] The method of aligning the surgical instrument 50 may further include the following steps: displaying a first augmented reality text label 310 for describing the distance from the tip 54 of the surgical instrument 50 to the target trajectory axis 210, and displaying a second augmented reality text label 310 positioned in or near the angular visualization for describing the deviation angle 206 between the instrument axis 240 and the target trajectory axis 210 in degrees.

[0086] The method of aligning surgical instruments 50 may further include the following steps: displaying an augmented reality visualization of target objects 130, 230 to be placed or removed at target points on the target trajectory axis 210.

[0087] The method of aligning the surgical instrument 50 may further include the following steps: displaying an augmented reality visualization of the target objects 130, 230 to be placed or removed at a target point on the target trajectory axis 210, wherein the augmented reality visualization of the target objects 130, 230 is positioned at the target location or at the current location of the surgical instrument 50.

[0088] The method of aligning the surgical instrument 50 may further include the step of displaying an augmented reality visualization of a slice of patient image data. This step may include the user selecting the augmented reality visualization location as one of the following: a fixed frame in space, a floating frame 300 that follows head movement, the location of the patient image data slice in situ, or a user-defined fixed spatial vector offset from the user's position. The user may manipulate the slice of patient image data using control buttons 320, 322, 324, 326, 328 displayed on a lens 36 of the head-mounted display 30. The user may use gestures and / or facial expressions to select the control buttons 320, 322, 324, 326, 328 displayed on the lens 36. This step may further include color mapping from patient image data color information to augmented reality visualization color information including an alpha transparent target range. This step may further include user interaction, through which the user may select and reposition the augmented reality visualization location and select the patient image data slice using voice, mouse, keyboard, gaze, or any of the surgical instruments.

[0089] The method of aligning the surgical instrument 50 may further include displaying a region of interest (ROI) indicator. The ROI indicator may include an augmented reality text label 310 displayed near the region of interest 62 or an augmented reality boundary visualization delineating the region of interest 62. The ROI indicator may further include a color mapping method that highlights the augmented reality text label 310 and the augmented reality boundary visualization to the user based on the distance of the surgical instrument 50 to the region of interest 62.

[0090] Several configurations of augmented reality visualization have been described above, such as augmented reality position alignment visualization 220 and / or augmented reality angle alignment visualization 200. While many configurations of augmented reality visualization are described with respect to display on lens 36 of head-mounted display 30 and / or on head-mounted display 30, it should be understood that any different configuration of augmented reality visualization may also be displayed on display 22. Display 22 may include a cathode ray tube display (CRT), a light-emitting diode display (LED), an electroluminescent display (ELD), a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a digital light processing display (DLP), a projection monitor, or a similar device.

[0091] Several configurations of the surgical navigation system 20 and / or augmented reality visualization have been discussed in the preceding description. However, the configurations discussed herein are not intended to be exhaustive or to limit this disclosure to any particular form. The terminology used is intended to be descriptive in nature and not restrictive. In light of the above teachings, many modifications and variations are possible, and this disclosure may be practiced in ways other than those specifically described.

[0092] Clauses for additional configuration (original EP claims)

[0093] I. A guidance system for aligning surgical instruments, comprising: a stereoscopic head-mounted display including at least an HMD tracker; a surgical navigation system having a tracking system; patient data registered to a patient tracker that can be tracked by the tracking system; a planning system for planning one or more trajectories on the patient data; a navigated instrument tracked by the tracking system; AR position alignment visualization; and AR angle alignment visualization.

[0094] The guidance system of Clause Ia. I can be configured to display two axis-aligned deviation vectors for visualization of position alignment, the two axis-aligned deviation vectors being a decomposition of the distance vector from the trajectory axis of the tip of the guided instrument to the nearest point on the tip of the guided instrument, and to scale the length using a scaling function to allow the user to see small deviations on a stereoscopic head-mounted display.

[0095] The guidance system of Clause Ib. or Ia may be further configured to display an angular alignment visualization consisting of one or two deviation angles, which show the angle between the direction vector of the instrument axis and the direction vector of the trajectory axis, or to decompose the angle into two deviation angles, each of which can be scaled proportionally to the open angle by a scaling function to allow the user to see small deviations.

[0096] The guidance system of any of Clauses I, Ia, and Ib may be further configured to continuously update the visualization based on the tracking system to display the relative position of the guided device and patient data from the perspective of both eyes of the stereoscopic head-mounted display.

[0097] The guidance system of any of the clauses I, Ia, Ib and Ic can be further configured to use different colors for the visualizations, enabling the user to distinguish the corresponding alignment visualizations and highlight the visualizations to the user based on the distance of the guided instrument to each trajectory.

[0098] II. A method for aligning surgical instruments using the guidance system of Clause I, the method comprising the steps of: continuously updating visualizations based on a tracking system to display the relative position of the navigated instrument to patient data from the perspective of both eyes of a stereoscopic HMD; using different colors for the visualizations so that the user can distinguish the corresponding alignment visualizations; and highlighting the visualizations to the user according to the distance of the navigated instrument to each trajectory.

[0099] II-a. The method according to Clause II may further include using and / or displaying two axis-aligned deviation vectors for position alignment visualization, the two axis-aligned deviation vectors being decomposed from the distance vectors on the trajectory axis of the tip of the navigated instrument to the nearest point on the tip of the navigated instrument, and the lengths being scaled by a scaling function to allow the user to see small deviations.

[0100] Il-b. The method according to Clause II or II-a may further include using and / or displaying an angular alignment visualization consisting of one or two deviation angles, said deviation angles showing the angle between the direction vector of the instrument axis and the direction vector of the trajectory axis, or decomposing said angle into two deviation angles, each of which can be scaled proportionally to the open angle by a scaling function to allow the user to see small deviations.

[0101] II-c. The method according to any one of clauses II, II-a and II-b, characterized in that the distance between the guided instrument and the trajectory is calculated based on the distance from the tip of the guided instrument to at least one selected from the group consisting of: a line segment connecting the target and the entry point of the trajectory, a line connecting the target and the entry point of the trajectory, and the entry point of the target or the trajectory.

[0102] II-d. The method according to any one of clauses II, II-a and II-b, characterized in that the angular deviation between the navigated instrument and the trajectory is calculated based on the angle between the normal to the axis of the navigated instrument and the normal to the line connecting the entry point of the target and the trajectory.

[0103] II-e. The method according to any one of clauses II, II-a and II-b, characterized in that the highlighting of the trajectory hides all trajectories except the trajectory with the minimum distance from the user, and displays all trajectories except the trajectory with the minimum distance with increased transparency.

[0104] II-f. The method according to any one of clauses II, II-a and II-b, characterized in that the distance vector is decomposed into two axis-aligned deviation vectors relative to two intrinsic vectors obtained from: the two main patient axes that form the largest angle with the trajectory axis among the three main patient axes, or the collimation axis projected onto a plane that is perpendicular to the trajectory axis and attached to the point on the trajectory axis closest to the tip of the navigated instrument, and a perpendicular vector in the same plane as the projected collimation axis.

[0105] II-g. The method according to any one of clauses II, II-a and II-b, characterized in that the scaling of the distance vector decomposition is limited to the absolute maximum visible length and the maximum visible length relative to the field of view of the head-mounted display.

[0106] II-h. The method according to any one of clauses II, II-a and Il-b is characterized in that the decomposition of the two deviation angles is relative to two eigenvectors obtained from: the two main patient axes that form the largest angle with the trajectory axis among the three main patient axes, or the collimation axis projected onto a plane that is perpendicular to the trajectory axis and attached to the five nearest points on the trajectory axis to the tip of the navigated instrument, and a vertical vector in the same plane as the projected collimation axis.

[0107] II-i. The method according to any one of clauses II, II-a and Il-b, characterized in that the visualization of the deviation angle includes: a position correction augmented reality visualization of the following: the current normal axis, the target normal axis, and the arc connecting the proximal ends of the current normal axis and the target normal axis, or the axis of the position correction augmented reality visualization.

[0108] II-j. The method according to any one of Clauses II, II-a, and II-b, further using AR text labels for describing the distance from the instrument tip to the trajectory target, and AR text labels positioned in or near the angular visualization for describing the angular deviation between the instrument axis and the trajectory axis in degrees.

[0109] II-k. The method according to any one of Clauses II, II-a and II-b is further used with AR visualization of the trajectory.

[0110] II-1. The method according to any one of Clauses II, II-a and II-b, further using AR visualization of a target object to be placed or removed at a target point on the trajectory, characterized in that the AR visualization of the target object is located at the target location or at the current location of the navigation tool.

[0111] II-m. The method according to any one of clauses II, II-a and Il-b further includes the step of AR visualization of the target object to be placed or removed at the target point of the trajectory.

[0112] II-n. The method according to any one of clauses II, II-a and Il-b is characterized in that different colors for visualization are selected from the base colors of the trajectory by arranging alternating colors equidistantly in a chromaticity space around the white point and selecting their dominant wavelengths by extrapolation, or the colors are selected from the following: high-intensity complementary colors selected from the group including yellow, pink, green and cyan.

[0113] II-o. The method according to any one of clauses II, II-a, and Il-b further includes the step of AR visualization of a patient image data slice, the step comprising the following steps: the surgeon selecting the AR visualization location as one of: a fixed box in space, a floating box that follows head movement, in situ at the location of the patient image data slice, or a user-defined fixed spatial vector offset from the patient's location; color mapping from patient image data color information to AR visualization color information including a target range of alpha transparency; and user interaction, through which the user is able to select and reposition the AR visualization location and select the patient image data slice using any one of voice, mouse, keyboard, gaze, or a guided surgical instrument.

[0114] II-p. The method according to any one of Clauses II, II-a and Il-b further includes the use of a region of interest indicator, the region of interest indicator comprising: an AR text label or an AR boundary visualization delineating the region of interest near the region of interest, and a color mapping method that highlights the AR text label and the AR boundary visualization to the user based on the distance of the navigated device from the region of interest.

[0115] This application includes the following examples.

[0116] Example 1. A surgical navigation system, comprising:

[0117] Head-mounted displays including lenses;

[0118] Surgical navigation systems including tracking units;

[0119] A patient tracker that is registered in the patient data and can be tracked by the surgical navigation system;

[0120] Surgical instruments, having an instrument tracker capable of being tracked by the surgical navigation system, the instruments defining an instrument axis; and

[0121] A controller is configured to generate augmented reality positional alignment visualization as two axis-aligned deviation vectors displayed on a lens of the head-mounted display, the two axis-aligned deviation vectors comprising a decomposition of the distance vector from a target point on the target trajectory axis to a point on the surgical instrument.

[0122] Example 2. The surgical navigation system of Example 1, wherein the controller is configured to scale two axis-aligned deviation vectors displayed on the lens.

[0123] Example 3. The surgical navigation system of Example 1, wherein the controller is configured to generate augmented reality angle alignment visualizations to be displayed on the lens.

[0124] Example 4. The surgical navigation system of Example 3, wherein the augmented reality angle alignment visualization includes a deviation angle, the deviation angle representing the angle between a first angle vector representing an axis parallel to the instrument axis and a second angle vector representing the target trajectory axis.

[0125] Example 5. The surgical navigation system of Example 4, wherein the controller is configured to scale the lengths of a first line and a second line representing the deviation angle to allow the user to see small deviations in the deviation angle.

[0126] Example 6. The surgical navigation system of Example 3, wherein the augmented reality angle alignment visualization includes decomposing the angle between a first angle vector representing the instrument axis and a second angle vector representing the target trajectory axis into two deviation angles.

[0127] Example 7. The surgical navigation system described in Example 6, wherein the length of the line representing the two deviation angles is configured to be scaled to allow the user to see small deviations in the two deviation angles.

[0128] Example 8. A surgical navigation system according to any one of Examples 3 to 7, wherein the augmented reality position alignment visualization includes a first color, and the augmented reality angle alignment visualization includes different colors that can be distinguished from the first color.

[0129] Example 9. A head-mounted display system for use with a surgical navigation system, a patient tracker capable of being tracked by the surgical navigation system, and surgical instruments capable of being tracked by the surgical navigation system, the head-mounted display comprising:

[0130] Lens; and

[0131] A controller is configured to visualize augmented reality position alignment on the lens as two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of the distance vector from the target point on the target trajectory axis to the surgical instrument.

[0132] Example 10. The head-mounted display system of Example 9, wherein the controller is configured to generate augmented reality angle alignment visualization on the lens.

[0133] Example 11. The head-mounted display system of Example 10, wherein the augmented reality angle alignment visualization includes a deviation angle, the deviation angle representing the angle between a first angle vector representing the axis of the instrument and a second angle vector representing the axis of the target trajectory, the lengths of the first and second lines representing the deviation angle being configured to be scaled to allow the user to see small deviations.

[0134] Example 12. The head-mounted display system of Example 10, wherein the augmented reality angle alignment visualization includes decomposing the angle between a first angle vector representing the axis of the instrument and a second angle vector representing the axis of the target trajectory of the angle into two deviation angles, wherein the lengths of the lines representing the two deviation angles are configured to be scaled to allow the user to see small deviations.

[0135] Example 13. The head-mounted display system of any one of Examples 10 to 12, wherein the augmented reality position alignment visualization includes a first color, and the augmented reality angle alignment visualization includes different colors that can be distinguished from the first color.

[0136] Example 14. A method for displaying surgical navigation information using a head-mounted display system, considering a surgical plan including a target trajectory axis, the head-mounted display system comprising a surgical navigation system and surgical instruments having a tip and at least partially defining an instrument axis, the method comprising:

[0137] An augmented reality position alignment visualization is displayed on a head-mounted display, comprising two axis-aligned deviation vectors, namely a first deviation vector and a second deviation vector, wherein the first and second deviation vectors represent the decomposition of distance vectors from a point on the target trajectory axis to the tip of the surgical instrument; and

[0138] The augmented reality position alignment visualization displayed on the head-mounted display is updated to indicate the relative position of the surgical instruments with respect to the target trajectory axis from the perspective of the head-mounted display.

[0139] Example 15. The method of Example 14, wherein decomposing the distance vector into two axis-aligned deviation vectors is relative to two eigenvectors derived from:

[0140] Two of the three main patient axes derived from the patient's orientation form the largest angle with the target trajectory axis.

[0141] Example 16. The method of Example 14, wherein decomposing the distance vector into two axis-aligned deviation vectors is relative to two eigenvectors derived from:

[0142] The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

[0143] Example 17. The method of Example 14 further includes displaying an augmented reality angular alignment visualization including a deviation angle, the deviation angle showing the angle between a first angular vector representing the axis of the surgical instrument and a second angular vector representing the axis of the target trajectory.

[0144] Example 18. The method of Example 17 further includes updating the augmented reality angle alignment to indicate the angle of the axis of the surgical instrument relative to the target trajectory axis from the angle of the head-mounted display.

[0145] Example 19. The method described in Example 17 or 18 further includes using a first color to display the augmented reality position alignment visualization or the augmented reality angle alignment visualization, and using a different color to display the other of the augmented reality position alignment visualization or the augmented reality angle alignment visualization.

[0146] Example 20. The method of Example 17 or 18 further includes highlighting the augmented reality position alignment visualization and / or the augmented reality angle alignment visualization to the user based on the distance of the surgical instrument to the target trajectory axis.

[0147] Example 21. The method of Example 14, wherein the distance between the surgical instrument and the target trajectory axis is calculated based on the distance from the tip of the surgical instrument to at least one basis selected from the group consisting of: a line segment connecting the target object and the entry point of the target trajectory axis, a line connecting the target object and the entry point of the target trajectory axis, and the entry point of the target object or the trajectory.

[0148] Example 22. A head-mounted display for use with a surgical navigation system including a patient tracker and a surgical instrument tracker, the surgical navigation system being configured to plan a target trajectory axis based on patient data and to align an instrument axis, at least partially defined by the tip of a surgical instrument, with the target trajectory axis, the head-mounted display comprising:

[0149] lens;

[0150] The head-mounted display is configured to display augmented reality position alignment visualization including two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of a distance vector from a point on the target trajectory axis to the tip of the surgical instrument; and / or

[0151] The head-mounted display is further configured to display an augmented reality angular alignment visualization including a deviation angle, which represents the angle between a first angular vector representing the axis of the instrument and a second angular vector representing the axis of the target trajectory.

[0152] Example 23. The system described in Example 22, wherein the two axis-aligned deviation vectors are configured to be scaled proportionally in length to allow a user to see small deviations in the distance from the surgical instrument axis to the target trajectory axis.

[0153] Example 24. The system described in Example 22, wherein the deviation angle is configured to be scaled to allow a user to see a small deviation in angle between the first angle vector and the second angle vector.

[0154] Example 25. The system described in Example 22, wherein the augmented reality angle alignment visualization includes decomposing the angle between a first angle vector representing the axis of the instrument and a second angle vector representing the axis of the target trajectory of the angle into two deviation angles.

[0155] Example 26. The system described in Example 22, wherein the augmented reality position alignment visualization includes a first color, and the augmented reality angle alignment visualization includes a second color; and

[0156] The first color can be distinguished from the second color.

[0157] Example 27. A method for aligning surgical instruments using a surgical navigation system, the surgical navigation system including a tracking unit configured to track the position of a head-mounted display, a patient tracker, and a surgical instrument tracker coupled to a surgical instrument having a tip and defining an instrument axis, wherein the surgical navigation system is configured to plan a target trajectory axis based on patient data registered to the patient tracker, the method comprising the steps of:

[0158] The augmented reality positioning alignment is visualized on the head-mounted display as two axis-aligned deviation vectors, which include a decomposition of the distance vector from a point on the target trajectory axis to the tip of the surgical instrument, and / or

[0159] An augmented reality angular alignment visualization, including a deviation angle, is displayed on the head-mounted display. The deviation angle shows the angle between a first angular vector representing the axis of the surgical instrument and a second angular vector representing the axis of the target trajectory.

[0160] The tracking unit continuously updates the augmented reality position alignment visualization and / or augmented reality angle alignment visualization displayed on the head-mounted display to indicate the position of the surgical instrument axis relative to the target trajectory axis from the perspective of the head-mounted display.

[0161] Example 28. The method described in Example 27, wherein the target trajectory axis is defined by a line connecting the target object and the entry point.

[0162] Example 29. The method of Example 27 or 28 further includes augmented reality visualization of the instrument axis of the surgical instrument on the head-mounted display.

[0163] Example 30. The method of Example 27 further includes using different colors to display the augmented reality position alignment visualization and / or the augmented reality angle alignment visualization, so that the user can distinguish the corresponding alignment visualization; and

[0164] Based on the distance from the surgical instrument to the target trajectory axis, the augmented reality position alignment visualization and / or augmented reality angle alignment visualization are highlighted to the user.

[0165] Example 31. The method of Example 27, wherein the distance between the surgical instrument and the target trajectory axis is calculated based on the distance from the tip of the surgical instrument to at least one selected from the group consisting of: a line segment connecting the target object and the entry point of the target trajectory axis, a line connecting the target object and the entry point of the target trajectory axis, the target object, and the entry point of the trajectory.

[0166] Example 32. The method of Example 30, wherein highlighting the target trajectory axis includes hiding all other trajectories except the one with the minimum distance from the user.

[0167] Example 33. The method described in Example 30 displays all other tracks except the one with the smallest distance with increased transparency.

[0168] Example 34. The method of Example 27, wherein decomposing the distance vector into two axis-aligned deviation vectors is relative to two eigenvectors derived from:

[0169] The two main patient axes that form the largest angle with the target trajectory axis out of the three main patient axes, and

[0170] The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

[0171] Example 35. The method of Example 27 further includes scaling the augmented reality position alignment visualization;

[0172] The scaling of the distance vector decomposition is limited to the absolute maximum visible length and the maximum visible length relative to the field of view of the head-mounted display.

[0173] Example 36. The method of Example 27, wherein the decomposition of the two deviation angles is relative to two eigenvectors derived from:

[0174] The two main patient axes that form the largest angle with the target trajectory axis out of the three main patient axes, and

[0175] The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

[0176] Example 37. The method described in Example 30, wherein the visualization of the deviation angle includes:

[0177] Augmented reality visualization of the following positional corrections: the instrument axis, the target trajectory axis, and the arc connecting the proximal ends of the instrument axis and the target trajectory axis.

[0178] Example 38. The method described in Example 30, wherein the visualization of the deviation angle includes:

[0179] The position correction of the instrument's axis is visualized using augmented reality.

[0180] Example 39. The method of Example 29 further includes displaying: a first augmented reality text label describing the distance from the tip of the surgical instrument to the target trajectory axis, and a second augmented reality text label positioned in or near the angle visualization to describe the angle of deviation between the instrument axis and the target trajectory axis in degrees.

[0181] Example 40. The method of Example 27 further includes displaying an augmented reality visualization of a target object to be placed or removed at a target point on the target trajectory axis, wherein the augmented reality visualization of the target object is positioned at the target location or the current location of the surgical instrument.

[0182] Example 41. The method of Example 27 further includes the step of augmented reality visualization of a target object to be placed or removed at a target point on the target trajectory axis.

[0183] Example 42. The method of Example 28, wherein different colors for augmented reality visualization are selected from the base color of the trajectory by arranging alternating colors equidistantly in a chromaticity space around a white point and selecting their dominant wavelengths by extrapolation; and / or

[0184] The colors mentioned are selected from the following: high-brightness complementary colors selected from the group consisting of yellow, pink, green and cyan.

[0185] Example 43. The method of Example 27 further includes the step of displaying an augmented reality visualization of patient image data slices, which includes the following steps:

[0186] The surgeon selects the augmented reality visualization location as one of the following: a fixed box in space, a floating box that follows head movement, in situ at the location of a slice of patient image data, or a user-defined fixed spatial vector offset from the patient's location;

[0187] Color mapping from patient image data color information to augmented reality visualization color information including alpha transparency target range; and

[0188] User interaction, through which users can use any of the following methods—voice, mouse, keyboard, gaze, or surgical instruments—to select and reposition augmented reality visualization locations and select slices of patient image data.

[0189] Example 44. The method of Example 27 further includes displaying a region of interest indicator, the region of interest indicator comprising:

[0190] Augmented reality text labels or augmented reality boundary visualizations delineating the region of interest are shown near the region of interest; and

[0191] A color mapping method that highlights augmented reality text labels and augmented reality boundary visualizations to the user based on the distance of the navigated device to the region of interest.

[0192] Example 45. A surgical navigation system, a patient tracker capable of being tracked by the surgical navigation system, and surgical instruments capable of being tracked by the surgical navigation system, the surgical instrument guidance system comprising:

[0193] monitor;

[0194] A controller configured to display on the display an augmented reality positioning alignment visualization as two axis-aligned deviation vectors, the two axis-aligned deviation vectors comprising a decomposition of the distance vector from the target point on the target trajectory axis to the surgical instrument; and / or

[0195] The controller is further configured to display an augmented reality angular alignment visualization on a display, including a deviation angle, which represents the angle between a first angular vector representing the axis of the instrument and a second angular vector representing the axis of the target trajectory.

Claims

1. A method for displaying surgical navigation information using a head-mounted display system, considering a surgical plan including a target trajectory axis, the head-mounted display system comprising a surgical navigation system and surgical instruments having a tip and at least partially defining an instrument axis, the method comprising: An augmented reality position alignment visualization is displayed on a head-mounted display, comprising two axis-aligned deviation vectors, namely a first deviation vector and a second deviation vector, wherein the first and second deviation vectors represent the decomposition of distance vectors from a point on the target trajectory axis to the tip of the surgical instrument; and The augmented reality positioning visualization displayed on the head-mounted display is updated to indicate the relative position of the surgical instruments with respect to the target trajectory axis from the perspective of the head-mounted display. The method includes displaying an augmented reality angle alignment visualization including a deviation angle, the deviation angle being the angle between a first angle vector representing the axis of the surgical instrument and a second angle vector representing the axis of the target trajectory, wherein the lengths of the first and second lines representing the deviation angle are scaled to allow the user to see small deviations.

2. The method according to claim 1, wherein, The decomposition of the distance vector into two axis-aligned offset vectors is relative to two eigenvectors derived from the following: Two of the three main patient axes derived from the patient's orientation form the largest angle with the target trajectory axis.

3. The method according to claim 1, wherein, The decomposition of the distance vector into two axis-aligned offset vectors is relative to two eigenvectors derived from the following: The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

4. The method of claim 1, further comprising updating the augmented reality angle alignment to indicate, from the angle of the head-mounted display, the angle of the axis of the surgical instrument relative to the target trajectory axis.

5. The method of claim 1 or 4, comprising using a first color to display the augmented reality position alignment visualization or the augmented reality angle alignment visualization, and using a different color to display the other of the augmented reality position alignment visualization or the augmented reality angle alignment visualization.

6. The method according to claim 1 or 4, further comprising highlighting the augmented reality position alignment visualization and / or the augmented reality angle alignment visualization to the user based on the distance from the surgical instrument to the target trajectory axis.

7. The method according to claim 6, wherein, The distance between the surgical instrument and the target trajectory axis is calculated based on the distance from the tip of the surgical instrument to at least one of the following: a line segment connecting the target object and the entry point of the target trajectory axis, a line connecting the target object and the entry point of the target trajectory axis, and the entry point of the target object or the trajectory.

8. A method of aligning surgical instruments using a surgical navigation system, the surgical navigation system comprising a tracking unit configured to track the position of a head-mounted display, a patient tracker, and a surgical instrument tracker coupled to the surgical instruments, the surgical instruments having a tip and defining an instrument axis, wherein the surgical navigation system is configured to plan a target trajectory axis based on patient data registered to the patient tracker, the method comprising the steps of: The augmented reality positioning alignment is visualized on the head-mounted display as two axis-aligned deviation vectors, which include a decomposition of the distance vector from a point on the target trajectory axis to the tip of the surgical instrument. An augmented reality angular alignment visualization, including a deviation angle, is displayed on the head-mounted display. The deviation angle is shown as the angle between a first angular vector representing the axis of the surgical instrument and a second angular vector representing the axis of the target trajectory. The lengths of the first and second lines representing the deviation angle are scaled to allow the user to see small deviations. The tracking unit continuously updates the augmented reality position alignment visualization and / or augmented reality angle alignment visualization displayed on the head-mounted display to indicate the position of the surgical instrument axis relative to the target trajectory axis from the perspective of the head-mounted display.

9. The method according to claim 8, wherein, The target trajectory axis is defined by a line connecting the target object and the entry point.

10. The method of claim 8 or 9, further comprising augmented reality visualization of the instrument axis of the surgical instrument on the head-mounted display.

11. The method of claim 8, further comprising using different colors to display the augmented reality position alignment visualization and / or the augmented reality angle alignment visualization, so that a user can distinguish the corresponding alignment visualization; and Based on the distance from the surgical instrument to the target trajectory axis, the augmented reality position alignment visualization and / or augmented reality angle alignment visualization are highlighted to the user.

12. The method according to claim 8, wherein, The distance between the surgical instrument and the target trajectory axis is calculated based on the distance from the tip of the surgical instrument to at least one of the following: a line segment connecting the target object and the entry point of the target trajectory axis, a line connecting the target object and the entry point of the target trajectory axis, the target object, and the entry point of the trajectory.

13. The method according to claim 11, wherein, The highlighting of the target trajectory axis includes hiding all other trajectories from the user except for the one with the minimum distance.

14. The method of claim 11, wherein all other tracks except the one with the minimum distance are displayed with increased transparency.

15. The method according to claim 8, wherein, The decomposition of the distance vector into two axis-aligned offset vectors is relative to two eigenvectors derived from the following: The two main patient axes that form the largest angle with the target trajectory axis out of the three main patient axes, and The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

16. The method of claim 8, further comprising scaling the augmented reality position alignment visualization; in, The scaling of the distance vector decomposition is limited to the absolute maximum visible length and the maximum visible length relative to the field of view of the head-mounted display.

17. The method according to claim 8, wherein, The augmented reality angle alignment visualization includes decomposing the angle between a first angle vector representing the instrument axis and a second angle vector representing the target trajectory axis into two deviation angles. The decomposition of the two deviation angles is relative to the two eigenvectors derived below: The two main patient axes that form the largest angle with the target trajectory axis out of the three main patient axes, and The collimation axis is projected onto the following plane: the plane is perpendicular to the target trajectory axis and attached to the point on the target trajectory axis closest to the tip of the surgical instrument, and the perpendicular vector in the same plane as the projected collimation axis.

18. The method according to claim 8, wherein, The visualization of the deviation angle includes: Augmented reality visualization of the following positional corrections: the instrument axis, the target trajectory axis, and the arc connecting the proximal ends of the instrument axis and the target trajectory axis.

19. The method according to claim 8, wherein, The visualization of the deviation angle includes: The position correction of the instrument's axis is visualized using augmented reality.

20. The method of claim 8, further comprising displaying: a first augmented reality text label describing the distance from the tip of the surgical instrument to the target trajectory axis; and a second augmented reality text label positioned in or near an angle visualization to describe, in degrees, the angle of deviation between the instrument axis and the target trajectory axis.

21. The method of claim 8, further comprising displaying an augmented reality visualization of a target object to be placed or removed at a target point on the target trajectory axis, wherein, The augmented reality visualization of the target object is located at the target point or the current position of the surgical instrument.

22. The method of claim 8, further comprising the step of augmented reality visualization of a target object to be placed or removed at a target point on the target trajectory axis.

23. The method according to claim 8, wherein, Different colors for augmented reality visualization are selected from the base color of the trajectory by arranging alternating colors equidistantly in the chromaticity space around the white point and selecting their dominant wavelengths by extrapolation. The colors mentioned are selected from the following: high-brightness complementary colors selected from the group consisting of yellow, pink, green and cyan.

24. The method of claim 8, further comprising the step of displaying an augmented reality visualization of slices of patient image data, the step comprising the following steps: The surgeon selects the augmented reality visualization location as one of the following: a fixed box in space, a floating box that follows head movement, in situ at the location of a slice of patient image data, or a user-defined fixed spatial vector offset from the patient's location; Color mapping from patient image data color information to augmented reality visualization color information that includes a target range of alpha transparency; and User interaction, through which users can use any of the following methods—voice, mouse, keyboard, gaze, or surgical instruments—to select and reposition augmented reality visualization locations and select slices of patient image data.

25. The method of claim 8, further comprising displaying a region of interest indicator, the region of interest indicator comprising: Augmented reality text labels or augmented reality boundary visualizations delineating the region of interest are shown near the region of interest. and A color mapping method that highlights augmented reality text labels and augmented reality boundary visualizations to the user based on the distance of the navigated device to the region of interest.

Images