Visualization system and method for ENT procedures

Abstract

The visualization system can be implemented with an image-guided surgical navigation system for use during ENT procedures. Correlated datasets, such as surgical paths and preoperative image slices, can be correlated with the navigation system's coordinate system for reference during the procedure. The surgeon can wear a head-mounted display ("HMD") with a transparent screen through which the patient's anatomy is viewed. The orientation and distance of the head-mounted display relative to the patient's anatomy can be determined using magnetic position tracking, image analysis of images captured by the HMD's camera, or both. The correlated dataset can then be transformed based on the relative distance and orientation and displayed through the transparent screen to overlay the directly viewed patient anatomy. Some implementations can include an optical fiducial integrated with the patient tracking assembly to track the patient and help determine the HMD's viewpoint.

Description

[Technical field]

[0001] (Priority) This application claims priority to U.S. Provisional Patent Application No. 62 / 925,441, filed October 24, 2019, entitled “Visualization System and Method for ENT Procedures.” [Background technology]

[0002] Image-guided surgical navigation systems are used during surgical procedures to provide additional information and visual perspectives of the surgical site or other patient anatomy. This may include displaying pre-operative images (e.g., CT images) of the surgical site from various viewpoints, and may also include overlaying markers on such displayed images to indicate static information, such as the planned path that a surgical instrument will follow during the procedure, as well as dynamic information, such as the current location of the distal tip of the surgical instrument. Such information may be used to improve the accuracy and safety with which surgical instruments are navigated to specific locations within a patient's body.

[0003] Given the breadth of additional information available during image-guided surgery, a surgeon or other practitioner may sometimes become spatially disoriented while switching between views of a surgical site or between viewing a direct view of the patient's anatomy and observing a simulated image of the patient's anatomy. For example, in some cases, a surgeon may occasionally view the patient's head directly while viewing a CT slice of an axial view of the patient's head on the display of an image-guided surgical navigation system. In such cases, the surgeon may become disoriented between their direct perception and the axial view and be unable to determine spatial and directional correspondence, resulting in erroneous movement of surgical instruments within the patient's head.

[0004] While several systems and methods have been made and used for ENT procedures, it is believed that no one prior to the inventors of the present invention has made or used the invention set forth in the appended claims. [Brief description of the drawings]

[0005] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description of the invention given above and the detailed description of the embodiments given below, serve to explain the principles of the invention. [Figure 1] 1 shows a schematic diagram of an exemplary surgical navigation system in use with a patient seated in an exemplary medical procedure chair. [Figure 2A] 2 shows a schematic diagram of an exemplary patient tracking device placed on the patient of FIG. 1; [Figure 2B] 2B illustrates a perspective view of an exemplary patient tracking device as shown in FIG. 2A. [Diagram 3] 2 shows a schematic diagram of an exemplary viewing device that can be used with the surgical navigation system of FIG. 1. [Figure 4] 4 illustrates a perspective view of one exemplary observation device shown in FIG. 3. [Diagram 5] 1 shows a flowchart of an exemplary set of steps that may be performed using the surgical navigation system of FIG. 1 to provide visualization of an ENT procedure. [Figure 6A] 1 illustrates an exemplary interface showing a frontal view of a patient's face that may be provided during visualization of an ENT procedure. [Figure 6B] 1 illustrates an exemplary interface showing a side view of a patient's face that may be provided during visualization of an ENT procedure. [Figure 6C] 1 illustrates an exemplary interface showing a perspective view of a patient's face that may be provided during visualization of an ENT procedure. [Figure 7A] 2 illustrates an exemplary computed tomography (CT) image that can be used with the surgical navigation system of FIG. 1. [Figure 7B]1 illustrates an exemplary interface showing a frontal view of a patient's face with a CT image overlaid, which may be provided during visualization of an ENT procedure. [Figure 7C] 1 illustrates an exemplary interface showing a side view of a patient's face with a CT image overlaid, which may be provided during visualization of an ENT procedure. [Figure 7D] 1 illustrates an exemplary interface showing a perspective view of a patient's face with a CT image overlaid, which may be provided during visualization of an ENT procedure. [Figure 8] FIG. 2 shows a schematic diagram illustrating the relative positions and orientations of the cameras and the patient's head within a three-dimensional coordinate system.

[0006] The drawings are not intended to be limiting in any manner, and it is contemplated that various embodiments of the invention may be embodied in other various ways, including those not necessarily depicted in the drawings. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention. It will be understood, however, that the invention is not limited to the precise arrangements shown. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The following description of specific examples of the present invention should not be used to limit the scope of the present invention. Other examples, features, aspects, embodiments, and advantages of the present invention will become apparent to those skilled in the art from the following description, which illustrates, by way of example, one of the best modes contemplated for carrying out the present invention. As will be understood, the present invention is capable of other different and obvious aspects, none of which departs from the present invention. Thus, the drawings and description should be regarded as illustrative in nature, and not restrictive.

[0008] For clarity of this disclosure, the terms "proximal" and "distal" are defined herein relative to a surgeon or other operator gripping a surgical instrument having a distal surgical end effector. The term "proximal" refers to the location of an element disposed closer to the surgeon, and the term "distal" refers to the location of an element disposed closer to the surgical end effector of the surgical instrument and farther from the surgeon. Also, to the extent that spatial terms such as "upper", "lower", "vertical", "horizontal", etc. are used herein with reference to the drawings, it will be understood that such terms are used for illustrative descriptive purposes only and are not intended to be limiting or absolute. In that regard, it will be understood that surgical instruments such as those disclosed herein may be used in a variety of orientations and positions that are not limited to those shown and described herein.

[0009] As used herein, the term "about" or "approximately" of any numerical value or range is intended to indicate a suitable dimensional tolerance that enables a portion or collection of components to function for its intended purpose as described herein.

[0010] I. Exemplary Image-Guided Surgical Navigation System When performing a medical procedure within the head (H) of a patient (P), it may be desirable to have information regarding the location of an instrument within the head (H) of the patient (P), especially when the instrument is in a location where it is difficult or impossible to obtain an endoscopic view of the working element of the instrument within the head (H) of the patient (P). FIG. 1 illustrates an exemplary IGS navigation system (10) that enables an ENT procedure to be performed using image guidance. In addition to or instead of having the components and operability described herein, the IGS navigation system (10) may be constructed and operative in accordance with at least a portion of the teachings of the following documents: No. 7,720,521, entitled "Methods and Devices for Performing Procedures within the Ear, Nose, Throat and Paranasal Sinuses," issued May 18, 2010, the disclosure of which is incorporated herein by reference; and U.S. Patent Application Publication No. 2014 / 0364725, entitled "Systems and Methods for Performing Image Guided Procedures within the Ear, Nose, Throat and Paranasal Sinuses," published December 11, 2014, the disclosure of which is incorporated herein by reference.

[0011] The IGS navigation system (10) of this embodiment includes a magnetic field generator assembly (20), which includes a set of magnetic field generators (24) integrated into a horseshoe-shaped frame (22). The magnetic field generators (24) are operable to generate alternating magnetic fields of different frequencies around the head (H) of a patient (P) to generate a tracked area to which the IGS navigation system (10) associates a coordinate system. In this embodiment, a navigation guidewire (40) is inserted into the head (H) of the patient (P). The navigation guidewire (40) may be a standalone device or may be positioned at an end effector of a medical instrument, such as a surgical cutting or dilating instrument, or elsewhere. In this embodiment, the frame (22) is mounted to a chair (30), and the patient (P) sits in the chair (30) such that the frame (22) is positioned adjacent to the head (H) of the patient (P). By way of example only, the chair (30) and / or the magnetic field generator assembly (20) may be constructed and operable in accordance with at least a portion of the teachings of U.S. Pat. No. 10,561,370, entitled "Apparatus to Secure Field Generating Device to Chair," issued Feb. 18, 2020, the disclosure of which is incorporated herein by reference.

[0012] The IGS navigation system (10) of the present embodiment further includes a processor (12) that controls the magnetic field generator (24) and other elements of the IGS navigation system (10). For example, the processor (12) is operable to drive the magnetic field generator (24) to generate an alternating electromagnetic field and to process signals from the navigation guidewire (40) to determine the location of a sensor in the navigation guidewire (40) within the head (H) of the patient (P). The processor (12) includes a processing unit (e.g., a set of electronic circuitry configured to evaluate and execute software instructions using combinatorial logic or other similar circuitry) in communication with one or more memories. The processor (12) of the present embodiment is mounted in a console (18) that includes a motion control (14) that includes a keypad and / or a pointing device such as a mouse or trackball. A surgeon uses the motion control (14) to interact with the processor (12) while performing a surgical procedure.

[0013] The navigation guidewire 40 includes a sensor (not shown) that responds to positioning within the alternating electromagnetic field generated by the magnetic field generator 24. A coupling unit 42 is secured to the proximal end of the navigation guidewire 40 and is configured to provide communication of data and other signals between the console 18 and the navigation guidewire 40. The coupling unit 42 may provide for wired or wireless communication of the data and other signals.

[0014] In this example, the sensor of the navigation guidewire 40 includes at least one coil at the distal end of the navigation guidewire 40. When such a coil is placed in an alternating electromagnetic field generated by the magnetic field generator 24, the alternating magnetic field may generate an electric current in the coil, which may be further transmitted along a conductive path(s) in the navigation guidewire 40 to the processor 12 via the coupling unit 42. This phenomenon allows the IGS navigation system 10 to determine the location of the distal end of the navigation guidewire 40 or other medical instrument (e.g., dilation instrument, surgical cutting instrument, etc.) in three-dimensional space (i.e., in the head (H) of the patient (P)). To accomplish this, the processor 12 executes an algorithm to calculate the position coordinates of the distal end of the navigation guidewire 40 from the position-related signals of the coil(s) in the navigation guidewire 40. In this example, the position sensor is located within the guidewire (40), however, such position sensors may be integrated into a variety of other types of instruments, such as dilation catheters, guide catheters, guide rails, suction instruments, pointer instruments, alignment probes, curettes, patient tracking devices, and other instruments, including those described in more detail below.

[0015] The processor (12) calibrates and operates the IGS navigation system (10) using software stored in the memory of the processor (12). Such operations include driving the magnetic field generator (24), processing data from the navigation guidewire (40), processing data from the motion control (14), and driving the display screen (16). In some implementations, operations may also include monitoring and enforcing one or more safety features or functions of the IGS navigation system (10). The processor (12) is further operable to provide real-time animation via the display screen (16) showing the position of the distal end of the navigation guidewire (40) relative to a video camera image of the patient's head (H), a CT scan image of the patient's head (H), and / or a computer-generated three-dimensional model of the anatomical structures within and adjacent to the patient's nasal cavity. The display screen (16) may display such images simultaneously and / or overlaid on one another during a surgical procedure. Such display images may also include graphical representations of instruments inserted into the patient's head (H), such as a navigational guidewire (40), so that the operator can see a virtual rendering of the instrument in its actual location in real time. By way of example only, the display screen (16) may provide images in accordance with at least some of the teachings of U.S. Pat. No. 10,463,242, entitled "Guidewire Navigation for Sinuplasty," issued Nov. 5, 2019, the disclosure of which is incorporated herein by reference. If the operator is also using an endoscope, endoscopic images may also be displayed on the display screen (16).

[0016] The images provided by the display screen 16 may help guide the operator in steering and otherwise maneuvering the instrument within the patient's head H if the instrument incorporates a navigation guidewire 40. It should also be understood that other components of surgical instruments and other types of surgical instruments may incorporate sensors, such as the sensors of the navigation guidewire 40, as described below.

[0017] In some implementations, the IGS navigation system (10) may include a patient tracking assembly (50), which may be located on the patient's head (H) or another suitable portion of the patient (P) as shown in FIG. 2A. By tracking the head (H) separately from the guidewire (40), but within the same coordinate system, the position and orientation of the guidewire (40) and head (H) relative to each other may be determined during the procedure. This may be advantageous when the head (H) is registered to determine its initial position in the coordinate system, but then moves during the procedure. By tracking the head (H) independently, rotation or other motion may be detected, and the initial registration may be updated to account for the new position of the head (H). In this manner, any image-guided navigation features used during the procedure, such as the display of CT image slices showing the position of the guidewire (40) within the head (H) with overlaid markers, may be automatically updated in response to such motion. Implementations of the patient tracking assembly (50) may be configured and operable with the IGS navigation system (10) in accordance with any of the teachings of U.S. Patent Application Publication No. 2019 / 0183582, filed December 14, 2017, entitled “Mounted Patient Tracking Component for Surgical Navigation System,” the disclosure of which is incorporated herein by reference.

[0018] As an example of a patient tracking assembly (50), Figure 2B illustrates a patient tracking assembly (200) that may be readily incorporated into the IGS navigation system (10). The patient tracking assembly (200) includes a disposable portion (210) and a reusable portion (250). The disposable portion (210) is configured to attach to a patient's head (H) or another suitable portion of the patient (e.g., using a flexible adhesive pad) and selectively couple with a coupling block (220) of the reusable portion (250) such that the reusable portion (250) is secured relative to the head or another portion of the patient (P) during exemplary use, and the reusable portion (250) is configured to communicate with the processor (12) to track a portion of the patient's head (H).

[0019] The reusable portion (250) includes a cable (252) and a sensor (255) extending proximally from a coupling assembly (254). The coupling assembly (254) is adapted to couple the reusable portion (250) to the disposable portion (210) during use. When properly coupled to the disposable portion (210), the sensor (255) may be utilized with the processor (12) to determine the location of the tracked anatomical structure, such that the processor (12) may accurately indicate the location of the navigation guidewire (40) (or any other suitable instrument) relative to the anatomical structure of the patient (P) during exemplary use. The cable (252) is configured to provide a conduit for communication between the sensor (255) and the processor (12) during exemplary use. Thus, the cable (252) may directly connect the sensor (255) for wired communication with the processor (110) via the cable (252). Alternatively, the cable (252) may connect the sensor (255) to a wireless communication device that communicates wirelessly with the processor (12), similar to how the connection unit (42) establishes wireless communication between the navigation guidewire (40) and the processor (12).

[0020] II. An Exemplary Visualization System for an ENT Procedure 3 illustrates a visualization system 60 that may be implemented with the IGS navigation system 10 to provide additional information and image navigation views during a surgical procedure. In particular, the visualization system 60 may provide a navigation view that displays information in an augmented reality view that is overlaid on images of the patient captured during the procedure in near real time. This may advantageously reduce the frequency and need for the surgeon to look at the display screen 16 or another display instead of looking at the patient during the procedure, and may also help the surgeon spatially orient themselves between the physical world and the image navigation view provided by the IGS navigation system 10.

[0021] The visualization system (60) may be implemented in conjunction with the IGS navigation system (10) by configuring one or more of a head mounted display ("HMD") (100), a hand held display ("HHD") (101), or another similar device to communicate with the IGS navigation system (10) during an image-guided surgical procedure, as described in more detail below. FIG. 4 shows a perspective view of the HMD (100) including a frame (102) that may be worn on the face of a surgeon or other user involved in an image-guided surgical procedure. Although a case (108) is shown attached to the frame (102), the case (108) may be worn or attached elsewhere and coupled to devices attached to the frame (102) via a wired or wireless connection to those devices. The case (108) includes a viewpoint sensor (110) operable to provide spatial information indicative of a viewpoint, location, or both of the HMD (100). The viewpoint sensor (110) may include, for example, one or more of a gyroscope, an accelerometer, or a position sensor (such as a sensor on the guidewire (40)). Information from the viewpoint sensor (108) may be usable to determine one or more aspects of the visual viewpoint of a camera (106) mounted on the frame (102), which may include determining the orientation of the optical axis of the camera (106) (e.g., rotational coordinates about one or more axes), the location of the camera (106) relative to the coordinate system of the IGS navigation system (10) (e.g., positional coordinates about one or more axes), or both, as described in more detail below.

[0022] The case (108) also includes a communication device (112), which may be a wired or wireless transceiver capable of communicating with the processor (12) or other devices, and a processor and memory (114) configured to process and store data and perform functions related to the functionality of the HMD (100). A power source (116), which may be a battery or a connection to an external power source, is also included and is configured to provide power to the processor and memory (114), the communication device (112), the camera (106), the display (104), and other components of the HMD (100).

[0023] When worn, the frame (102) positions the camera (106) mounted on the frame (102) and / or the case (108) such that the optical axis of the camera is substantially parallel to the optical axis of the wearer's eye when the wearer looks straight ahead. As used herein, the term "neutral optical axis" may refer to the optical axis of the wearer's eye when the wearer of the HMD (100) looks substantially straight ahead (e.g., when the pupil of the eye is substantially centered both vertically and horizontally within the eye socket). In this manner, the camera (106) captures an image having a field of view similar to the field of view of the wearer of the HMD (100). As an example, the camera (106) may capture an image including part or all of the field of view of the wearer's right eye, which the camera (106) is closest positioned to when the wearer looks straight ahead. The camera (106) may be capable of capturing images, video, and audio that may be stored by the processor and memory (114) and transmitted to the processor (12) via the communication device (112) or transmitted to another device for display or presentation, as would be apparent to one of ordinary skill in the art in light of this disclosure. Image data captured by the camera (106) may also be used for computer vision and other analysis, which may include, for example, identifying objects or other visual characteristics in the captured image. Such analysis may be performed by the processor (114), the processor (12), or both, or may be performed using various cloud computing or edge processing techniques, as would be apparent to one of ordinary skill in the art in light of this disclosure.

[0024] The display (104) is also attached to the frame (102) and / or the case (108) and positioned to be within the field of view of a wearer of the HMD (100). In some implementations, the display (104) is at least partially translucent, if not transparent, and operable by the processor (114) to render an image that appears to be overlaid on the field of view of the wearer. As one example, the display (104) may display an image captured by the camera (106), which may block some or all of the wearer's view from the proximal eye, but otherwise appears similar to the wearer's normal view for that eye. As another example, image analysis may be performed on the captured image to identify an object of interest within that image (e.g., in the context of surgical navigation, a human face, or another part of the human anatomy), and the display (104) may be operated to render a visual marker that appears to the wearer to be overlaid on a direct view of the identified object (e.g., seen through a transparent portion of the display (104)). In some implementations described above, optical markers or other reference markers may be placed on an object of interest to provide image data having objects that are easily identifiable by their reflectivity, shape, or other visual characteristics such that the optical markers placed on a human face can be identified, rather than the human face itself.

[0025] As yet another example, the display (104) may be operated to render visual markings to overlay a direct view of the wearer's field of view instead of, or in addition to, image analysis or machine vision, based on other inputs. This may include, for example, rendering visual markings based on information from the viewpoint sensor (110), the processor (12), the patient tracking device (50) (e.g., via communication with the processor (12)), and other devices. This may include rendering visual markings that provide information associated with the rotational viewpoint of the HMD (100) (e.g., based on the gyroscopic characteristics of the viewpoint sensor (100)). As another example, this may include rendering visual markings to overlay a surgical instrument (e.g., a guidewire (40)) based on tracking information associated with the surgical instrument and the HMD (100). In other words, if the processor (12) can track and determine the relative positions of the surgical instrument and the HMD (100), and the orientation of the HMD (100) can be determined (e.g., using the gaze sensor (110)), the position and scale of the tracked objects relative to each other can be determined and generated as markings rendered via the display (104).

[0026] As would be apparent to one of ordinary skill in the art in light of this disclosure, some or all of the above features may be implemented using other displays besides the display (104). For example, in some implementations, a separate display (e.g., the display screen (16) or a wall-mounted display visible from across the room) may be configured to receive and display the captured images from the camera (106) and any markings, renderings, or other overlay data that may be added. In such implementations, the display (104) may render only the overlay images, but the separate display may render a combination of the images captured by the camera (106) and any corresponding overlay images. This may allow other participants in the procedure to view additional navigation views in addition to the wearer of the HMD (100). In some such implementations, the HMD (100) may not include a display (104), and the combination of the captured images and the rendered overlays may be viewable on the display screen (16) or another display positioned near the patient.

[0027] The HHD (101) may share some or all of the capabilities of the HMD (100), and may be, for example, a smartphone, tablet, proprietary device, or other handheld computing device with the ability to process and store data, perform functions, send and receive data, capture images, provide spatial information (e.g., orientation, location, or both), etc. The display of the HHD (101) may generally be an LED or LCD display, and therefore may not be able to overlay rendered markings on transparent surfaces where objects are directly viewed, as is possible with the display (104). In some implementations, the HMD (100) or the HHD (101) may be modified to include additional capabilities. For example, some implementations of the HMD (100) may not include the ability to determine self-location relative to the coordinate system of the IGS navigation system (10). In such cases, the sensor may be mounted (e.g., externally or internally) on the HMD (100) to enable the HMD (100) to interact with and be tracked by the IGS navigation system (10), as well as the guidewire (40) and patient tracking assembly (50). Thus, the capabilities of the gaze sensor (110) may include both capabilities that are present in the HMD (100) by default, as well as capabilities that may be added later, whether they are within the case (108) or mounted on the outside of the HMD (100).

[0028] In some implementations, information indicative of the orientation and location of the HMD (100) may be available from multiple sources. As an example, the gaze sensor (110) may include a gyroscope feature capable of determining a rotational orientation, may include or incorporate a three-axis sensor capable of being tracked by the IGS navigation system (10) to determine a rotational orientation, and may be configured to identify optical fiducials present in images captured by the camera (106). In such examples, the processing component (e.g., the processor (114), the processor (12), or another processor) may be configured to determine the orientation or location in various ways by balancing performance and accuracy, as would be apparent to one of ordinary skill in the art in light of this disclosure. For example, some implementations may determine orientation or location based solely on gyroscope information from the HMD (100) itself or tracking information from the IGS navigation system (10) in order to emphasize performance, while other implementations may use a combination of gyroscope, magnetic tracking, and image analysis information in order to achieve greater accuracy at some potential cost of performance (e.g., delay, if any, between the orientation or location of the HMD (100) changing and the new orientation or location being determined).

[0029] The visualization system (60) allows additional input and information to be collected and used during surgical navigation to provide additional navigational views and other feedback to a user of the HMD (100) or HHD (101), or an observer of another display configured to display the images from the camera (106) and any corresponding overlay renderings. Without the visualization system (60), there is a wide range of useful information available to the IGS navigation system (10), but most of it is limited to being used or displayed in conjunction with pre-operative images (e.g., 3D patient models generated from pre-operative imaging, CT, MRI, or ultrasound image sets).

[0030] As one example, the IGS navigation system (10) may allow a surgeon to view a set of CT images of a patient prior to a procedure and envision a surgical plan or path along which the surgeon will navigate one or more surgical instruments during the procedure. Then, during the procedure, the surgical path is overlaid on the CT image set and displayed via the display screen (16), allowing the surgeon to switch between a limited number of views (e.g., axial, coronal, sagittal) and slices as needed. As another example, during the procedure, a surgical instrument (e.g., a guidewire (40)) may also be tracked and displayed on the CT image set as well. When displayed together, the surgeon may find it beneficial to view CT images that correlate the tracked positions of the surgical instruments and the planned surgical path relative to each other.

[0031] Although useful, the above features may distract the surgeon from the patient, since the surgeon may need to look away from the patient to view a nearby display device. Also, the surgeon may be disoriented when switching between axial, coronal, and sagittal views, since the surgeon's actual location relative to the patient has not changed. For example, the surgeon may be looking at an axial plane CT image of the patient's head, and when he or she turns his or her gaze back to the patient, he or she may be looking at the sagittal plane of the patient's head. In order to use the information displayed in the CT image, such as the location and orientation of the surgical instruments, the surgical path, and nearby anatomical cavities and structures, the surgeon must first mentally convert from the axial plane to the sagittal plane, or relate the spatial understanding of the axial plane to the sagittal plane, in order to know the direction to navigate the surgical instruments. This process may be mentally taxing, may consume valuable time during the procedure, and may result in incorrect navigation of the surgical instruments.

[0032] To address this, the visualization system (60) provides a framework for relating coordinate systems and associated information to the physical world perceived by the wearer of the HMD (100). Such relevant information may include, for example, CT images, configured surgical paths, real-time surgical instrument tracking, real-time patient tracking, configured objectives indicating areas to be investigated or avoided, and other similar information that may be correlated to the coordinate system of the IGS navigation, which may be collectively referred to herein as correlated datasets.

[0033] Once correlated, these correlated data sets may then be displayed via the display (104) as they are available when looking directly at the patient, instead of being displayed only on the display screen (16) or another nearby display. In addition to reducing the need to refer to an external display, such implementations allow the wearer of the HMD (100) to browse or navigate the correlated data sets by changing the viewpoint relative to the patient, instead of using a mouse or keyboard to step through image slices, switch between viewable planes, or rotate a 3D model. For example, in the case of tracked surgical instrument locations and surgical paths, the surgeon may be able to use a mouse or keyboard interface to step through CT image slices and see rendered overlays of the instrument locations and surgical paths within the patient's head from different perspectives by moving and viewing from different angles, rather than being limited to moving between CT image planes.

[0034] As an example implementation of the above, FIG. 5 illustrates a set of steps (300) that may be performed using a visualization system (60) to render and display the correlation data set via an HMD (100) or another viewing device, and FIGS. 6A-6C and 7A-7C illustrate example interfaces that may be displayed or viewed using the visualization system (60). After positioning the patient for the procedure, the patient may be aligned (block 302) and, in an implementation that includes a patient tracking assembly (50), tracked within the IGS navigation coordinate system. Aligning the patient may include using an alignment probe or other device to provide a coordinate system with multiple locations that correspond to the patient's anatomy, may include positioning, calibrating, and using the patient tracking assembly (50), or may include both, or may include other alignment techniques. Alignment (block 302) may also include aligning and tracking other devices and instruments, such as guidewires (40) and other trackable surgical instruments. Aligning (block 302) may also include aligning and tracking the HMD (100), such that the HMD (100) can be positionally tracked by the IGS navigation system (10).

[0035] The IGS navigation system (10) may also receive one or more correlated datasets associated with the patient and procedure (block 304), which may include pre-operative images of the patient's anatomy (e.g., CT, MRI, and ultrasound image sets), pre-configured surgical plans and surgical paths, and other pre-configured or pre-operatively captured or generated datasets that may be associated with the IGS navigation coordinate system. The received (block 304) correlated datasets may also include data captured in real time during the procedure and then associated with the coordinate system, such as position tracking data indicating the location of the guidewire (40) and other tracked surgical instruments, and position tracking data indicating the location of the HMD (100).

[0036] If an HMD (100) or another device (e.g., HHD (102)) is used with the visualization system (60), images may be captured by the camera (106) (block 306) as described. In some implementations, images are always captured based on the configured frame rate of the camera (106) (block 306), so that each subsequent image may vary slightly from the previous image based on the movement of the wearer of the HMD (100). The captured (block 306) images may be stored by the processor and memory (114) and may be transmitted to the processor (12) or another device.

[0037] As images are captured (block 306), the visualization system (60) may iteratively determine the orientation of the HMD (100) relative to the anatomical structures (block 308) and iteratively determine the distance of the HMD (100) relative to the anatomical structures (block 310). These determinations (blocks 308, 310) may be made sequentially and independently, or on a one-to-one basis for each image captured (block 306) (e.g., if the camera (106) captures 30 images or frames per second, then the visualization system (60) would determine the orientation (block 308) and distance (block 310) 30 times per second, once for each image), or in some other correspondence (e.g., the visualization system may be configured to determine the orientation (block 308) and distance (block 310) once for every three images captured (block 306)).

[0038] The orientation (308) and distance (310) may be determined in a variety of ways, as previously described. For example, in some implementations, the HMD (100) and the patient's head (H) may each be positionally and orientationally tracked by an IGS navigation system (10), and the distance and orientation may be determined using the IGS navigation coordinate system. In some implementations, the orientation and / or distance may be determined using a gaze sensor (110) in the HMD (100).

[0039] In some implementations, the orientation and / or distance may be determined using image analysis of the captured (block 306) image to identify particular objects (e.g., optical fiducials) or patient anatomical structures (e.g., eyes). For example, image analysis of the image including the optical fiducials may indicate the distance and viewpoint from which the optical fiducials are viewed, especially for optical fiducials having predictable sizes, shapes, and other characteristics. With reference to FIG. 6A, the patient tracking assembly (200) can be seen positioned on the patient's head (H). One or more optical fiducials (230, 232) may be positioned on the patient tracking assembly (200) or elsewhere, as desired.

[0040] The appearance of the optical fiducial (230) in the image provides an indication of the viewpoint from which the image was captured (e.g., the optical fiducial (230) may appear as a circle when viewed as shown in FIG. 6A, but may appear as an oval as the observer moves left or right) as well as distance (e.g., the optical fiducial may have a diameter of 2 cm). If the optical fiducial (230) has an asymmetric shape of sufficient complexity, or if the surface of the optical fiducial (230) has a pattern or other visible characteristic of sufficient complexity, a single fiducial may provide enough information to completely determine orientation and distance.

[0041] In addition, as shown in FIG. 6A, when several optical fiducials (230, 232) are used, the apparent position of each fiducial relative to the other fiducials can be used as an indication of orientation. Therefore, it can be advantageous to place two or more optical fiducials on the tracking assembly (200) or head (H). In some implementations, one or more fiducials can be integrated into the surface of the tracking assembly (200) at the time of manufacture, which can advantageously indicate a static, known positioning and distance from each other that can be used to assist in subsequent fiducial-based orientation and distance determinations.

[0042] As described, implementations may vary in the particular approach taken to determine orientation (block 308) and distance (block 310), with some implementations relying entirely on tracking each object of interest within the IGS navigation coordinate system, while other implementations may combine such tracking with optically based image analysis or other techniques to improve the resulting accuracy, performance, or other characteristics. Thus, it should be understood that various combinations of the disclosed and other methods exist and will provide different advantages for determining orientation (block 308) and distance (block 310), and such combinations will be apparent to one of ordinary skill in the art based on this disclosure.

[0043] Once the distance and orientation relative to the observed anatomical structure is determined, the distance and orientation may be used to transform the correlated dataset so that it may be displayed via the display (104) as a rendered overlay of the observed anatomical structure or via another device as a rendered overlay of the captured image. The correlated dataset transformation may include, for example, transforming the surgical path (block 312) to match the scale and perspective of the captured (block 306) image, transforming the CT image or other image type (block 314) to match the scale and perspective of the captured (block 306) image, transforming the tracked position of the distal tip of the surgical instrument (block 315) to match the scale and perspective of the captured (block 306) image, and other transformations. Although discussed within the context of FIG. 5, it should be understood that the surgical path, CT image, and tracked surgical tool need not be present in the received (block 304) correlated dataset.

[0044] Transformations of the correlated data set will vary depending on the particular data represented in the correlated data set. For example, with reference to FIG. 8, if the system is configured to overlay some or all of the CT image slices on the patient's head (H), the distance (506) between the viewpoint (502) (e.g., the lens of the camera (106)) and the observation point (504) (e.g., the first point, voxel, or other object in a coordinate system that intersects with the optical axis of the camera (106)) may be used to determine and apply a scaling factor to transform and control the scale at which the CT image slices are displayed via the display (104). As the wearer of the HMD (100) moves towards the head (H), the distance (506) decreases and the scale of the displayed CT image slices increases. Similarly, movement away from the head (H) increases the distance (506) and the scale of the displayed CT image slices decreases. In this manner, the scaling factor may be configured to provide an appropriately sized rendering of the CT image based on the distance (506) corresponding to the perceived size of the head (H) at that distance.

[0045] Continuing with the above example, the position of the viewpoint (502) relative to the observation point (504) may be used to determine, within the three-dimensional coordinate system (501), the appropriate CT image slice to render, and may be used to transform the appropriate CT image slice so that it can be overlaid on the head (H). As an example, in some implementations, CT image slices of the head (H) may be selected and rendered as an overlay depending on the viewpoint from which the head (H) is viewed, such that a viewpoint above the head (H) may indicate an axial view, a viewpoint from in front of the head may indicate a coronal view, and a viewpoint from the side of the head may indicate a sagittal view, and the views may be switched automatically as the wearer of the HMD (100) moves between viewpoints.

[0046] As another example, in some implementations, a CT image slice may be transformed to create a new image that has the appearance of the two-dimensional input image of the CT image slice as if it were fixed in place and viewed from a different viewpoint (e.g., two-dimensional image viewpoint transformation). In this way, the two-dimensional CT image slice displayed on the head (H) may be viewpoint transformed as the wearer of the HMD (100) moves between viewpoints.

[0047] As yet another example of a transformation, for a correlation data set including a surgical path, the coordinates of the surgical path may be rendered as a set of points, lines, or dashed lines and overlaid on the head (H). The distance (506) may be used to transform the scale of the surgical path so that when overlaid on the head (H), each coordinate of the surgical path is properly positioned relative to the head (H). The position of the viewpoint (502) relative to the observation point (504) in the coordinate system (501) may be used to transform the surgical path as the wearer of the (HMD) moves between viewpoints. Such a transformation may be performed as a viewpoint transformation, as described above, or may be performed using other three-dimensional rotation and depth transformations, as would be apparent to one of ordinary skill in the art based on the disclosure herein.

[0048] After each correlated data set has been transformed (e.g., scaled, viewpoint transformed, or otherwise), they may be rendered or displayed (block 316) on one or more viewing devices, which may include displaying the rendered markers as an overlay via the display (104), displaying the rendered markers as an overlay on the captured image via the HHD (101) or another display, or both. To a user wearing the HMD (100), the rendered markers may appear to overlay objects in their field of view (e.g., the patient's head or other anatomical structures) as seen through the transparent display (104). To a user of the HHD (101) or an observer on a wall-mounted display or other device, the captured image may be displayed with the rendered markings overlaid. As described, the steps of capturing images, determining the viewpoint, and transforming the correlated data set may be repeated continuously as images are captured so that the user can move and look around the treatment area as normal while receiving continuous updates of the overlaid information.

[0049] For example, if either movement of the viewed anatomical structure (block 318) or movement of the HMD (100), HHD (101), or other viewing device occurs (block 320), the next captured (block 306) image is received and the orientation (block 308) and distance (block 310) are redetermined. The newly determined orientation and distance are then used in one or more transformations such that the newly generated overlay will take into account any movement or changes that have occurred since the previous image.

[0050] 6A-6C show an example of an interface (400) including a rendered and transformed surgical path as it may be displayed via the display (104) and overlaid on a patient's head (H) viewed directly, or as it may be overlaid and displayed on a captured image of the head (H). In FIG. 6A, the head (H) is viewed from the front (e.g., a coronal view) and rendered markers are overlaid on the image captured via the display (104) or directly. The rendered markers include a surgical path (402) indicating the path the surgical tool should follow, a target marker (404) indicating the destination or portion of the anatomical structure involved in the procedure, and a tool marker (406) indicating the current location of the tracked surgical instrument. The scale of the markers is determined based on the determined distance (310) so that the surgical path (402) accurately overlays the patient's head (H) and appears as if viewed directly on a coronal CT image slice, rather than as an overlay rendered via the display (104) or another device. The location and orientation of the markers are also determined based on the determined orientation (308).

[0051] FIG. 6B shows the head (H) from a side view (e.g., a sagittal view). Each marker is rendered in a new position and orientation based on the change in viewpoint. For example, in FIG. 6A, the surgical path (402) appears to traverse along the coronal plane of the head (H). The coronal plane is not seen in FIG. 6B, and instead, the surgical path (402) is seen traversing along the sagittal plane. Similarly, the positions and orientations of the target marker (404) and tool marker (406), respectively, are changed from FIG. 6A, and the depth at which they are positioned within the head (H) can be determined. FIG. 6C shows the head (H) from a third viewpoint, as seen when an observer moves between the views of FIG. 6A and FIG. 6B. As the observer moves between the views of FIG. 6A and FIG. 6B, or to other viewpoints from which the head (H) may be seen, each of the markers may be updated and newly rendered to provide additional information and context, without being limited to only the viewpoints provided by the CT image set, and without the distraction and disorientation that can come from looking away from the head (H) to the display screen (16) or another display, as described.

[0052] As another example of an interface that may be provided by the visualization system 60, Figure 7A shows a selected image 430 from a CT image set, and Figures 7B-7D show an example of an interface 440 including a rendered overlay of the image 430 and the surgical path 402, as may be displayed via the display 104 and overlaid on a directly viewed patient's head H, or as may be overlaid and displayed on a captured image of the head H. The image 430, the surgical path 402, and the target markers 404 may be received 304 as a correlated data set prior to the procedure.

[0053] In FIG. 7B, the head (H) is viewed from the same perspective as in FIG. 6A. The image (430) is also overlaid on the head (H) after the scale of the image (430) has been transformed to correspond to the perceived size of the head (H) (e.g., as opposed to the image (430) being too large and extending beyond the head (H)). The tool markers (406), surgical path (402), and target markers (404) may also be overlaid and visible with the CT image (430), each of which may be rendered with various degrees of transparency to either completely obscure a direct view of the head (H) through the rendering or allow some transparency. Various implementations of the visualization system (60) may provide interfaces as shown in FIGS. 6A-6C or 7B-7D, while some implementations may provide both, for example allowing certain layers to be enabled or disabled as desired. For example, this may include enabling or disabling the overlay of image (430) on the head (H) as may be desired.

[0054] 7C and 7D show the head (H) from different viewpoints, consistent with those shown in FIGS. 6B and 6C, respectively. The image (430) is simulated as a shaded region, and other rendered markings are transformed to account for the altered viewpoint, respectively. As described, FIG. 7B may show a substantially unaltered CT image of the coronal plane as image (430), while FIG. 7C may show a substantially unaltered CT image of the sagittal plane as image (430). In some implementations, FIG. 7D may show a CT image transformed from the coronal or sagittal plane, depending on whether the observer's viewpoint is closer to the coronal or sagittal plane.

[0055] It should be understood that the interfaces of Figures 6A-7C represent interfaces that may be displayed when the correlated data set is rendered and overlaid on an object seen through a transparent portion of the display (104). For example, referring to Figure 6A, the head (H) is not rendered through the display (104), but is seen through a transparent portion of the display (104) and therefore becomes a visible part of the augmented reality interface. The interfaces of Figures 6A-7C also represent interfaces that may be displayed when the correlated data set is rendered and overlaid on an image captured through the camera (106), such as may be displayed through the display screen (16) or through the HHD (101). In such an interface, the head (H) is also rendered on the display based on the captured image.

[0056] III. Exemplary Combinations The following examples relate to various non-exhaustive methods in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to limit the scope of any claims that may be presented at any time in this application or in a later filing of this application. No disclaimer is intended. The following examples are provided for illustrative purposes only. It is contemplated that the various teachings herein may be configured and applied in many other ways. It is also contemplated that certain features mentioned in the following examples may be omitted in some variations. Thus, none of the aspects or features mentioned below should be considered as critical unless expressly indicated to be so later by the inventors or by the inventors' successors in interest. If any claim is presented in this application or in a later filing related to this application that includes additional features other than those mentioned below, those additional features should not be assumed to be added for any reason related to patentability. EXAMPLES

[0057] A system for ENT visualization, comprising: (a) an image-guided surgery ("IGS") navigation system operable to (i) maintain a coordinate system corresponding to a tracked region, (ii) track one or more position sensors with the coordinate system, and (iii) align a location of a patient's anatomy with the coordinate system; (b) a head mounted display ("HMD") including a wearable frame and a display, the display being positioned on the wearable frame such that it intersects with a neutral optical axis of a wearer; (c) a sensor operable to generate a set of viewpoint data indicative of a viewpoint of the HMD relative to the patient's anatomy; and (d) a processor operable to generate a set of viewpoint data indicative of a viewpoint of the HMD relative to the patient's anatomy. and a processor configured to: (i) receive one or more correlated data sets, each of the one or more correlated data sets including data associated with a coordinate system; (ii) determine an orientation of a neutral optical axis with respect to an observation point on the patient's anatomy based on the set of viewpoint data while the patient's anatomy is viewed from a current viewpoint; (iii) determine a distance between an origin of the neutral optical axis and the observation point based on the set of viewpoint data; (iv) transform the one or more correlated data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomy at the current viewpoint; and (v) render the overlay image via the display. EXAMPLES

[0058] The system of Example 1, wherein the one or more correlation data sets include (i) a surgical path indicating a planned path of the surgical instrument within the patient's anatomy, and (ii) a distal tip location indicating a current location of the distal tip associated with one of the one or more position sensors. EXAMPLES

[0059] The system of example 2, wherein the one or more correlation data sets further include image slices selected from a set of pre-operative images of the patient's anatomy. EXAMPLES

[0060] The system of any one or more of Examples 1-2, further comprising a patient tracking assembly positioned on the patient's anatomical structure, (i) the patient tracking assembly including one of the one or more position sensors, and (ii) the IGS navigation system configured to update the location of the patient's anatomical structure in the coordinate system based on movement of the position sensor. EXAMPLES

[0061] The system of Example 4, wherein the sensor is attached to the HMD and is one of one or more position sensors, and the processor is further configured, when determining the orientation of the neutral optical axis relative to the observation point, to: (i) determine the orientation of the neutral optical axis based on a coordinate system, (ii) determine the orientation of the patient's anatomical structure based on the coordinate system, and (iii) correlate the orientation of the neutral optical axis with the orientation of the patient's anatomical structure based on the coordinate system. EXAMPLES

[0062] The system of any one or more of Examples 4 to 5, wherein the sensor is attached to the HMD and is one of one or more position sensors, and the processor is further configured, when determining the distance between the origin and the observation point, to: (i) determine a location of the origin based on a coordinate system, (ii) determine a location of a patient's anatomical structure based on the coordinate system, and (iii) determine the distance between the location of the origin and the location of the patient's anatomical structure based on the coordinate system. EXAMPLES

[0063] The system of any one or more of Examples 1 to 6, wherein the processor is further configured to, when transforming one or more correlation data sets, (i) determine a scaling factor based on the distance between the origin and the observation point, (ii) determine a viewpoint transformation based on an orientation of the neutral optical axis relative to the observation point, and (iii) generate an overlay image including a scale determined by the scaling factor and a viewpoint determined by the viewpoint transformation. EXAMPLES

[0064] The system of Example 7, wherein the display includes a transparent screen, and the processor is further configured to render the overlay image on the transparent screen such that the overlay image appears on a directly viewed portion of the patient's anatomical structure. EXAMPLES

[0065] The system of any one or more of Examples 1 to 8, wherein the processor is further configured to (i) repeatedly redetermine the distance and orientation, and (ii) update and render the overlay image via the display as the distance and orientation change. EXAMPLES

[0066] The system of any one or more of Examples 1 to 9, wherein the sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor is further configured, when determining the orientation and distance, to: (i) store an object dataset indicative of one or more visible characteristics of the object and indicative of a relationship between the object and an observation point in a coordinate system; (ii) receive an image including the object from the camera; and (iii) determine the orientation and distance based on the object dataset and the image. EXAMPLES

[0067] The system of Example 10, further comprising one or more optical fiducials positioned in proximity to the patient's anatomical structure, the object described by the object dataset being the one or more optical fiducials. EXAMPLES

[0068] The system of Example 11 further comprises a patient tracking assembly positioned on the patient's anatomical structure, (i) the patient tracking assembly includes one of the one or more position sensors, (ii) the IGS navigation system is configured to update the location of the patient's anatomical structure in the coordinate system based on movement of the position sensor, and (iii) one or more optical fiducials are positioned on a surface of the patient tracking assembly. EXAMPLES

[0069] The system of any one or more of Examples 1 to 12, wherein the sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor, in addition to rendering the overlay image via the display, is further configured to: (i) receive an image from the camera and correlate the image to the overlay image based on a known static offset value, (ii) add the overlay image to the image to generate an augmented image, and (iii) display the augmented image on one or more displays other than the display. EXAMPLES

[0070] The system of any one or more of Examples 1 to 13, wherein (i) the processor includes one or more of a first processor of an IGS navigation system and a second processor of an HMD; (ii) the sensor includes one or more of a gyroscope sensor coupled to the HMD, a camera attached to the HMD, and a position sensor of one or more position sensors attached to the HMD; and (iii) the set of viewpoint data includes one or more of a set of gyroscope orientation data, a set of position coordinates associated with a coordinate system, a set of orientation coordinates associated with the coordinate system, and an image captured by the camera. EXAMPLES

[0071] A method comprising: (a) configuring an image-guided surgery ("IGS") navigation system to (i) maintain a coordinate system corresponding to a tracked region and (ii) track one or more position sensors with the coordinate system; (b) aligning a location of a patient's anatomy with the coordinate system; (c) receiving one or more correlation data sets, each of the one or more correlation data sets including data associated with the coordinate system; and (d) viewing the patient's anatomy from a current viewpoint while wearing a head mounted display ("HMD") including a wearable frame and a display. (e) determining an orientation of the neutral optical axis relative to an observation point on the patient's anatomical structure based on a set of viewpoint data received from a sensor of the HMD; (f) determining a distance between an origin of the neutral optical axis and the observation point based on the set of viewpoint data; (g) transforming one or more correlation data sets based on the orientation and distance to generate an overlay image corresponding to the patient's anatomical structure at the current viewpoint; and (h) rendering the overlay image via the display. EXAMPLES

[0072] The method of example 15, wherein receiving the one or more correlated data sets includes: (i) receiving a surgical path indicating a planned path of the surgical instrument within the patient's anatomy; (ii) receiving a distal tip location indicating a current location of a distal tip associated with one of the one or more position sensors; and (iii) receiving a selected image slice from a set of pre-operative images of the patient's anatomy. EXAMPLES

[0073] The method of any one or more of Examples 15-16, wherein the sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the method further includes: (a) positioning a patient tracking assembly over the patient's anatomical structure, the patient tracking assembly including (i) a position sensor of the one or more position sensors and (ii) one or more optical fiducials; (b) storing an object dataset indicating one or more visual characteristics of the one or more optical fiducials and indicating a relationship between the one or more optical fiducials and an observation point in a coordinate system; (c) receiving an image from the camera including the one or more optical fiducials; and (d) determining an orientation and distance based on the object dataset and the image. EXAMPLES

[0074] 18. The method of example 17, wherein the sensor includes a second position sensor of the one or more position sensors, and further includes determining an orientation and distance based on the object dataset, the image, and the coordinate system. EXAMPLES

[0075] A system for ENT visualization, comprising: (a) an image-guided surgery ("IGS") navigation system operable to (i) maintain a coordinate system corresponding to a tracked region, (ii) track one or more position sensors with the coordinate system, and (iii) align a location of a patient's anatomy with the coordinate system; (b) a handheld display ("HHD") including a camera and a display; (c) a sensor operable to generate a set of viewpoint data indicative of a viewpoint of the HHD relative to the patient's anatomy; and (d) a processor, the processor operable to (i) receive one or more correlated data sets, the one or more correlated data sets being operable to generate a set of viewpoint data indicative of a viewpoint of the HHD relative to the patient's anatomy. each of the correlation data sets includes data associated with a coordinate system; (ii) receiving images from the camera while the HHD is pointed at the patient's anatomy from a current viewpoint; (iii) determining an orientation of an optical axis of the camera relative to an observation point on the patient's anatomy based on the set of viewpoint data; (iv) determining a distance between an origin of the optical axis and the observation point based on the set of viewpoint data; (v) transforming one or more correlation data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomy in the image; and (vi) displaying an augmented image via the display based on the overlay image and the image. EXAMPLES

[0076] The system of Example 19, wherein (i) the sensor is one position sensor of the one or more position sensors, (ii) a second position sensor of the one or more position sensors is positioned on the patient's anatomical structure, and (iii) the one or more correlated data sets include (A) a surgical path indicating a planned path of the surgical instrument within the patient's anatomical structure, and (B) a distal tip location indicating a current location of the distal tip associated with a third position sensor of the one or more position sensors.

[0077] IV.Other It should be understood that any one or more of the teachings, expressions, embodiments, examples, etc. described herein can be combined with any one or more of the other teachings, expressions, embodiments, examples, etc. described herein. Thus, the above teachings, expressions, embodiments, examples, etc. should not be considered in isolation with respect to each other. Various suitable ways in which the teachings herein can be combined will be readily apparent to those skilled in the art in light of the teachings herein. Such modifications and variations are intended to be included within the scope of the claims.

[0078] Any patent, publication, or other disclosure referred to herein as being incorporated by reference should be understood to be incorporated herein, in whole or in part, only to the extent that the incorporated content does not contradict the current definitions, opinions, or other disclosures set forth in this disclosure. As such, and to the extent necessary, the disclosures explicitly set forth herein shall take precedence over any conflicting statements incorporated herein by reference. Any content, or portions thereof, that is referred to herein as being incorporated by reference but that contradicts the current definitions, opinions, or other disclosures set forth herein shall be incorporated only to the extent that no conflict arises between the incorporated content and the current disclosures.

[0079] The device variations described above can be designed to be disposed of after a single use, or they can be designed to be used multiple times. The variations, in either or both cases, can be reconditioned for reuse after at least one use. Reconditioning can include any combination of the steps of disassembly of the device, followed by cleaning or replacement of particular parts, and subsequent reassembly. In particular, some device variations can be disassembled, and any number of the particular parts or pieces of the device can be selectively replaced or removed in any combination. Upon cleaning and / or replacement of particular parts, some device variations can be reassembled for subsequent use at a reconditioning facility, or by a user immediately prior to a procedure. Those skilled in the art will appreciate that reconditioning of a device can utilize a variety of techniques for disassembly, cleaning / replacement, and reassembly. The use of such techniques, and the resulting reconditioned device, are all within the scope of the present application.

[0080] By way of example only, the variations described herein may be sterilized before and / or after treatment. In one sterilization technique, the device is placed in a closed and sealed container, such as a plastic or TYVEK bag. The container and device may then be placed in a radiation field that can penetrate the container, such as gamma radiation, x-rays, or high-energy electron beams. The radiation may kill bacteria on the device and in the container. The sterilized device may then be stored in the sterile container for later use. The device may also be sterilized using any other technique known in the art, including but not limited to beta or gamma radiation, ethylene oxide, or steam.

[0081] Although various embodiments of the present invention have been shown and described, appropriate modifications by those skilled in the art may be made to further adapt the methods and systems described herein without departing from the scope of the present invention. Although some of such possible modifications have been described, other modifications will be apparent to those skilled in the art. For example, the above examples, embodiments, geometries, materials, dimensions, proportions, steps, etc. are illustrative and not required. Thus, the scope of the present invention should be considered in terms of the following claims and should not be understood to be limited to the details of structure and operation shown and described in the specification and drawings.

[0082] [Embodiment] (1) A system for ENT visualization, comprising: (a) An image-guided surgery ("IGS") navigation system, comprising: (i) maintaining a coordinate system corresponding to the tracked region; (ii) tracking one or more position sensors with the coordinate system; (iii) an image-guided surgery ("IGS") navigation system operable to register a location of a patient's anatomy with said coordinate system; and (b) a head mounted display ("HMD") including a wearable frame and a display, the display being positioned on the wearable frame such that it intersects with a neutral optical axis of the wearer; and (c) a sensor operable to generate a set of viewpoint data indicative of a viewpoint of the HMD relative to an anatomy of the patient; (d) a processor; The processor, (i) receiving one or more correlated data sets, each of the one or more correlated data sets including data associated with the coordinate system; (ii) determining an orientation of the neutral optical axis relative to a viewpoint on the patient's anatomy based on the set of viewpoint data while the patient's anatomy is viewed from a current viewpoint; and (iii) determining a distance between an origin of the neutral optical axis and the observation point based on the set of viewpoint data; and (iv) transforming the one or more correlated data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomy at the current viewpoint; and (v) rendering the overlay image through the display. (2) the one or more correlated data sets include: (i) a surgical path indicating a planned path of a surgical instrument within the patient's anatomy; (ii) a distal tip location indicating a current location of a distal tip associated with one of the one or more position sensors. (3) The system of embodiment 2, wherein the one or more correlation data sets further include image slices selected from a set of preoperative images of the patient's anatomical structure. (4) a patient tracking assembly positioned over the patient's anatomy; (i) the patient tracking assembly includes a position sensor of the one or more position sensors; (ii) The system of embodiment 1, wherein the IGS navigation system is configured to update the location of the patient's anatomical structure in the coordinate system based on movement of the position sensor. (5) the sensor is attached to the HMD and is one of the one or more position sensors, and the processor, when determining the orientation of the neutral optical axis with respect to the observation point, (i) determining an orientation of the neutral optical axis based on the coordinate system; (ii) determining an orientation of the patient's anatomy based on the coordinate system; (iii) the system of claim 4, further configured to correlate the orientation of the neutral optical axis with the orientation of the patient's anatomical structure based on the coordinate system.

[0083] (6) the sensor is attached to the HMD and is one of the one or more position sensors, and the processor, when determining the distance between the origin and the observation point, (i) determining a location of the origin based on the coordinate system; (ii) determining a location of the patient's anatomical structure based on the coordinate system; (iii) determining the distance between the location of the origin and the location of the patient's anatomical structure based on the coordinate system. (7) The processor, when transforming the one or more correlation data sets, (i) determining a scaling factor based on the distance between the origin and the observation point; (ii) determining a viewpoint transformation based on the orientation of the neutral optical axis relative to the observation point; (iii) generating the overlay image having a scale determined by the scaling factor and a viewpoint determined by the viewpoint transformation. (8) The system of embodiment 7, wherein the display includes a transparent screen, and the processor is further configured to render the overlay image on the transparent screen such that the overlay image appears on a directly viewed portion of the patient's anatomical structure. (9) The processor: (i) iteratively redetermining said distance and said orientation; (ii) updating and rendering the overlay image via the display in response to changes in the distance and the orientation. (10) The sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor, when determining the orientation and the distance, (i) storing an object data set indicative of one or more visual characteristics of an object and indicative of a relationship between said object and said observation point in said coordinate system; (ii) receiving an image from the camera that includes the object; (iii) determining the orientation and the distance based on the object dataset and the image.

[0084] (11) The system of embodiment 10, further comprising one or more optical fiducials positioned in proximity to the patient's anatomical structure, the object described by the object dataset being the one or more optical fiducials. (12) further comprising a patient tracking assembly positioned over the patient's anatomy; (i) the patient tracking assembly includes a position sensor of the one or more position sensors; (ii) the IGS navigation system is configured to update the location of the patient's anatomy in the coordinate system based on movement of the position sensor; (iii) The system of embodiment 11, wherein the one or more optical fiducials are positioned on a surface of the patient tracking assembly. (13) The sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor, in addition to rendering the overlay image via the display, further comprises: (i) receiving an image from the camera and correlating the image to the overlay image based on a known static offset value; (ii) adding the overlay image to the image to generate an augmented image; (iii) The system of embodiment 1, further configured to display the augmented image on one or more displays other than the display. (14) (i) the processor includes one or more of a first processor of the IGS navigation system and a second processor of the HMD; (ii) the sensors include one or more of a gyroscope sensor coupled to the HMD, a camera attached to the HMD, and a position sensor of the one or more position sensors attached to the HMD; (iii) The system of embodiment 1, wherein the set of viewpoint data includes one or more of a set of gyroscope orientation data, a set of position coordinates associated with the coordinate system, a set of orientation coordinates associated with the coordinate system, and an image captured by the camera. (15) A method comprising the steps of: (a) configuring an image-guided surgery ("IGS") navigation system, (i) maintaining a coordinate system corresponding to the tracked region; (ii) configuring one or more position sensors to be tracked using the coordinate system; and (b) registering a location of a patient's anatomy with said coordinate system; (c) receiving one or more correlated data sets, each of the one or more correlated data sets including data associated with the coordinate system; and (d) viewing the patient's anatomy from a current viewpoint while wearing a head mounted display ("HMD") including a wearable frame and a display, the display positioned on the wearable frame such that it intersects with a neutral optical axis of the wearer; and (e) determining an orientation of the neutral optical axis relative to a viewing point on the patient's anatomy based on a set of gaze point data received from a sensor of the HMD; (f) determining a distance between an origin of the neutral optical axis and the observation point based on the set of viewpoint data; and (g) transforming the one or more correlated data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomy at the current viewpoint; and (h) rendering the overlay image through the display.

[0085] (16) Receiving the one or more correlated data sets comprises: (i) receiving a surgical path indicating a planned path of a surgical instrument within the patient's anatomy; (ii) receiving a distal tip location indicating a current location of a distal tip associated with a position sensor of the one or more position sensors; (iii) receiving a selected image slice from a set of preoperative images of the patient's anatomical structure. (17) The sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the method further includes: (a) positioning a patient tracking assembly over the patient anatomy, the patient tracking assembly comprising: (i) a position sensor of the one or more position sensors; and (ii) one or more optical fiducials; and (b) storing an object data set indicative of one or more visual characteristics of the one or more optical fiducials and indicative of a relationship between the one or more optical fiducials and the observation point in the coordinate system; (c) receiving an image from the camera that includes the one or more optical fiducials; and 16. The method of claim 15, further comprising: (d) determining the orientation and the distance based on the object dataset and the image. (18) The method of embodiment 17, wherein the sensor includes a second position sensor of the one or more position sensors, and further comprising determining the orientation and the distance based on the object dataset, the image, and the coordinate system. (19) A system for ENT visualization, comprising: (a) An image-guided surgery ("IGS") navigation system, comprising: (i) maintaining a coordinate system corresponding to the tracked region; (ii) tracking one or more position sensors with the coordinate system; (iii) an image-guided surgery ("IGS") navigation system operable to register a location of a patient's anatomy with said coordinate system; and (b) a handheld display ("HHD") including a camera and a display; (c) a sensor operable to generate a set of viewpoint data indicative of a viewpoint of the HHD relative to an anatomy of the patient; (d) a processor; The processor, (i) receiving one or more correlated data sets, each of the one or more correlated data sets including data associated with the coordinate system; (ii) receiving an image from the camera while the HHD is pointed at the patient's anatomy from a current viewpoint; (iii) determining an orientation of the optical axis of the camera relative to a viewpoint on the patient's anatomy based on the set of viewpoint data; and (iv) determining a distance between the origin of the optical axis and the observation point based on the set of viewpoint data; and (v) transforming the one or more correlation data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomical structure in the image; and (vi) displaying an augmented image through the display based on the overlay image and the image. (20) (i) the sensor is one of the one or more position sensors; (ii) a second position sensor of the one or more position sensors is positioned on the patient's anatomy; and (iii) the one or more correlated data sets include: (A) a surgical path illustrating a planned path of a surgical instrument within the patient's anatomy; (B) a distal tip location indicating a current location of a distal tip associated with a third position sensor of the one or more position sensors.

Claims

1. 1. A system for ENT visualization, comprising: (a) an image-guided surgery ("IGS") navigation system, comprising: (i) maintaining a coordinate system corresponding to the tracked region; (ii) tracking a plurality of position sensors using the coordinate system; (iii) an image-guided surgery ("IGS") navigation system operable to register a location of a patient's anatomy with said coordinate system; and (b) a head mounted display ("HMD") including a wearable frame and a display, the display being positioned on the wearable frame such that it intersects a neutral optical axis of a wearer; and (c) a sensor operable to generate a set of viewpoint data indicative of a viewpoint of the HMD relative to the patient's anatomy; and (d) a processor; The processor, (i) receiving one or more correlated data sets, each of the one or more correlated data sets including data associated with the coordinate system including a distal tip location indicative of a current position of a distal tip of a surgical instrument, distal tip location data being obtained from an instrument position sensor of the plurality of position sensors, the instrument position sensor being disposed at the distal tip of the surgical instrument; (ii) determining an orientation of the neutral optical axis relative to a viewpoint on the patient's anatomy based on the set of viewpoint data while the patient's anatomy is viewed from a current viewpoint; and (iii) determining a distance between an origin of the neutral optical axis and the observation point based on the set of viewpoint data; and (iv) transforming the one or more correlated data sets based on the orientation and the distance to generate an overlay image corresponding to the patient's anatomy at the current viewpoint; and (v) rendering via the display the overlay image including a view of the current position of the distal tip having the instrument position sensor; The system further comprises a patient tracking assembly positioned over the patient's anatomy; (i) the patient tracking assembly includes a patient anatomy position sensor among the plurality of position sensors; (ii) the IGS navigation system is configured to update the location of the patient's anatomy in the coordinate system based on signals from the patient anatomy position sensor.

2. The one or more correlated data sets include: The system of claim 1 , further comprising: (i) a surgical path indicating a planned path of the surgical instrument within the patient's anatomy.

3. The system of claim 2 , wherein the one or more correlated data sets further comprise image slices selected from a set of pre-operative images of the patient's anatomy.

4. The sensor is attached to the HMD and is one of the plurality of position sensors, and the processor, when determining the orientation of the neutral optical axis relative to the observation point, further comprises: (i) determining an orientation of the neutral optical axis based on the coordinate system; (ii) determining an orientation of the patient's anatomy based on the coordinate system; 2. The system of claim 1, further configured to: (iii) correlate the orientation of the neutral optical axis with the orientation of the patient's anatomy based on the coordinate system.

5. The sensor is attached to the HMD and is one of the plurality of position sensors, and the processor, when determining the distance between the origin and the observation point, (i) determining a location of the origin based on the coordinate system; (ii) determining a location of the patient's anatomical structure based on the coordinate system; 2. The system of claim 1, further configured to: (iii) determine the distance between the location of the origin and the location of the patient's anatomy based on the coordinate system.

6. The processor, when transforming the one or more correlated data sets, (i) determining a scaling factor based on the distance between the origin and the observation point; (ii) determining a viewpoint transformation based on the orientation of the neutral optical axis relative to the observation point; The system of claim 1 , further configured to: (iii) generate the overlay image having a scale determined by the scaling factor and a viewpoint determined by the viewpoint transformation.

7. 7. The system of claim 6, wherein the display includes a transparent screen, and the processor is further configured to render the overlay image on the transparent screen such that the overlay image appears on a directly viewed portion of the patient's anatomy.

8. The processor, (i) iteratively redetermining said distance and said orientation; The system of claim 1 , further configured to: (ii) update and render the overlay image via the display in response to changes in the distance and the orientation.

9. The sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor, when determining the orientation and the distance, further comprises: (i) storing an object data set indicating one or more visual characteristics of an object and indicating a relationship between said object and said observation point in said coordinate system; (ii) receiving an image from the camera that includes the object; The system of claim 1 , further configured to: (iii) determine the orientation and the distance based on the object dataset and the image.

10. 10. The system of claim 9, further comprising one or more optical fiducials positioned proximate the patient's anatomy, the object described by the object dataset being the one or more optical fiducials.

11. The system described in claim 10, wherein the one or more optical fiducials are positioned on a surface of the patient tracking assembly.

12. The sensor includes a camera positioned on the wearable frame and having an optical axis parallel to and statically offset from the neutral optical axis, and the processor, in addition to rendering the overlay image via the display, further comprises: (i) receiving an image from the camera and correlating the image to the overlay image based on a known static offset value; (ii) adding the overlay image to the image to generate an augmented image; The system of claim 1 , further configured to: (iii) display the augmented image on one or more displays other than the display.

13. (i) the processor includes one or more of a first processor of the IGS navigation system and a second processor of the HMD; (ii) the sensor includes one or more of a gyroscope sensor coupled to the HMD, a camera attached to the HMD, and a position sensor of the plurality of position sensors attached to the HMD; 2. The system of claim 1 , wherein (iii) the set of viewpoint data includes one or more of a set of gyroscope orientation data, a set of position coordinates associated with the coordinate system, a set of orientation coordinates associated with the coordinate system, and an image captured by the camera.

Images