Systems and methods for assisting visualization during a program

Abstract

A system operable to permit viewing of a physical subject using information regarding at least one object separated from the subject overlying it. The system includes a view screen. The system is further capable of determining the position of the object relative to the subject.

Description

Technical Field

[0001] The present disclosure relates to performing procedures, and more particularly to registering an image space with a real or subject space. Background Art

[0002] During a selected procedure, a user may obtain an image of a subject based on image data of the subject. Generally, various imaging techniques or systems may be used to obtain the image data, and the image data may be reconstructed for the user to view on a display device (such as, a tablet or flat screen, cathode ray tube, etc.) located away from the surgical area. The surgical area may be associated with a subject, such as a human patient, for performing the selected procedure. For example, a sinus procedure may be performed and an image of the subject's sinus may be displayed on a display device that does not cover the subject.

[0003] The procedure may further include directly viewing at least a portion of the area of interest or surgical area, such as with an endoscope. The endoscope may position a camera at a selected location, such as within the nose or a palpable sinus cavity. The endoscope may have a limited range of movement and / or field of view at various positions within the selected anatomical region. Summary of the Invention

[0004] This section provides an overview of the present disclosure and is not an exhaustive disclosure of its full scope or all of its features.

[0005] In various embodiments, an instrument may be positioned relative to a portion of a subject to perform a procedure. The subject may include a living subject or an inanimate subject. In various embodiments, a living subject may include a human, and the procedure being performed may be relative to a nasal passage and / or sinus cavity. For example, a balloon sinus dilation procedure may occur, such as a procedure performed using an EM balloon sinus dilation system sold by Medtronic, Inc. having a place of business in Minnesota. It should be understood that an electromagnetic (EM) navigation instrument is not required to perform sinus dilation. However, dilating a sinus with an inflatable instrument may include an instrument incorporating various inflation and expansion features, such as a sinus dilation procedure.

[0006] In various embodiments, the procedure may occur in a surgical area of the subject. The surgical area may be a specific or limited area or volume on or relative to which the procedure is being performed. The surgical area may also be referred to as the area of interest or may include the area of interest therein. In various embodiments, for example, surgery may be performed on a human subject's sinus, such as for performing debridement, dilation, or other appropriate procedures. The procedure may occur within the surgical area, while the area of interest may include the entire head of the patient or skull.

[0007] Surgeries performed on the sinuses can generally be minimally invasive or non-open procedures. In minimally invasive procedures, various natural body cavities such as the nasal passages are used to access the sinuses. Thus, when accessing the sinuses, the user may not be able to see the operating end of the instrument.

[0008] The instrument for performing the procedure may include a tracking device configured to be or operable to be tracked by a tracking system. In various embodiments, the tracking system may include a visual or optical tracking system that tracks the tracking device on the instrument, such as by viewing or recording. Then, a navigation or tracking system including a processor system may determine the position of the operating end relative to the tracking device based on known and / or predetermined geometric relationships between the operating end and the tracking device.

[0009] The tracking system may include one or more cameras or optical tracking devices positioned relative to a display device. The display device may include a transparent or translucent view screen. The view screen may be positioned relative to the user, such as to allow the user to view the subject through the view screen. The view screen may be mounted to a structure that allows the user to wear the view screen relatively close to the user's eyes such that the view screen fills all or a substantial portion of the user's field of view. Then, the displayed image may be shown on the view screen to allow the user to view the subject while also viewing the image. The tracked position of the instrument or at least a portion thereof (such as the operating end) may also be displayed on the display using the view screen. Thus, the camera may be associated with a device worn by the user to allow determination of the position of the instrument relative to the surgical area and / or the area of interest in the surgical area, and to overlay or enhance the user's view of the subject by displaying the tracked position of the instrument on the image.

[0010] The system may include a viewing portion or system and a tracking portion. The viewing portion may view the real object (e.g., the subject) and the displayed image. Thus, the viewing system may be an augmented reality viewing system. In addition or as an alternative, the viewing system may include a view screen or monitor that is separate or spaced apart from the subject. Additionally, an image may be captured in real time using a selected imaging system, such as an endoscope. The endoscope may be positioned relative to the subject, such as within the nasal passages and / or sinuses, and the image obtained using the endoscope may be displayed simultaneously with a view of the subject obtained prior to the surgical procedure. Thus, the viewing system may be used to display and view real-time and pre-acquired images. The two types of images may be registered with the subject using various techniques, such as those described elsewhere herein.

[0011] According to the specific embodiments provided herein, other application areas will become apparent. The description and specific examples in this summary are intended for illustrative purposes only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The figures described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0013] Figure 1 is an environmental view for the user and the viewing system;

[0014] Figure 2A is a real / physical world view of the region of interest including the instrument according to various embodiments;

[0015] Figure 2B is a real / physical world view of the region of interest including the instrument according to various embodiments;

[0016] Figure 3A is the viewing point of the user viewing the real / physical world and at least one displayed icon according to various embodiments;

[0017] Figure 3B is the viewing point of the user viewing the real / physical world and at least one displayed icon according to various embodiments;

[0018] Figure 3C is the display of the viewing point of the user viewing at least one displayed icon and a live image using a display device according to various embodiments; and

[0019] Figure 4 is a flowchart showing the operation of the display system.

[0020] Corresponding reference numerals indicate corresponding parts throughout the several views of the drawings. Detailed Description

[0021] Example embodiments will now be described more fully with reference to the accompanying drawings.

[0022] Reference Figure 1 , in an operating room or surgical suite 10, a user such as a surgeon 12 may perform a procedure on a subject such as a patient 14, who may be lying on or supported by a hospital bed or support frame 15. The patient 14 may define a patient longitudinal axis 14l. To assist in performing the procedure, the user 12 may use an imaging system 16 to acquire image data of the patient 14 to allow the selected system to generate or create images to assist in performing the procedure. The imaging system 16 may include any suitable imaging system, such as a computed tomography (CT) imager, an imaging system sold by Medtronic, Inc., and / or a VGi evo cone beam imager sold by NewTom, having a place of business in Verona, Italy.

[0023] An image data generation model (such as a three-dimensional (3D) image) can be used. The generated model can be displayed as an image 18 on a display device 20. In addition to or as an alternative to the display device 18, a projection image (e.g., a 2D x-ray projection) captured by the imaging system 16 can be displayed. Further, an augmented view screen (AV) or a display device 21 can be provided to or used by the user 12. As discussed elsewhere herein, the AV 21 can be worn by the user 12. Additionally, as discussed herein, the AV 21 can also be referred to as a viewing system, i.e., an integrated system or a part of a system for viewing various objects.

[0024] Either or both of the display device 20 or the augmented view screen 21 can be part of and / or connected to a processor system 22, which includes an input device 24 (the input device can include a keyboard, a mouse, a microphone for voice input, and an input from a camera) and a processor 26 (which can include one or more processors or microprocessors in combination with the processing system 22, and a non-transitory and / or transitory memory 27 of a selected type. A connection 28 can be provided between the processor 26 and the display device 20 or the augmented view screen 21 for data communication to allow driving the display device 20 to display or show the image 18.

[0025] As discussed above, the imaging system 16 can include an imaging system sold by Medtronic Navigation, Inc., which has a business location in Louisville, CO, USA. The imaging system. Including The imaging system 16 or other suitable imaging systems can be used during a selected procedure, such as the imaging systems described in U.S. Patent Nos. 8,238,631; 9,411,057; and 9,807,860, which are hereby incorporated by reference in their entirety. The imaging system 16 can be used to acquire image data of the patient 14 before or during the use of the AV 21.

[0026] When, for example, including an imaging system, the imaging system 16 can include a mobile cart 30 that includes a controller and / or a control system 32. The control system can include a processor 33a and a memory 33b (e.g., a non-transitory memory). The memory 33b can include various instructions executed by the processor 33a to control the imaging system (including various parts of the imaging system 16).

[0027] The imaging gantry 34 that locates the source unit 36 and the detector 38 can be connected to the mobile cart 30. The gantry 34 can be O-shaped or annular, where the gantry 34 is substantially annular and includes a wall that forms a volume in which the source unit 36 and the detector 38 can move. The mobile cart 30 can be moved from one operating room to another operating room, and the gantry 34 can be moved relative to the cart 30, as discussed further herein. This allows the imaging system 16 to be mobile and movable relative to the subject 14, thereby allowing it to be used in multiple positions and with multiple procedures without the need for capital expenditure or space dedicated to the imaging system. The control system can include a processor 33a (which can be a general-purpose processor or a specific application processor) and a memory system 33b (e.g., non-transitory memory, such as a spinning disk or solid-state non-volatile memory). For example, the memory system can include instructions to be executed by the processor to perform functions and determine results, as discussed herein.

[0028] The source unit 36 can be an x-ray emitter that emits x-rays through the patient 14 to be detected by the detector 38. As understood by those skilled in the art, the x-rays emitted by the source 36 can be emitted in a cone and detected by the detector 38. The source 36 / detector unit 38 is generally diametrically opposed within the gantry 34. The detector 38 can move around the patient 14 within the gantry 34 in a 360° motion, while the source 36 remains approximately 180° opposite the detector 38 (e.g., using a fixed internal gantry or a moving system). In addition, the gantry 34 can move equidistantly relative to the subject 14 that can be placed on the patient support or workbench 15, generally in the direction of the arrow 40 as shown in Figure 1 as shown. The gantry 34 can also be tilted relative to the patient 14, as shown by the arrow 42, move longitudinally along the longitudinal axis 14L of the patient 14 and the cart 30 along line 44, and can move up and down generally along line 46 relative to the cart 30 and transversely to the patient 14 to allow the source 36 / detector 38 to be positioned relative to the patient 14. The imaging device 16 can be precisely controlled to move the source 36 / detector 38 relative to the patient 14 to generate precise image data of the patient 14. The imaging device 16 can be connected to the processor 26 via a connection 50, and the connection 50 can include a wired or wireless connection or physical medium transmission from the imaging system 16 to the processor 26. Thus, the image data collected using the imaging system 16 can be transmitted to the processing system 22 for navigation, display, reconstruction, etc.

[0029] The source 36 can be any suitable x-ray source, such as a multi-power x-ray source. However, it should be understood that the imaging system 16 can be any suitable imaging system, such as a magnetic resonance imaging (MRI) system, a C-arm x-ray imaging system; a computed tomography (CT) imaging system, etc. However, the image data and / or images obtained using the selected imaging system can be displayed on one or more of the display devices 20, 21.

[0030] It should also be understood that prior to performing a procedure on patient 14, the imaging system 16 can be operated to acquire image data and / or images. For example, images can be acquired and studied to diagnose and / or plan a procedure for patient 14. Thus, the user 12 performing the procedure on patient 14 does not need to use the imaging system 16 in the same room as the procedure being performed.

[0031] According to various embodiments, the imaging system 16 can be used with a tracking system and a navigation system, the tracking system and the navigation system including various parts as discussed herein, and can be operated to track the position of the imaging device 16 and / or other parts. The tracking system can include a locator and / or digitizer, including any one or both of an optical locator 60 and an electromagnetic locator 62, and can be used to generate a field and / or receive and / or transmit signals within a navigation domain relative to patient 14. The navigation space or navigation domain relative to patient 14 can be registered with the image 18. As understood in the art, registration allows for the registration of the navigation space defined within the navigation domain and the image space defined by the image 18.

[0032] In various embodiments, a patient tracker or dynamic reference frame 64 can be attached to patient 14 to allow for dynamic tracking and maintenance of the registration of patient 14 with the image 18. The patient tracking device or dynamic registration device 64 allows for the registration of the image and then its use for the selected procedure. In various embodiments, the locators 60, 62 can track the patient tracker. Additional communication lines 74 can be provided between various features such as the locators 60, 62, the imaging system 16, and the interface system 76 and the processor system 22 (which can be a navigation processor system). In various embodiments, the communication system 74 can be wired, wireless, or use a physical media transmission system (e.g., read / write to a memory chip).

[0033] Additionally, the gantry 34 can include a tracking device such as an optical tracking device 82 or an electromagnetic tracking device 84 that is tracked using one or more of the optical locator 60 or the electromagnetic locator 62. Thus, the imaging device 16 can be tracked relative to patient 14, and the instrument 66 can also be tracked to allow for initial registration, automatic registration, or continued registration of patient 14 with the image 18. The registration and navigation processes are discussed in U.S. Patent No. 8,238,631, which is incorporated herein by reference.

[0034] Those skilled in the art will understand that the instrument 66 can be any suitable instrument, such as a ventricular or vascular stent, a spinal implant, a nerve stent or stimulator, an ablation device, a dilator, etc. The instrument 66 can be an interventional instrument, or can include or be an implantable device. Tracking the instrument 66 allows the use of the registration image 18 to view the position of the instrument 66 relative to the patient 14 (including x, y, z positions and orientation) without directly viewing the instrument 66 within the patient 14.

[0035] Continuing to refer Figure 1 And additionally referring Figure 2A , in addition to and / or as an alternative to the patient tracker 64, the patient 14 can include one or more patient markers or trackers 100 (referred to herein as 100 and lowercase letters). The patient tracker 100 can include various features, such as being opaque or being imaged by various imaging systems (e.g., X-ray or MRI). According to various embodiments, the tracker 100 according to various embodiments can generally be visible or captured in the image data acquired using the imaging system 16. Thus, the patient marker 100 can be identified in the image or image data of the subject 14.

[0036] In various embodiments, the patient marker 100 can be identified substantially automatically by the processor system 26 and / or 33a or any other suitable imaging processor or processor system. The marker 100 can include a selected and / or unique geometric shape that can be identified in the image or image data. Various techniques can be used to segment and identify the marker 100 in the selected image or image data. However, it should also be understood that the user 12 can identify the marker in the image data, such as by selecting a portion of the image and using one or more user inputs 24 to identify the portion as the marker 100.

[0037] The marker 100 can be positioned on the patient 14 in any suitable manner. For example, the marker 100 can be adhered to the patient 14 using a self-adhesive backing, a suitable glue or adhesive material added to the marker 100 and / or the patient 14, or other suitable mechanisms. Additionally, or alternatively, the marker 100 (e.g., markers 100d and 100e) can be fixed to the bone structure of the patient 14. The marker 100 can be formed or provided as a screw, or have a threaded portion to allow them to be screwed into and fixedly positioned in the bone structure, such as the skull of the patient 14.

[0038] Regardless of the connection technique, the marker 100 is positioned on the patient 14 at a selected time. Generally, the marker 100 is positioned on the patient 14 before imaging, such as when acquiring the image data or image of the patient 14 using the imaging system 16. Thus, when operating the image system 16 to acquire an image of the patient 14, the marker 100 is positioned on the patient 14 and will appear in any acquired image data or image.

[0039] Accordingly, marker 100 can be used as a reference point. It should be understood that patient 14 may also include various physical and anatomical reference points, such as the tip of the nose, the corners of the eyes, the earlobes, etc. Nevertheless, the reference points, whether provided by marker 100 and / or the anatomical reference points of patient 14, can be used to register the images acquired by imaging system 16. Those skilled in the art should understand that the images acquired of patient 14 can define an image space. The image space can be the surgical area or procedure area (RO) and / or an area larger than but at least including the RO, such as an area referred to as the region of interest (ROI). It should be understood that the procedure can occur in a specific area within the region of interest. Nevertheless, the coordinates of the image data or space can be related or registered to patient 14 in the physical or real space. Marker 100 can be used to register the image space to the patient and / or the navigation space defined by patient 14 and / or the surgical area within or relative to patient 14.

[0040] As described above, instrument 66 can be any suitable instrument. Instrument 66 can include a dilator instrument that includes a distal end 110 that can be positioned within a portion of patient 14, such as through nasal passage 112, and into one or more sinuses of patient 14, such as the frontal sinus. Instrument 66 can further include a proximal end 114 to which a connector 116 is connected, and an inflation system (not shown) can be connected to the connector 116, such as connected to a connector or fitting. Connector 116 can allow material to pass through instrument 66, such as handle 120, and into a balloon or other expandable portion 122 at or near the distal end 110 of instrument 66. As is generally understood by those skilled in the art, inflation of balloon 122 can expand to dilate or expand a portion of the sinus.

[0041] In various embodiments, user 12 can grasp instrument 66, such as using hand 130. Then, user 12 can move instrument 66 relative to patient 14, such as within patient 14, so as to move the distal end 110 of the instrument into a sinus such as maxillary sinus 134. However, it should be understood that various instruments including instrument 66 according to various configurations can also be used to access one or more portions of other sinuses of patient 14, such as frontal sinus 136 and / or sphenoid sinus 138.

[0042] The instrument 66 may further include a tracking device 146. The tracking device 146 may be attached to the instrument 66, such as to the elongate handle portion 120. Generally, the tracking device 146 is substantially fixed relative to the handle 120 such that movement of the handle 120 by the user 12 (e.g., the user's hand 130) moves the handle 120 and the tracking device 146. According to various embodiments, the tracking device 146 is also rigidly fixed in space relative to the distal end 110 of the instrument 66. Thus, knowing the position (e.g., orientation and / or direction) of the tracking device 146 will allow the position of the distal end 110 to be known. Additionally, the distal end 110 may extend along an axis 110a that is angled 150 relative to the axis 120a of the handle 120. Thus, the tracking device 146 may be positioned and have a known or predetermined position and geometry relative to the distal end 110 to be tracked to determine the position of the distal tip 110 relative to the tracking device 146 attached to the handle 120.

[0043] In addition to the instrument 66 that may be used by the user 12, additional instruments may be used relative to the subject 14. By way of example, as Figure 2B shown, the user 12 may include or operate an imaging system 137. The imaging system 137 may include a distal image capture portion 139, such as a camera lens or a camera. The imaging instrument 137 may generally be understood to be an endoscope, such as the EVIS EXERAIII endoscope sold by Olympus America. According to various generally known techniques and embodiments, the endoscope 137 may be positioned by the user 12 relative to the subject 14. The image collection portion 139 may image various portions of the subject 14, such as the internal portion of the subject 14 including the sinus cavity 134, or other suitable portions including the nasal passage 112. As further discussed herein, the endoscope 137 may capture images that are substantially real-time, such as during positioning of the instrument 66 within the subject 14. The real-time images captured using the endoscope 137 may be displayed on various display devices or viewing systems, such as the display device 20 and / or the AV 21. Thus, the endoscope 137 may be used to capture and display images at the imaging portion or end 139 according to generally known techniques. The images may be sent to a processing system 22 via various systems such as a wireless or wired transmission system for display on a selected display or viewing system including the display device 20 and / or the AV 21. Signals from the endoscope 137 may be digital signals and / or analog signals and may be sent directly from the endoscope and / or via an interface system 76. In any case, the user 12 and / or any other suitable individual may view the images obtained at the imaging portion 139 of the endoscope 137. Additionally, images may be captured and recorded for various purposes.

[0044] As discussed above, the endoscope 137 can be used to obtain images of the subject 14. To assist in obtaining images or performing procedures, the position of the endoscope, and in particular the position of the image being obtained, can be determined by one or more tracking devices. For example, an endoscope tracking device 147 can be coupled to the endoscope 137, similar to the tracking device 146 coupled to the instrument 66, as discussed above. The tracking device 147 can include one or more visible markers or portions 147a, similar to the markers 146a on the tracking device 146. Thus, as discussed herein, the tracking device 147 can be viewed or imaged with the AV 21 to be tracked by a selected tracking system, as further discussed herein. The tracking device 147 can be used to determine the position of the end 139 that captures the image to assist in determining the position of the image within the patient 14. The endoscope 137 can be positioned within the subject 14, such as within the sinus cavity 134 that is not directly viewable by the user 12. Additional and / or alternative tracking devices can include an end tracking device 143 that can be positioned at or near the image capture end 139 or coupled to the endoscope 137. The tracking device 143 can be similar to the tracking device 64 discussed above. The tracking device 143 can be an optical tracking device, an EM tracking device, an ultrasonic tracking device, or other suitable tracking device. As further discussed herein, registration of the instrument 137, such as with the tracking device 143, and registration of the patient or subject tracker 64 can be used to assist in registering and maintaining the registration of the endoscope 137 relative to the subject 14.

[0045] The user 12 can also have and / or use an alternative or enhanced view screen or viewing system 21 for use during the procedure. The AV 21 can be a suitable device that includes at least one view screen and typically includes two view screens, including a first view screen 160 and a second view screen 162. The view screens can be fixed to a frame member 166 that can have one or more temple members 168 to allow the AV 21 to be worn by the user 12 in a manner similar to glasses. Thus, the view screens 160, 162 can typically be positioned in front of the two eyes 172 and 174 of the user 12. In this way, images can be displayed on one or both of the view screens 160, 162 to allow the user 12 to view the images. The AV 21 can include one or more various devices and systems, such as a wearable computer peripheral configuration sold by Microsoft Corporation, an R-9 Smartglasses wearable computer peripheral configuration sold by Osterhout Design Group having a place of business in San Francisco, California, and / or a DAQRI sold by DAQRI having a place of business in Los Angeles, California Wearable computer peripheral configuration.

[0046] In various embodiments, in addition to the portions that display images (e.g., icons or renderings), the view screens 160, 162 may also be substantially transparent. Thus, the user 12 can view the patient 14 as well as any images displayed by the view screens 160, 162. Additionally, since the two view screens 160, 162 display the selected images, the user 12 can perceive the display as being substantially stereoscopic and / or three-dimensional relative to the patient 14. Thus, as further discussed herein, the user 12 can view the patient 14 and the images while performing a procedure.

[0047] The AV 21 may also include one or more cameras, such as a first camera 180 and a second camera 182. The two cameras 180, 182 can be used to view the area of interest, such as the head of the patient 14. As Figure 2A shown, when performing a procedure in the surgical area, such as in the sinuses of the patient 14, the user 12 can substantially view the head and neck of the patient. Thus, the cameras 180, 182 can also view the patient 14 for various purposes, as further discussed herein. Additionally, the cameras can view other objects in the area of interest, such as the tracking device 146 and / or the marker 100 on the instrument 66. The tracking device 146 can include one or more viewable markers or portions 146a that can be viewed by the cameras 180, 182 for the AV 21 to determine the perspective or view of the tracking device 146.

[0048] Although this disclosure and discussion use two cameras 180, 182 to view and determine the position of the tracking device 146 and / or the marker 100, those skilled in the art should understand that tracking may require only one camera, such as only one of the cameras 180, 182, as discussed herein. Based on various features (e.g., shape, image, etc.) on the tracking device 146, the tracking device 147, and / or the marker 100, a single camera (such as camera 180) can be used to determine the position (i.e., x, y, z coordinates and orientation) relative to the camera 180 and / or relative to other trackable objects. For example, the camera 180 can be used to determine the relative position of the tracking device 146 (and thus the instrument 66) relative to the marker 100. Additionally, the camera 180 can be placed at any position relative to the user 12, such as separated from the user 12's head and the AV 21. However, the camera 180 can still communicate with the processor system 22 to display various images on one or more of the view screens 160, 162.

[0049] Thus, the AV 21, including cameras 180, 182, can view the marker 100 on the patient 14 in conjunction with the tracking device 146 on the instrument 66. The marker 100 on the patient 14 can be viewed by the cameras 180, 182 and can be identified by the user 12 and / or substantially automatically identified by executing instructions on a processor system (such as by using the processor 26 to execute instructions).

[0050] As discussed above, the processor system 22 can access instructions, such as those stored on the memory 27, to assist in identifying the marker 100 in the image. The cameras 180, 182 can have a field of view that includes a region of interest that includes the head of the patient 14 and also views the marker 100. Instructions that can be included in the selected software can identify the marker 100 in the viewed image, such as by segmenting the image and identifying the selected shape, density, color, etc. of the marker 100.

[0051] Once the marker 100 is identified, the images acquired using the imaging system 14 can be registered, such as using the processor system 22, to register the image that includes the marker 100 in the field of view of the cameras 180, 182 of the AV 21. Registration can occur by matching the identified marker 100 in the image data acquired by the image device 16 with the marker 100 in the field of view image acquired using the cameras 180, 182 of the AV 21. During the acquisition of the image data using the imaging system 16 and when in the field of view of the cameras 180, 182 during the procedure, the marker 100 generally remains in the same position on the patient. Registration can also occur due to the tracking device 143 on the endoscope 137 and the patient tracker or dynamic reference frame 64. As discussed above, registration can occur due to various registration techniques, such as those disclosed in U.S. Patent No. 8,238,631, which is incorporated herein by reference. Registration can be performed by tracking the tracking device 143 (associated with the endoscope 137) and / or the patient tracker 64. Additionally, the AV 21 can include a tracking device 149, which can allow the AV 21 to be tracked in the same or a similar tracking system and / or reference frame with respect to the subject or dynamic reference frame 64 and / or the tracker 143 on the endoscope 137. Thus, the AV 21 can be registered with respect to the endoscope 137 and / or the subject 14. This image acquired using the endoscope 137 can be used for display with respect to the view of the user 12 or for viewing by the user 12, as further discussed herein.

[0052] As discussed above, reference Figure 2A and Figure 2B, the image data and / or images obtained using the imaging system 16 in which the marker 100 is connected to the patient 14 will include data of the marker 100. As discussed above, the marker 100 can be identified in the image data or images obtained using the imaging system 16. The marker 100 can also be viewed by the cameras 180, 182 associated with the AV 21. The cameras 180, 182 can also view the patient 14 to identify or assist in identifying anatomical fiducial markers. Nevertheless, the marker 100 identified in the image data obtained using the imaging system 18 can also be identified in the images obtained using the cameras 180, 182. Those skilled in the art should understand that the images obtained using the cameras 180, 182 can be any suitable type of image, such as a color image, an infrared light image, etc. Nevertheless, the matching of the marker 100 identified in the image data obtained using the imaging system 16 can match the identification position of the marker viewed using the cameras 180, 182 to register the space or field of view viewed by the AV 21 with the image space of the images obtained using the imaging system 16. Therefore, as further discussed herein, the images or portions of the images obtained using the imaging system 16 can be displayed together with the view screens 160, 162 to appear as if superimposed on the patient 14.

[0053] Due to the subject tracker 64 and the tracker 143 on the endoscope 137, the subject 14 can also be registered relative to the currently obtained images or the real-time images obtained using the endoscope 137. The subject 14 can be registered, for example, using the fiducial marker 100d and / or other fiducial features. Therefore, the real-time images obtained using the endoscope 137 can also be registered with the pre-acquired images. Therefore, the pre-acquired images can be registered with the instrument 66, for example, using the cameras 180, 182 and / or registered with the AV 21 via the AV 21 tracker 149. Therefore, the images obtained using the endoscope 137 can be registered with the pre-acquired images of the subject 14 and used for viewing by the user 12 (for example, using the AV 21).

[0054] The user 12 can view the subject 14 through the AV 21, as Figure 3A and Figure 3B shown. The user 12 can view or have a field of view 21f, as Figure 3A and Figure 3B indicated by the dashed lines in. The field of view 21f can represent the view of the user 12 through the AV 21 when viewing the subject 14, or any area through the AV 21. Therefore, the user 12 can view the real or physical subject 14, and the view can also be enhanced by the graphical representations (also referred to herein as icons) displayed by the AV 21. When viewing the field of view through the AV 21, the icons or graphical representations can be displayed in the field of view 21f for viewing by the user 12.

[0055] In addition to the icons, as discussed elsewhere in this document, additional images or image regions 153 can be displayed within the AV field of view 21f for viewing by the user 12. The supplementary viewing region 153 can be used to display various images or information for use or viewing by the user 12. For example, real-time images obtained within the endoscope 137 can be displayed in the auxiliary or enlarged viewing region 153. Thus, the user 12 can view the subject 14 within the field of view 21f to view various graphical representations of the subject 14, as well as additional images (such as endoscope images) in the auxiliary or additional viewing region 153, as further discussed in this document. The user 12 can also selectively select or choose the information to be displayed in the auxiliary display region 153, such as pre-acquired images, real-time images using the endoscope, or other appropriate information.

[0056] Additionally referring Figure 3A and Figure 3B , for example, a sinus such as the maxillary sinus 134 can be displayed as a maxillary sinus icon 134'. Figure 3A is an illustration of the perspective from which the user 12 views the patient 14 via the AV 21 and the respective parts displayed on the view screens 160, 162. Thus, as described above, the maxillary sinus icon 134' can be displayed for the user 12 to view as if the user 12 could see into the patient 14 and view the maxillary sinus 134. The maxillary icon 134' can be a graphical rendering of the image data or an artificially created icon representing the maxillary sinus.

[0057] It can also be selected to illustrate other anatomical structures of the patient 14 such as the frontal sinus 136 and one or more sphenoid sinuses 138. The user 12 can also view any real-world objects such as the patient 14 and / or the marker 100 fixed to the patient 14. The user can also view other real-world parts, such as the patient support frame 15. Thus, by viewing through the transparent portions of the view screens 160, 162, the user 12 can view the features superimposed on the patient 14 due to the screens 160, 162 as well as the objects in the real world.

[0058] Additionally, the cameras 180, 182 can view the tracking device 146. By viewing the tracking device 146, the cameras 180, 182 can determine the position of the tracking device 146 relative to the marker 100. The position of the marker 100 is placed on the patient 14 to identify the position of the patient 14. The known position of the instrument tracking device 146 relative to one or more of the markers 100 allows for the determination of a portion of the instrument 66 relative to the tracking device 146 and the patient 14. As discussed above, the distal end 110 of the instrument can be in a known and fixed position relative to the tracking device 146. Thus, the known and fixed relative position (e.g., geometry) of the distal end 110 relative to the tracking device 146 can be stored in the memory 27 or other suitable memory.

[0059] The tracked position of the tracking device 146 can be determined by triangulating the position of the tracking device 146 with one or more of the cameras 180, 182 based on the "view" of the tracking device. The processor system 22 can execute instructions commonly understood in the art to then determine the position of the distal end 110 and / or the working portion (such as the inflatable member of 122), and by displaying it on the view screens 160, 162, the instrument icon can be shown to include or show the respective parts relative to the patient 14. The instructions executed by the processor system 22 can include instructions stored in and called from the memory 27. The instructions can include those instructions based on an algorithm for triangulating the position of the viewed part (such as the tracking device 146) based on the separate views from the two cameras 180, 182. The separate views can be used to generate signals (e.g., including image data) from the two cameras 180, 182, and the signals can be sent to the processor system 22. The triangulation of the position of the tracking device 146 can be based on the known distance between the two cameras 180, 182 and each separate view captured by each of the two cameras 180, 182.

[0060] Accordingly, the view screens 160, 162 can include graphical representations, also referred to as the icon 110' of the distal end and / or the icon 122' of the inflatable portion of the instrument 66. Icons, such as the distal end icon 110', can be shown as an icon on one or more of the view screens 160, 162 as it appears superimposed or displayed relative to the patient 14. It should be understood that icons of more than one part of the instrument 66 can be used, and thus, the instrument icon 66' can be shown as the entire instrument including all parts of the instrument 66.

[0061] The AV 21 can communicate with the processor system 22 and / or can include on-board processing and / or other communication features to communicate with other processor systems. Accordingly, views of an area of interest such as the head of the patient 14 can be sent to the processor system 22 via the cameras 180, 182 of the AV 21. At least due to the spacing of the cameras 180, 182, triangulation can be determined for each viewing point in space, such as the markers 100 and / or the tracking device 146 relative to the cameras 180, 182. The relative position of the tracking device 146 relative to one or more of the markers 100 can be determined, such as by executing instructions with the processor system 22.

[0062] A processor system 22 that receives images from one or both of two cameras 180, 182 can process and determine distances between various tracked or any viewed portions, such as fiducial 100 and tracking device 146. Thus, the processor system 22 that executes instructions accessed in memory 27 can then provide selected and / or appropriate image portions, such as instrument icon 66' or a part thereof, and / or other imaged features, such as an icon representing a sinus including frontal sinus 136' or other appropriate portions from the image data, to view screens 160, 162. Registration of pre-acquired images, such as those acquired with imaging system 16 based on the tracking position of tracking device 146 and / or fiducial 100, can be based on known registration techniques, such as those disclosed in U.S. Patent No. 8,238,631, which is incorporated herein by reference. The registration can be substantially automatic and / or based on the identification of fiducial marks, such as fiducial 100, in images 18 and / or on patient 14.

[0063] Accordingly, user 12 can view patient 14 and other features, such as instrument icon 66', relative to patient 14 based on view screens 160, 162. Cameras 180, 182 can provide all tracking information relative to user 12 and patient 14 for determining the positions of various portions of instrument 66, such as distal tip 110, for display thereof on view screens 160, 162. The perception of user 12 can be that even though instrument 66 is within patient 14, it is visible relative to patient 14. Additionally, image data acquired with imaging system 16 can be displayed on view screens 160, 162 as features relative to patient 14, such as icons. Similarly, the perception of user 12 of patient 14 can be that due to AV 21, user 12 can view various portions, such as sinuses 134, 136, 138. Thus, as Figure 3A shown, the view of patient 14 can be enhanced to show features that user 12 would not otherwise be able to see with user 12's eyesight. In other words, user 12 can view patient 14 in physical space, as Figure 3A shown, and as a representation of regions within patient 14 that can be viewed with the icons or renderings discussed above. The view can also be 3D and can undergo angular changes as the user moves relative to patient 14 and / or instrument 66.

[0064] As in Figure 3AAs specifically shown, the patient 14 can be viewed through the view screens 160, 162. Various icons can be displayed relative to the icon 66' of the instrument 66, such as the maxillary icon 134' and the sphenoid icon 138'. These icons can have various selected opacities and / or cross-sections to view the instrument icon 66' relative to the anatomical structure icon (such as the sphenoid bone icon 138'). Thus, a user 12 viewing a field of view including icons such as the sphenoid bone icon 138' and the instrument icon 66' can see both icons simultaneously. In addition, the user 12 can perceive the position of the instrument 66 within the selected sinus (such as the sphenoid sinus 138) by viewing the instrument icon 66' and the sinus icon (such as the sphenoid sinus icon 138') substantially simultaneously. In other words, the opacity of various icons (such as, the sinus icon) can be selected to have a transparent view, enabling the viewing of the instrument icon within or as if within the selected anatomical portion. This allows the user 12 to view the patient 14 and the icons of the instrument and the anatomical structure substantially simultaneously and as if present on the patient 14, as Figure 3A shown.

[0065] In addition to or alternatively, various icons can be displayed at a location remote from the patient 14. For example, as Figure 3B shown, the instrument icon 66' can be displayed remote from the patient 14, but at a tracked and determined position relative to the anatomical part icon such as the sphenoid bone icon 138'. Only those anatomical parts that interact with or have been passed through by the instrument 66 can be selected for display, so it may not be necessary to show all icons. It should be understood that even when the instrument 66 is within the nasal passage 112 and not clearly visible from the non-enhanced view of the user 12, various passageways, such as the icon passageway 139' through the nasal passage 112 between various parts of the anatomical structure (see Figure 3A and Figure 3B ) are also the case. Thus, as Figure 3B shown, it should be understood that the displayed portion of the anatomical structure representing or based on the acquired image data of the patient 14 can be displayed at a location remote from the corresponding and relative physical position on the patient 14. Thus, an icon such as a sinus icon can be displayed at a distance from the patient 14. This can allow the user 12 to obtain a more unobstructed and / or substantially unobstructed view of the patient 14 while still being able to view the relative position of the instrument 66 relative to the selected anatomical part.

[0066] In addition, the view screens 160, 162 can be used to display other images, such as endoscopic images, which can be acquired substantially simultaneously and in real time if so selected. That is, the user 12 can also place the endoscope in the nasal passage 112, and one or more of the view screens 160, 162 can display the endoscopic view. Thus, it should be understood that the user 12 can position the endoscope together with the instrument 66 through the nasal passage 112 to provide a real-time and endoscopic viewing point, which can also be displayed on the view screens 160, 162 and relative to the selected icons such as the sinus icon and / or the instrument icon 66'.

[0067] In addition, it should be understood that various images can be displayed on both view screens 160 and 162, or on only one of the view screens 160, 162. It should be understood that the images displayed on the two view screens 160, 162 can be substantially similar, but altered to allow the user 12 to perceive the depth and / or three-dimensionality of the selected portions such as the sinuses and / or the instrument 66 based on the image data and / or the icons. Thus, the displays 160, 162 can have the same display, substantially different displays, or only one display per view screen, or can be similar to provide depth perception for the user 12 to view.

[0068] As discussed above, the auxiliary image 153 can show or illustrate the position of the instrument 66, such as the distal end image 110” shown in the auxiliary image 153. The auxiliary image 153 can be a real-time image acquired with the endoscope 137, as discussed above. Thus, the distal end image 110” can also be a real-time image of the instrument 66. The auxiliary image 153 can also display the surface of the anatomical structure, such as within the sinus, for the user 12 to view during the selected procedure. Thus, the field of view 21f can allow the user 12 to view the subject 14, a graphical representation of the instrument displayed relative to and / or superimposed on the subject 14, a pre-acquired image of the subject displayed relative to and / or superimposed on the subject 14, and / or an auxiliary image, such as a real-time image of the instrument 66. Thus, the user 12 can select which images to view in the field of view 21f. It should be understood that any image or graphical representation can also be displayed on various other display devices such as the display device 20. The display device 20 can also view or display, individually or in combination, a graphical representation of the position of the instrument 66, a pre-acquired image, and a real-time image.

[0069] Reference Figure 3C, the field of view can also be displayed on the display device 20. In various embodiments, the view of the endoscope can be displayed as an auxiliary view 153' on the display 20. The icon 110' and portions of the anatomical structure, such as the sinus 134', can also be displayed on the display device 20. When displayed on the display device 20, the graphical representation can be substantially three-dimensional (3D). Thus, the field of view display 21f can be substantially reproduced on the display device 20, although the patient 14 may not be displayed, but only the acquired images such as the sinus 134' and the images acquired in real time with the endoscope 137 can be displayed. It should be understood that the display device 20 can be movable and positioned for the best view of the user 12.

[0070] Whether displayed on the display 20 and / or displayed with the AV 21 in the field of view 21f, the display of the graphical representation (e.g., the sinus 134' and the instrument 110') can be from the perspective of the user 12. Thus, when the user 12 moves relative to the subject 12, the display in the field of view 21f and / or on the display device 20 can change to provide a display for the user 12 as if the user 12 were viewing from the selected position inside the subject 12. In other words, if the user 12 moves to a position at the head of the subject 12 to view from below rather than above, the display of the graphical representation will be changed to match the position of the user 12 relative to the subject 14. In various embodiments, the determined position of the user 12 can be determined by the tracker 149 and / or the views of the imaging devices 180, 182 associated with the AV 21.

[0071] Continue to refer to Figure 1 - 3B And additionally refer to Figure 4, a method for a user 12 to use an AV 21 to assist in performing a procedure, such as a procedure relative to one or more of the sinuses including the frontal sinus 136, is described in the flowchart 210. Generally, the process can start at the start box 214. After starting the process at the start box 214, acquisition of image data or an image of the subject 14 can be performed. The acquisition of image data or the image can be performed in any suitable manner. For example, an image of the subject 14 can be acquired and stored on a storage system such as storage system 27 and / or storage system 33b at a selected time before performing the procedure, such as when the patient 14 is ready to have the instrument 66 introduced into the patient 14. The image can be acquired using a selected imaging system such as a CT scanner and / or an MRI and saved in a suitable format such as raw data and / or a reconstructed image. The reconstructed image can include an image that has been rendered in three dimensions for the user 12 to view using different display devices such as the display device 20 and / or the AV 21. Additionally, individual portions of the image or image data can be segmented, such as segmenting the sinuses including the frontal sinus 136 from other image data. Furthermore, identification of the marker 100 in the image or image data can be performed, such as by a processor including processor 26 and / or 33a. Various segmentation techniques can be used to perform segmentation of individual portions of the anatomical structure (such as the frontal sinus 136) and / or identification of the marker 100. The segmentation techniques can include those incorporated in various imaging and navigation systems such as the FUSION TM navigation system sold by Medtronic.

[0072] The image data can also be acquired substantially during the procedure or just before the procedure, such as using the imaging device 14, which can be used substantially during the surgery (e.g., when the patient is ready for the procedure). Individual portions of the image or image data can be segmented as discussed above. Instead of being stored in the memory for a selected period of time before the procedure, the data can be transmitted substantially in real time to the processor system 22 for use during the procedure. However, it can be understood that the image data can be stored for a selected period of time, such as to analyze and / or process the image data or the image for use during the procedure.

[0073] The acquisition of the image data can be optional because it is not necessary to prepare the image for display by the AV 21 and / or for use during the procedure. For example, as discussed herein, a system including the AV 21 and the processor system 22 can track the marker 100 and the tracking device 146 to represent the position of the instrument without image data of the patient 14. In various embodiments, the processor system 22 can access the image for display by the AV 21 as discussed above.

[0074] In block 224, the AV 21 can be positioned for the procedure. Positioning the AV 21 for the procedure can include positioning the AV 21 on the user 12, as discussed elsewhere herein. Additionally, the AV 21 can be positioned to communicate with the processor system 22, such as to provide processing power for tracking the patient 14 with markers 100 and / or the instrument 66, such as with the tracking device 146. Thus, the AV 21 can view the region of interest in block 226, and the user can confirm the ability to view the region of interest in block 228. When viewing the region of interest, the cameras 180, 182 can be capable of viewing at least the portion of the patient 14 on which the procedure is to be performed, such as typically the head region. As discussed above, the user 12 may not be able to substantially view the operative area through the various external tissues of the patient 14. Thus, the area of the procedure can include the sinuses, such as the frontal sinus 136, and the region of interest can include the entire head of the patient 14. Accordingly, the cameras 180, 182 can be positioned to view the region of interest, and the user 12 can confirm viewing the region of interest through the view screens 160, 162. When no icons are displayed on a portion of the view screens 160, 162, the view screens 160, 162 are substantially transparent.

[0075] Once the AV 21 has a view of the region of interest, recording of the region of interest can be performed with the AV 21 cameras in block 232. Recording the region of interest in block 232 can allow for collection of images with the cameras 180, 182 (however, it should be understood that more than two or fewer than two cameras can be used). Recording the region of interest can include imaging at least a portion of the patient 14 in a continuous manner, such as throughout the procedure. Imaging the region of interest of the patient 14 can include imaging the markers 100 and / or other fiducial points or portions of the patient 14. Thus, in block 236, the recorded region of interest can include identifying the patient markers.

[0076] Identifying the patient markers can include segmenting the image data recorded at the region of interest in block 232 to identify the patient markers 100 in the image. The identified patient markers 100 can be displayed as icons on the view screens 160, 162, such as together with the icon 100a', which can include a three-dimensional cube positioned above the marker 100a located on the patient 14 when viewed by the user 12, as shown in FIG. 3. However, identifying the patient markers in block 236 may not require or provide the display of the icon 100a', but rather can be simply performed by the processor system 22 to identify the markers and thereby identify the region of the patient 14, such as for identifying the position of the instrument 66 or a portion thereof (such as the distal end 110), as discussed elsewhere herein.

[0077] Identifying the marker 100 on patient 14 allows a processor system, such as part of a navigation system, to track patient 14 when the marker 100 is within the field of view of the cameras of the AV 21. The marker 100 can include parts that are recognizable in the images acquired by cameras 180, 182, such as color, pattern, shape, etc. Additionally, the marker 100 can include features recognizable for determining position, including the pose, position, and orientation of the marker relative to the AV 21. Thus, patient 14 can be tracked relative to the AV 21 worn by user 12.

[0078] In block 240, the instrument tracker 146 can be identified. The instrument tracker 146 can include parts that are recognizable in the images acquired by cameras 180, 182, such as color, pattern, shape, etc. Additionally, the instrument tracker 146 can include features recognizable for determining position, including the pose, position, and orientation of the marker relative to the AV 21. For example, the tracking device 146 can include a pattern that can be seen by cameras 180, 182. The pattern on the tracking device 146 can be substantially or completely unique when viewed from different angles relative to the tracking device. Thus, the viewing pattern on the tracking device 146 can be used to determine the position of the instrument tracker 146 and, thus, the position of the instrument 66.

[0079] The instrument tracker 146 can be fixed to the instrument 66, as discussed above. The geometric relationships between the various parts of the instrument 66, such as the distal tip 110 and / or the operating part 122, can be predetermined and input for processing by the processor system 22. In various embodiments, the geometry can be stored in the memory 27 and invoked due to the automatic identification of the instrument 66 (e.g., by viewing the instrument with cameras 180, 182) and / or the identification of the instrument input by the user 12. However, the AV 21 can be used to view the tracking device 146 to determine the position, including position (e.g., three-dimensional coordinates) and orientation in various degrees of freedom (e.g., three degrees of freedom). The tracked position of the instrument 66 can be used by the processing system 22 for various purposes.

[0080] For example, as shown in FIG. 3, executed in block 244, the instrument icon 66' can be displayed with the AV 21, such as on one or more of the view screens 160, 162. Except for the part showing the icon, the view screens 160, 162 can be substantially transparent. The icon 66' and / or parts of the instrument, such as the distal tip icon 110' and / or the operating part 122', can be shown on the view screens 160, 162 relative to the patient 14. The user 12 can then view the patient 14 and one or more icons through the view screens 160, 162. Thus, the user 12 can view the position of at least a part of the instrument 66 relative to the patient 14 (including a part of the patient 14).

[0081] In block 248, a display of a subject portion icon may be selectively or alternatively shown. For example, as Figure 3A and / or Figure 3B shown, a frontal sinus 136' icon may be shown. The frontal sinus icon 136' may be shown relative to the instrument icon 66' and the patient 14. Thus, the user 12 may view the patient 14, the instrument icon 66', and the frontal sinus icon 136'. Since the tracking marker 100 is on the patient 14, the relative position of the instrument 66 may be shown on the view screens 160, 162 together with the instrument icon 66'. Additionally, since the pre-acquired image is registered to the patient using the marker 100, the relative position of a subject portion (such as the frontal sinus 136) may be shown, as discussed above.

[0082] Similarly, because the AV 21 is able to track the patient 14 due to the marker 100, the relative positions of the instrument 66 and a subject portion such as the frontal sinus 136 may be updated and shown on the view screens 160, 162 substantially in real time for viewing by the user 12 with the subject 14. It should be understood that icons such as the instrument icon 66' and the subject portion icon 136' may be generated and shown on the view screens 160, 162, and the user is able to view the patient 14 in real time and in physical space through the view screens 160, 162. It should also be understood that the icons may be shown on one or both of the view screens 160, 162 as selected by the user 12.

[0083] As discussed above, user 12 may also optionally choose to have a live image displayed in frame 252. The live image may be an image obtained with endoscope 137, as discussed above and as is generally understood by those skilled in the art. The live image may include the surface of subject 14, such as an inner surface. Additionally or alternatively, the image may include a display or image of instrument 66 (such as the distal end 110” of instrument 66). User 12 may choose to have auxiliary image 153 displayed in the field of view 21f or on any suitable display, such as display device 20. User 12 may also choose to turn off or not display auxiliary display 153, such that user 12 only uses subject 14 and the selected augmented reality portions, such as the graphical representations or icons discussed above. Additionally, it should be understood that user 12 may choose to have the graphical representation displayed in auxiliary display area 153 and have the live image displayed superimposed or in the field of view 21f relative to subject 14. As discussed above, due to the selected reference portions and / or markers on subject and patient or subject tracker 64, the image obtained with endoscope 137 may be registered relative to subject 14. Thus, user 12 may view icons relative to instrument 66, icons relative to the selected sinuses or internal portions, partially or due to pre-acquired images (e.g., MRI and / or CT images) and live images obtained with endoscope 137 or other suitable imaging systems.

[0084] In block 262, a procedure may be performed by viewing subject 14 and the selected icons due to AV 21 (such as instrument icon 66' and / or frontal sinus icon 136'). Additionally, it should be understood that other suitable icons may be displayed, such as maxillary sinus icon 134' and / or sphenoid sinus icon 138'. Further, due to the various tracking devices associated with the instrument, additional instrument icons may also be displayed. Additionally, different instruments may have different geometries, which may also be input and / or invoked before displaying the icons on the display device including AV 21. The method 210 may then end at block 272, which includes various other procedures, such as various hemostasis and / or closure procedures.

[0085] Providing example embodiments enables the present disclosure to be comprehensive and conveys the scope fully to those skilled in the art. Many specific details are set forth, such as examples of specific components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed and that example embodiments may be embodied in many different forms and neither should be construed as limiting the scope of the present disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies have not been described in detail.

[0086] The foregoing description of the embodiments has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure. The individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in a selected embodiment, even if not specifically shown or described. Many modifications may be made in various ways. Such variations are not to be regarded as a departure from the disclosure, and all such modifications are intended to be included within the scope of the disclosure.

Claims

1. A system for viewing a subject during surgery on the subject, comprising: A wearable view screen and camera system configured to be worn by a user and having: A first view screen configured to be positioned adjacent to the first eye of the user; A second view screen configured to be positioned adjacent to the second eye of the user; And At least a first camera configured to have a first field of view; An instrument tracking device fixed to an instrument and viewable by the first camera; A subject marker viewable by the first camera, wherein the subject marker is a real-world object configured to be attached to the subject and used as a reference point; And A processor system configured to execute instructions (i) to determine the relative position of the instrument tracking device relative to the subject marker based on a first signal from the first camera and thereby determine the relative position of the instrument relative to the subject marker, and (ii) to generate an instrument representation to be displayed on at least one of the first view screen or the second view screen of the wearable view screen and camera system to be viewed by at least one of the first eye or the second eye; Wherein the first view screen or the second view screen is substantially transparent and configured to allow the user to view the subject through the first view screen or the second view screen; Wherein the processor system is further configured to generate an instrument icon and an anatomical structure icon representing the instrument representation and display the instrument icon and the anatomical structure icon at a position spaced apart from the subject on at least one of the first view screen or the second view screen to allow the user to have a substantially unobstructed view of the subject when viewing the instrument icon and the anatomical structure icon.

2. The system for viewing a subject during a surgical procedure on the subject according to claim 1, wherein, The wearable view screen and camera system further comprises: A second camera configured to have a second field of view; Wherein the processor system is further operable to execute instructions (i) to determine the relative position of the instrument tracking device relative to the subject marker based on the first signal from the first camera and a second signal from the second camera regarding the view of the instrument tracking device relative to the subject marker.

3. The system for viewing a subject during surgery on the subject according to claim 2, wherein, When displayed on the first view screen and the second view screen, the instrument representation is perceived by the user as a single stereoscopic representation.

4. The system for viewing a subject during surgery on the subject according to claim 3, wherein, The instrument representation appears to the user as part of the instrument in the user's field of view, wherein the user views both the instrument representation and the subject through the first view screen and the second view screen.

5. The system for viewing a subject during surgery on the subject according to claim 3, further comprising: A memory storing an image of at least a part of the surgical area of the subject; Wherein the image includes an image portion representing the subject marker attached to the subject; Wherein, the processor system executes additional instructions to (i) register the image of at least a portion of the surgical area, and (ii) display an anatomical structure representation using at least one of the first view screen or the second view screen; Wherein, the anatomical structure representation is configured to appear in the user's field of view related to the subject; Wherein the user can view both the anatomical structure representation and the subject in the user's field of view.

6. The system for viewing a subject during surgery according to claim 5, wherein both the instrument representation and the anatomical structure representation are configured to appear in the user's field of view related to the subject.

7. The system for viewing a subject during surgery on the subject according to claim 2, wherein, The first field of view and the second field of view overlap.

8. The system for viewing a subject during surgery according to claim 1, further comprising an instrument that is movable relative to the subject and connectable to the instrument tracking device.

9. A system for viewing a subject during surgery, comprising: A wearable view screen and a camera system, which are configured to be held relative to the user's first eye and have: A mounting structure; A first view screen, which is configured to be positioned close to the user's first eye and fixed to the mounting structure; And A first camera, which is configured to have a first field of view; An instrument tracking device fixed to the instrument, which can be viewed by the first camera; A subject marker, which can be viewed by the first camera, wherein the subject marker is a real-world object configured to be attached to the subject and used as a reference point; And A processor system, which is configured to execute instructions (i) to determine the relative position of the instrument tracking device relative to the subject marker based on a first signal from the first camera regarding the view of the instrument tracking device relative to the subject marker, and thereby determine the relative position of the instrument relative to the subject marker, and (ii) generate an instrument icon to be displayed on the first view screen of the wearable view screen and camera system to be viewed by the first eye, Wherein, the first view screen is substantially transparent and configured to allow the user to view the subject through the first view screen; Wherein the processor system is further configured to generate the instrument icon and the anatomical structure icon, and display the instrument icon and the anatomical structure icon at a position spaced apart from the subject on the first view screen, so as to allow the user to have a substantially unobstructed view of the subject when viewing the instrument icon and the anatomical structure icon.

10. The system for viewing a subject during surgery according to claim 9, wherein the wearable view screen and camera system further comprises: A second camera, which is configured to have a second field of view and fixed to the mounting structure; Wherein, the first field of view is different from the second field of view; Wherein, the first camera is fixed to the mounting structure; and A second view screen configured to be positioned adjacent the user's second eye.

11. The system for viewing a subject during a surgical procedure on the subject according to claim 10, wherein, The wearable view screen and camera system are configured such that the user can view the subject through the first view screen and the second view screen simultaneously.

12. The system for viewing a subject during surgery on the subject according to claim 10, wherein, The wearable view screen and camera system are configured such that the user can view the subject, the instrument icon, and the auxiliary image through the first view screen and the second view screen simultaneously.

13. The system for viewing a subject during surgery on the subject according to claim 12, wherein, The processor system is configured to execute additional instructions to register (i) a pre-acquired image of the subject with (ii) a view of the subject using the first camera and the second camera; wherein at least a portion of the pre-acquired image registered with the subject is displayed using at least one of the first view screen or the second view screen; wherein the instrument icon is shown using at least one of the first view screen or the second view screen at a determined position of the instrument relative to at least a portion of the pre-acquired image.

14. A system for viewing a subject during a surgical procedure on the subject, the system comprising: A first real-time image capture system operable to obtain an image of the subject; A wearable view screen and camera system configured for wear by a user and having: A first view screen configured to be positioned adjacent the user's first eye; A second real-time image capture system configured to have a first field of view; An instrument tracking device attachable to an instrument and viewable by the second real-time image capture system; A subject marker viewable by the second real-time image capture system, wherein the subject marker is a real-world object configured to be attached to the subject and used as a reference point; An auxiliary display device separate from the wearable view screen and camera system, wherein the auxiliary display device is separable from and viewable separately from the first view screen; A processor system configured to execute instructions (i) to determine a relative position of the instrument tracking device relative to the subject marker based on a first signal from the second real-time image capture system and thereby determine a relative position of the instrument relative to the subject marker, and (ii) to generate an instrument representation for display using at least the auxiliary display device, wherein the first view screen is substantially transparent and configured to allow the user to view the subject through the first view screen; wherein the processor system is further configured to generate an instrument icon and an anatomical structure icon representing the instrument representation and display the instrument icon and the anatomical structure icon at a position spaced apart from the subject on the first view screen to allow the user to have a substantially unobstructed view of the subject while viewing the instrument icon and the anatomical structure icon.

15. The system for viewing a subject during surgery on the subject according to claim 14, further comprising a tracking system configured to track at least the first real-time image capture system and the subject tracking device, wherein the subject tracking device is configured to track the position of the subject, The processor system is further configured to execute instructions to: register the tracked position of the first real-time image capture system and the tracked position of the subject; and generate a representation of the first real-time image capture system for display using at least the auxiliary display device.

16. The system for viewing a subject during surgery on the subject according to claim 15, wherein, The subject marker and the subject tracking device are integrated in a single device configured to be connected to the subject.

17. The system for viewing a subject during surgery on the subject according to claim 14, wherein, The first real-time image capture system is configured to move within the subject and obtain an image of the subject; and The auxiliary display device is operable to display the obtained image.

18. The system for viewing a subject during a surgical procedure on the subject according to claim 14, wherein, The first real-time image capture system is configured to move within the subject; and At least one of the first view screen or the second view screen of the wearable view screen and camera system is operable to display the obtained image.

19. The system for viewing a subject during surgery on the subject according to claim 17, wherein, The displayed obtained image is displayed separately and spaced apart from the generated instrument representation in a supplementary viewing area configured to display information for use or viewing by the user.

20. The system for viewing a subject during surgery on the subject according to claim 18, further comprising: a user input for turning on and off the display of the obtained image.

21. The system for viewing a subject during surgery on the subject according to claim 14, further comprising: a view screen and camera system tracking device; wherein the processor system is further configured to execute instructions to register the reference frame of the view screen and camera system tracking device and the reference frame of the tracked position of the subject; wherein the display using the auxiliary display device is at least partially based on the tracked position of the view screen and camera system tracking device.

Images