Virtual reality or augmented reality visualization of 3D medical images

Abstract

Systems and methods of virtual reality or augmented reality (VR / AR) visualization of 3D medical images using a VR / AR visualization system are disclosed. The VR / AR visualization system includes a computing device operably coupled to a VR / AR device, and the VR / AR device includes a holographic display and at least one sensor. The holographic display is configured to display a holographic image to an operator. The computing device is configured to receive at least one stored 3D image of an anatomical structure of a subject and at least one real-time 3D position of at least one surgical instrument. The computing device is further configured to register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one 3D image of the anatomical structure of the subject, and generate the holographic image including the at least one real-time position of the at least one surgical instrument overlaid on the at least one 3D image of the anatomical structure of the subject.

Description

[0001] This application is a divisional application of Chinese Patent Application No. 201780018916.1, the original application of which has an application date of March 20, 2017, a priority date of March 21, 2016, an international application number of PCT / US2017 / 023221, an entry into the Chinese national phase date of September 20, 2018, and an invention title of "Virtual Reality or Augmented Reality Visualization of 3D Medical Images."

[0002] Cross Reference to Related Applications

[0003] This application claims the benefit of U.S. Provisional Application No. 62 / 310,969, filed March 21, 2016, the entire contents of which are incorporated herein. BACKGROUND

[0004] A central problem of interventional surgery remains the visualization of unexposed anatomy within organs and blood vessels and the positioning of medical devices such as catheters, stents, probes, etc. With surgery moving from maximum exposure to minimally invasive, the requirement for enhanced visualization is even more profound. For example, minimally invasive, trans-catheter ablation for cardiac arrhythmias.

[0005] In a healthy heart, organized electrical excitation causes the heart to contract. When this electrical activity becomes irregular, the heart no longer pumps effectively, and the patient can experience dizziness, syncope, and / or sudden death. According to statistics from the Centers for Disease Control and Prevention, there are over 600,000 victims of sudden cardiac death in the United States each year. Unstable, irregular electrical heart activity is called an arrhythmia, and is often caused by abnormal electrical connections in the heart. Arrhythmias affect people of all ages.

[0006] Applying one or more energy pulses to selected regions of the heart through a catheter placed in the heart can effectively remove these short circuits, called trans-catheter ablation. Non-limiting examples of types of energy pulses that can be applied using trans-catheter ablation include radiofrequency energy pulses, low-energy pulses, and high-frequency ultrasound pulses. As a mainstay of modern arrhythmia therapy, ablation procedures require multiple catheters to be inserted into the heart to record electrical activity, identify key locations of arrhythmia causes, and ablate tissue using radiofrequency energy or cryotherapy. Currently, data segregation (i.e., electrical signals, anatomical locations, etc.) in the chest wall masking the heart and electrophysiology labs complicates ablation procedures, requiring physicians to mentally reconstruct a model of the heart.

[0007] These procedures have been significantly enhanced by the development of electroanatomical mapping systems that construct a point-by-point map of the endocardial surface (endocardium) of the heart that includes both the anatomical location and the local electrical signal. However, these systems are limited by the display of key measurements on multiple two-dimensional screens. The skill of mentally relating the electrical recordings to the overall multi-dimensional anatomical structure of the heart remains a key challenge in the intraoperative collaboration training of cardiac electrophysiologists and surgeons. As a result, training of new physicians is very difficult and significant skill-dependent variation in outcomes is common. SUMMARY

[0008] In one aspect, a VR / AR visualization system is provided. The VR / AR visualization system includes a VR / AR device including a holographic display configured to display a holographic image to an operator, and a computing device operably coupled to the VR / AR device. The computing device includes a non-volatile memory and a processor. The computing device is configured to receive at least one stored 3D image of an anatomical structure of a subject, receive at least one real-time 3D position of at least one surgical instrument, register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one stored 3D image of the anatomical structure of the subject, and generate the holographic image. The holographic image includes the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one 3D image of the anatomical structure of the subject.

[0009] In another aspect, a method of VR / AR visualization of 3D medical images is provided. The method includes receiving, using a computing device, at least one stored 3D image of an anatomical structure of a subject. The computing device is operably coupled to a VR / AR device, and the VR / AR device includes a holographic display and at least one sensor. The method further includes receiving, using the computing device, at least one real-time 3D position of at least one surgical instrument, registering the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one stored 3D image of the anatomical structure of the subject, and displaying, using the holographic display, a holographic image to an operator, the holographic image including the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one 3D image of the anatomical structure of the subject.

[0010] In another aspect, at least one non-transitory computer-readable storage medium for providing a VR / AR visualization of a three-dimensional medical image to an operator is provided. The computer-readable storage medium has computer-executable instructions embodied thereon, wherein the computer-executable instructions, when executed by at least one processor, cause the processor to receive at least one stored 3D image of an anatomical structure of a subject, receive at least one real-time 3D position of at least one surgical instrument, register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one stored 3D image of the anatomical structure of the subject, and display a holographic image. The holographic image includes the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one 3D image of the anatomical structure of the subject for the operator.

[0011] In one aspect, the present disclosure is a virtual reality / augmented reality (VR / AR) system for a procedure occurring within a portion of a patient's body. In one aspect, the present disclosure is a VR / AR system for a medical procedure occurring within a portion of human anatomy that is difficult to observe and access. In one aspect, the VR / AR system provides a multi-dimensional experience for a user (e.g., an operator) during a procedure with a 3D representation of internal human organs / systems of a patient in addition to other procedure-related data. In one aspect, the 3D representation and additional information can be presented in an augmented reality environment. In other aspects, such information can be provided in a VR / AR environment. Additionally, the system is capable of mapping and representing real-time positioning of instruments (e.g., catheters) used during the procedure. The 3D representation and related data are configured to be presented and allow for interaction with the user such that the user does not need to communicate with anyone else or break sterility. For example, the user can provide commands to the system without needing physical contact of an input device with any portion of the user's body, such as the user's hand.

[0012] In an exemplary aspect, the VR / AR system is used for a cardiac interventional procedure. The system provides a 3D model of a patient's heart in real-time as the procedure occurs, including the ability to track positioning of catheters used within the patient's heart. Additional information can be provided to the user through other senses (e.g., auditory signals) in a manner described below.

[0013] In one aspect, a VR / AR system receives, using a computing device, at least one stored 3D image of an anatomical structure of a subject (e.g., an anatomical structure of a patient). The computing device is coupled to a VR / AR device, which includes a holographic display and at least one sensor. The computing device receives at least one real-time 3D position of at least one surgical instrument. The at least one real-time 3D position of the at least one surgical instrument is registered to the at least one stored 3D image corresponding to the anatomical structure of the subject. The holographic display displays a holographic image to an operator, the holographic image including the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one 3D image of the anatomical structure of the subject.

[0014] In another aspect, a VR / AR system generates a first 3D image representing an electroanatomical visualization of a cardiovascular organ of a subject in 3D by processing a first set of sensor data generated by a catheter inserted within the cardiovascular organ. The first 3D image is provided to a holographic display, including but not limited to a head-mounted display (HMD) worn by an operator, to display the electroanatomical visualization in a field of view of the operator. A second set of sensor data is received from an input device, where the second set of sensor data is indicative of a motion of a body part of the operator interacting with the electroanatomical visualization. A motion path of the body part is determined by processing the second set of sensor data. A rotation angle of the electroanatomical visualization is determined based on the motion path. A second 3D image is provided to the HMD to update the display of the electroanatomical visualization by rotating the cardiovascular organ by the rotation angle in the field of view of the operator.

[0015] These and other objects and advantages of the present application will become apparent from the following detailed description of the preferred embodiment. Both the foregoing general description and the following detailed description are exemplary and explanatory only, and are intended to provide further explanation of the application as claimed.

[0016] The accompanying drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification, illustrate several embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0017] FIG. 1 is a fluoroscopy image for positioning a catheter within a heart according to prior methods.

[0018] Figure 2 is a schematic representation of a VR / AR system for internal medical procedures according to an aspect of the present disclosure.

[0019] Figure 3 is an image of an electroanatomical map according to an aspect of the present disclosure.

[0020] Figure 4A 3D model of a patient's heart is shown.

[0021] Figure 5A An image of a graph including a drop-down menu provided to an operator of a VR / AR system according to one aspect.

[0022] Figure 5B An image of a graph including an insert image drop-down menu provided to an operator of a VR / AR system according to one aspect.

[0023] Figure 5C An image of a graph including a cross-sectional view of a 3D heart model provided to an operator of a VR / AR system according to one aspect.

[0024] Figure 6 A block diagram illustrating a schematic representation of components of a VR / AR system of Figure 2 DETAILED DESCRIPTION

[0025] In the following detailed description of preferred embodiments, reference is made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific embodiments in which the disclosure can be practiced. It is understood that other embodiments can be utilized and structural changes can be made without departing from the scope of the present disclosure.

[0026] As used in the specification and the appended claims, the singular forms “a,” “an” and “the” include plural referents unless the context clearly dictates otherwise. Ranges can be expressed as from “about” one particular value, and / or to “about” another particular value. When such ranges are expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when values are expressed as approximations by use of the antecedent “about,” it will be understood that the particular value forms another embodiment. It will be further understood that the endpoints of the ranges are significant, and that the ranges are only approximations of the actual range, and are only used to illustrate the range included in another embodiment.

[0027] “Optional” or “optionally” means that the subsequently described event or circumstance can or can not occur, and that the description includes instances where the event or circumstance occurs and instances where it does not.

[0028] Throughout the description and claims of this specification, the word “comprise” and variations of the word, such as “comprising” and “comprises,” mean “including but not limited to,” and are not intended to exclude, for example, additional, adjunct, complementary, or other features. “Exemplary” means “an example of’ and is not intended to convey an indication of a preferred or ideal embodiment. “Such as” is not used in a restrictive sense, and is used to mean “comprising at least the recited items.”

[0029] ​Disclosed are components that can be used to perform disclosed methods and systems. These and other components are disclosed herein, and can be used individually or in any combination. It is to be understood that when a combination is disclosed, that combination is intended to also encompass any individual component of the combination, as well as any combination of individual components of the disclosed combination. Thus, if a combination is disclosed comprising a first component and a second component, the combination is intended to also cover a first component alone, a second component alone, as well as any of the individual components in any combination with each other, as well as any combination of individual components of the disclosed combination. This applies to every combination disclosed or contemplated herein. Thus, if there are a variety of additional steps that can be performed, it is understood that each of the additional steps can be performed with any specific embodiment or combination of embodiments of the disclosed methods.

[0030] As will be appreciated by those of skill in the art, aspects of the present disclosure can be embodied as a device, a method or a computer program product. Accordingly, aspects of the present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. In one aspect, the present disclosure can include a combination of physical components configured to perform certain steps and functions controlled by a combination of hardware and software components (e.g., obtaining electroanatomical measurements, etc.). Furthermore, aspects of the present disclosure can take the form of a computer program product on a computer-readable non-transitory storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. Any suitable computer readable storage medium can be utilized, including a hard disk, CD-ROM, optical storage device, flash device, solid-state storage device, and magnetic storage device.

[0031] Furthermore, the components and methods used by the present disclosure as described below can be implemented in a program environment comprising general purpose computers or specialized devices such as hardware devices, controllers or hand-held computers. In addition, the techniques described herein can be implemented using various technologies known in the art. For example, the methods can be implemented in software executed on a computer system, or in hardware utilizing combinations of off-the-shelf components or other specially designed application-specific integrated circuits, programmable logic devices, or various combinations thereof.

[0032] Some aspects of the methods and systems are described below with reference to block and flow diagrams of methods, systems, apparatus, and computer program products. It will be understood that each block of the block diagrams and flow diagrams, and combinations of blocks in the block diagrams and flow diagrams, can be implemented by computer program instructions. These computer program instructions can be loaded onto a general purpose computer, a special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute on the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flow diagram block or blocks.

[0033] These computer program instructions can also be stored in a non-transitory computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including computer-readable instructions for implementing the functions specified in the block or blocks of the flowchart. The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in the block or blocks of the flowchart.

[0034] Accordingly, the blocks of the flowchart and the block diagrams support combinations of means for performing the specified functions, combinations of steps for performing the specified functions, and program instruction means for performing the specified functions. It will also be understood that each block of the flowchart and the block diagrams, and combinations of blocks in the flowchart and the block diagrams, can be implemented by special purpose hardware-based computer systems that perform the specified functions or combinations of special purpose hardware and computer instructions.

[0035] In various aspects, the present disclosure relates to virtual reality and / or augmented reality (VR / AR) systems, which will be discussed in detail below. Although the term virtual reality is used throughout to describe the system, those skilled in the art will appreciate that, in the general use of the term, virtual reality can include both virtual reality (VR) and augmented reality (AR). In some cases, the term "VR / AR" will be used to identify the system. Thus, when the term "virtual reality" or "VR / AR" is used herein, it should be understood to include all types of modified reality, unless specifically distinguished.

[0036] "Virtual reality" as used herein refers to a method of displaying and / or interacting with one or more elements representing computer-generated data. Typically, all elements visible within the field of view of a virtual reality display are computer-generated elements.

[0037] "Augmented reality" as used herein refers to a method of displaying and / or interacting with one or more elements representing computer-generated data. Augmented reality is a hybrid of virtual reality and real life, where a typical augmented reality display includes one or more computer-generated elements overlaid on real objects visible to the operator. The term "augmented reality" as used herein can further include "mixed reality", the term "mixed reality" referring to an augmented reality display method that specifically includes the ability for a user or operator to interact with the computer-generated elements.

[0038] "Holographic display" as used herein refers to a method of displaying and / or interacting with virtual 3D objects, where the virtual 3D objects are dynamically updated in response to motion of an operator, or an operator's request to modify the view of the virtual 3D objects (such as zoom in / out, panning / rotating cross-sectional views of the virtual 3D objects, etc.) to modify the operator's view of the virtual 3D objects.

[0039] The present disclosure relates to a VR / AR visualization system 10 (also referred to herein as VR / AR system 10 or system 10) for the visualization and manipulation of 3D imaging data as well as additional information including, but not limited to, 2D imaging data, vital sign data, and subject demographic data associated with medical diagnostic and therapeutic procedures occurring within portions of the subject's anatomy that are difficult to observe / access. By way of non-limiting example, the VR / AR visualization system 10 can be used for procedures within the heart, gastrointestinal system, ear canal, and other types of anatomical structures or biological systems. While the embodiments described below relate to a VR / AR visualization system 10 associated with cardiac interventional procedures, those skilled in the art will recognize that the use of other 3D localization or pose estimation modalities, including but not limited to impedance-based localization, magnetic localization, marker-based localization, or 3D ultrasound in conjunction with the VR / AR visualization system 10 described herein, can be readily extended to diagnostic or interventional procedures in other organ systems and / or other anatomical regions.

[0040] In one aspect, the present disclosure relates to a VR / AR visualization system 10 for use in association with cardiac diagnostic procedures and / or cardiac interventional procedures. The VR / AR system 10 is capable of acquiring anatomical features of a heart 15 of a patient 20 (e.g., a subject) via a 3D medical imaging device 40. Additionally, electrical data of the heart 15 can be obtained using one or more electroanatomical mapping devices 50 configured to collect electrical data as well as the position of the associated instrument and electrical data measurements. Non-limiting examples of instruments associated with the one or more electroanatomical mapping devices 50 include diagnostic catheters, reference catheters, ablation catheters, monitoring catheters, non-contact mapping catheters, multi-electrode array catheters, multipolar catheters, multipolar circular mapping catheters, and magnetic sensor catheters. In one aspect, the data obtained by the one or more electroanatomical mapping devices 50 can be analyzed by an associated electroanatomical mapping system to determine an endocardial map that describes the spatial disposition of the interior surfaces of the atria and ventricles of the heart 15 as well as the spatial disposition of the veins and arteries (including but not limited to the vena cava, the aorta, the pulmonary artery, etc.) in relative proximity to the heart 15.

[0041] In one aspect, the one or more electroanatomical mapping devices 50 can provide for combining at least one 3D map of an anatomical feature of the heart 15 of the patient 20 obtained using the 3D medical imaging device 40 and at least one 3D endocardial surface map obtained using the one or more electroanatomical mapping devices 50. In one aspect, a 3D coordinate system within which the 3D endocardial surface map is defined can be registered to a coordinate system within which the 3D map of the anatomical feature of the heart is defined, such that the 3D map of the anatomical feature and the 3D endocardial surface map are defined within the same 3D coordinate system.

[0042] In one aspect, data defining the 3D map of the co-registered anatomical feature and the 3D endocardial surface map produced by the one or more electroanatomical mapping devices 50 can be received by the computing device 30 of the VR / AR visualization system 10. Using the data defining the co-registered 3D map, the computing device 30 generates a 3D model 65 of the heart 15 that is configured to be displayed within a holographic image produced by the holographic display of the VR / AR device 60.

[0043] In one aspect, the holographic image can consist solely of at least a portion of the 3D model 65 of the heart 15. In various other aspects, the computing device 30 of the VR / AR visualization system 10 can receive additional data that can be incorporated into the holographic image for display to an operator on the holographic display of the VR / AR device 60.

[0044] In various aspects, the VR / AR system 10 can include a plurality of VR / AR devices 60. As a non-limiting example, a first operator can wear a first VR / AR device 60, including but not limited to, a first head-mounted display, and a second operator can wear a second VR / AR device 60, including but not limited to, a second head-mounted display. In this non-limiting example, the first operator and the second operator can perform a surgical procedure on a patient together. The first VR / AR device 60 and the second VR / AR device 60 can display different views of the same organ or portion of the patient's anatomy. For example, the displayed view of the patient's heart 15 can be positioned at different angles based on the corresponding VR / AR device 60 position relative to the patient 20.

[0045] In various other aspects, additional data can be received by the computing device 30 of the VR / AR visualization system 10 and incorporated into the holographic image. Non-limiting examples of additional data suitable for incorporation into the holographic image include: real-time 3D data defining locations and / or measurements within the coordinate system defining the 3D model 65; real-time 2D data generated by a 2D imaging device such as a fluoroscopic imaging device; real-time digital measurements such as one or more vital signs, predetermined data such as patient demographic data, and any combination thereof. In these various other aspects, the additional data can be incorporated and / or overlaid on the 3D model 65 of the heart 15, or the additional data can be displayed as a separate element within the holographic image, as described in additional detail below.

[0046] In one aspect, additional data obtained by the electroanatomical mapping device 50 can be received by the computing device 30 and incorporated into the holographic image. In one aspect, the electroanatomical mapping device can further include an instrument position sensor configured to obtain at least one real-time 3D position of at least one surgical instrument, including but not limited to an electrophysiology (EP) catheter. In this aspect, the computing device 30 can receive the at least one real-time 3D position of the at least one surgical instrument and generate a holographic image including the at least one real-time position of the at least one surgical instrument overlaid on at least one 3D image of the subject’s anatomical structure, including but not limited to the 3D model 65 of the heart 15. In another aspect, the at least one real-time 3D position of the at least one surgical instrument can be obtained using a separate device, including but not limited to a separate instrument position sensor of a surgical instrument system. Non-limiting examples of suitable instrument position sensors include one or more electroanatomical mapping devices, in addition to the electroanatomical mapping device 50 used to obtain 3D endocardial surface mapping data, other real-time position mapping systems utilizing position sensing devices with ultrasound, magnetic fields, electric fields, and / or any other existing suitable position sensing methods.

[0047] In another aspect, the additional data obtained by the electroanatomical mapping device 50 can include additional electrophysiology measurements obtained by one or more electrophysiology (EP) catheters. In one aspect, the data can include additional 3D real-time data of one or more data sets defining real-time electrophysiology measurements mapped to the coordinate system of the 3D model 65 of the heart 15. Non-limiting examples of real-time electrophysiology measurements include voltage maps, activation timing maps, and propagation maps. In this one aspect, a map of the additional electrophysiology measurements received by the computing device 30 can be overlaid on the 3D model 65 of the heart 15 within the holographic image. In one aspect, the map of the additional electrophysiology measurements can be selected for display, made transparent, or removed from the holographic image defined by one or more prompts generated by the operator. In another aspect, the additional data obtained by the electroanatomical mapping device 50 can include additional electrophysiology measurements associated with ablation by the ablation catheter, including but not limited to the amount of radiofrequency (RF) or cryogenic energy applied at a particular location within the 3D model 65 of the heart 15. In this other aspect, the additional electrophysiology measurements can be incorporated into the holographic image in the form of 3D visual elements, such as circles or other symbols located at the point of application of the ablation energy within the 3D model 65, and / or colors, sizes, numerical values, or other visual elements to indicate the amount and / or direction of the ablation energy or force measured during an ablation event effected by the ablation catheter.

[0048] In further aspects, the computing device 30 can receive one or more additional data sets defining at least one additional 2D image obtained from at least one additional medical imaging device. In one aspect, the at least one additional 2D image can include a 2D representation of a 3D real-time image, including but not limited to a real-time fluoroscopic image. In this one aspect, the at least one additional 2D image can be incorporated into the holographic image in the form of a 2D visual element displayed separately from the 3D model 65. Non-limiting suitable 2D visual elements incorporated into the holographic image include virtual 2D monitors, inset images, and any other known representation of a 2D visual element within a 3D holographic image. By way of non-limiting example, an additional data set defining a real-time fluoroscopy image can be incorporated into the holographic image in the form of an inset image, as shown in FIG. 6B. Figure 5B

[0049] ​In one aspect, the holographic image can be displayed to an operator on the holographic display of the VR / AR device 60. In response to one or more prompts from the operator, the computing device 30 can modify the holographic image displayed on the holographic display according to the operator's preferences. By way of non-limiting examples, the operator can zoom in, zoom out, rotate, or move the holographic image displayed on the holographic display to facilitate diagnosis and / or surgery. In various aspects, if the VR / AR system 10 includes multiple VR / AR devices 60, a first VR / AR device 60 worn by a first operator can be operatively coupled to the computing device 30 such that only the first operator can modify the holographic image displayed on all holographic displays of all VR / AR devices 60 in the VR / AR system 10. In these various other aspects, the computing device 30 can receive the relative position and orientation of each of the plurality of VR / AR devices 60 and generate a holographic image for each VR / AR device 60, the holographic image corresponding to each position and orientation of each VR / AR device 60 relative to a first VR / AR device 60 worn by a first operator, who also controls modifications to the holographic image.

[0050] In a non-limiting example, computing device 30 can generate an electrocardiogram (ECG) of the patient's heart to be displayed in an interpolated image of the hologram. Computing device 30 can receive user input instructions (i.e., prompts) from VR / AR device 60 or a separate input device (e.g., a camera or motion sensor) to position virtual markers on the ECG without compromising sterility. Based on the user input, computing device 30 can determine the measurements to be displayed in the interpolated image and on the ECG. For example, the virtual markers could be virtual calipers configured to measure the cycle or amplitude of the patient's heartbeat (e.g., a metric).

[0051] In another additional aspect, the computing device 30 may receive one or more additional alphanumeric datasets, including but not limited to patient demographic datasets and / or real-time measurements of vital signs of an object obtained from existing vital sign measurement devices. In this further aspect, one or more additional alphanumeric datasets may be incorporated into the holographic image in the form of transparent alphanumeric elements overlaid within the holographic image.

[0052] In another aspect, the computing device 30 can generate menus or other graphic elements, including user-selectable elements displayed as alphanumeric strings, symbols, and / or icons in non-transparent or transparent rows or columns. In these additional aspects, the user-selectable elements provide a way to select one or more instructions executed by one or more processors of the computing device 30 to implement the operation of the VR / AR system 10. By way of non-limiting example, the computing device 30 can, for example... Figure 5A A transparent menu is generated within the holographic image shown.

[0053] By selecting one or more interactive menu elements within the holographic image, the operator of the VR / AR device 60 can also observe and interact with the 3D model 65 and additional information without having to break sterility, or communicate with anyone present during a diagnostic or surgical procedure. As disclosed herein, the VR / AR device 60 can present and remove information as needed in a real virtual reality environment or augmented reality environment depending on the needs of the operator of the system 10. In one aspect, the arrangement of elements within the holographic image can be saved by the computing device 30 and retrieved by the VR / AR system 10 for subsequent use by the same operator as a preferred element arrangement.

[0054] In various aspects, the computing device 30 can update the holographic image over time to account for changes due to one or more time-varying factors, including but not limited to: selection or deselection of user-selectable menu elements by the operator, receipt of updated real-time data (such as updated vital sign data), changes in additional ablation events, other data sets (such as real-time data sets associated with fluoroscopic imaging). The computing device 30 can update the holographic image to include a representation of a portion of the instrument positioned relative to the patient’s organ, for example, the location of the catheter tip relative to the patient’s heart. The computing device 30 can update the holographic image based on sensor data from the instrument indicative of detected motion of the instrument.

[0055] In one aspect, the computing device 30 of the VR / AR system 10 is configured to receive a reconstructed 3D model 65 of the patient’s cardiac anatomy. Here, the 3D medical imaging device 40 is configured to acquire specific physical information related to the patient’s heart 15. This information will encode the dimensions of the heart 15 and its constituent / chambers and predefined anatomical landmarks. These anatomically distinct landmarks are used to register the 3D model 65 to an electroanatomical map of the endocardial surface obtained by the electroanatomical mapping device 50. By registering the electroanatomical map to the coordinate system of the 3D anatomical model within which it is obtained by the electroanatomical mapping device 50, other measurements of the electroanatomical map mapped to the endocardial surface are also registered to the coordinate system of the 3D model 65 defining the heart 15, such that these mapped electrophysiology measurements can be visualized relative to the 3D model 65 within the holographic image displayed to the operator. In some aspects, the 3D model 65 and the electroanatomical map are generated in different physical units. Thus, to normalize to the same coordinate system, the computing device 30 registers (e.g., maps) a first set of points of the 3D model 65 to a second set of points of the electroanatomical map, for example, based on the anatomical landmarks.

[0056] In various aspects, registration of the endocardial surface map with the 3D anatomical model obtained using the at least one 3D medical imaging device is performed by the computing device 30 and / or the electroanatomical mapping device 50. Registration of the two maps can be performed using any suitable existing method without limitation. In one aspect, at least a plurality of predetermined landmarks are identified in the 3D anatomical model and the 3D endocardial surface map to provide a defined intersection of the two maps. Non-limiting examples of suitable cardiac landmarks include the superior vena cava / right atrium junction, the inferior vena cava / right atrium junction, the coronary sinus, the right ventricular apex, and the right ventricular outflow tract. While other known portions of the heart can be associated with landmarks in other aspects of the present disclosure, these landmarks are readily accessible and identifiable via MRI and fluoroscopy procedures and employ grid registration as discussed in greater detail below. In other aspects, statistical or feature-based inference can be used to determine the location and number of points.

[0057] In one aspect, the 3D medical imaging device 40 can include any device capable of acquiring 3D data related to the anatomical structure of the heart. For example, the 3D medical imaging device can include, but is not limited to, a fluoroscopy device, an echocardiography device (e.g., transthoracic, transesophageal, intracardiac), an X-ray device, an exploratory endoscopic system, an MRI, and a CT scanner, among others. The 3D medical imaging device 40 can be selected depending on the type of diagnosis and / or surgical procedure to be performed in combination with the VR / AR visualization system 10. For example, an echocardiography device 40 provides rapid acquisition of the cardiac anatomical structure, is non-invasive, can be completed quickly, does not expose the patient to radiation, and is without anesthesia. Further, there is no long-term requirement for immobility of the subject. In one aspect, data defining the reconstructed 3D anatomical model can be acquired and analyzed by the 3D medical imaging device 40 prior to the diagnosis or surgical procedure performed in conjunction with the VR / AR visualization system 10.

[0058] 3D medical imaging device 40 acquires spatial information used to reconstruct a 3D model 65 of the subject's heart 15. In one aspect, 3D medical imaging device 40 reconstructs spatial data and creates a 3D anatomical model of the subject's cardiac anatomy. The 3D anatomical model can be created using various techniques known in the art. By way of non-limiting example, a plurality of 2D views of the patient's heart can be collected and then recombined into a rough 3D image of the patient's anatomy. Once the 3D image of the patient's anatomy is generated, electroanatomical mapping device 50 provides to computing device 30 a generic high-resolution cardiac mesh to be modified and / or transformed to match the overlap of corresponding landmarks identified in the anatomical heart model and the electroanatomical endocardial surface model. Computing device 30 transforms the cardiac mesh using data acquired from the created 3D image of the patient's cardiac anatomy, including the spatial dimensions of the acquired landmarks (as described above). Once transformed, a full 3D model 65 of the patient's cardiac structure is generated by computing device 30 or electroanatomical mapping device 50. By way of one non-limiting example, 3D medical imaging device 40 can create the 3D anatomical model by generating a series of connected 2D polygons (e.g., a mesh) representing the 3D geometry of the corresponding anatomy. By way of another non-limiting example, device 40 can use ray casting and / or tracing to create the 3D anatomical model.

[0059] After the transformed full 3D model 65 of the patient's cardiac structure has been created by electroanatomical mapping device 50, electroanatomical data including electrophysiology data specific to the ablation needs is collected from the subject and mapped to the transformed full 3D model 65. In one aspect, the electroanatomical data is collected from one or more electroanatomical mapping devices 50. In one aspect, electroanatomical mapping device 50 includes an electrophysiology (EP) catheter placed within the patient's heart 15. The measured electroanatomical data can include electrical activity (e.g., voltage data, activation timing, and propagation maps) occurring at a given location of the patient's heart 15 that can be used to determine where ablation needs to occur using existing diagnostic methods.

[0060] To perform the EP study, the VR / AR system 100 can use one or more electroanatomical mapping devices 50. In one aspect, a diagnostic catheter 50 can be used initially, and additional ablation catheters 50 can be used if ablation is indicated based on the diagnostic electrophysiology measurements. Additionally, any energy applied by the ablation catheter 50 can be measured and recorded by one or more electroanatomical mapping devices 50. By way of non-limiting example, the ablation catheter can be configured to apply radiofrequency or low energy at the catheter tip. In another aspect, the ablation catheter can be configured to sense the level of force (including directionality, i.e., axial or lateral) applied by the catheter at the tip of the catheter 50. Such information can be used to simulate actual lesion creation in the virtual heart of the 3D model 65, as by using the measured force and impedance to make lesions in real time. In one aspect, the force data measured from the ablation catheter can then be used to employ auditory feedback to the operator with the VR / AR device 60. By way of non-limiting example, the VR / AR device 60 can provide auditory feedback including pitch and frequency of tone that changes, which indicates the level of force with which the ablation catheter 50 is pushing against the heart tissue.

[0061] The EP catheters 50 can be moved by the operator throughout the patient’s heart 15 and collect diagnostic information. Both the diagnostic and ablation catheters 50 are configured to obtain electroanatomical data throughout their use, including during ablation. In addition to collecting electroanatomical data, the VR / AR system 100 can also acquire spatially related information, i.e., where within the patient’s heart the electroanatomical data occurred, so that the electroanatomical data can be mapped to the transformed full 3D model. In one aspect, the electroanatomical mapping devices 50 also acquire the positioning of the instruments within the patient. For example, when utilizing the EP catheters 50, the positioning of the electrodes and the distal portion of the shaft of the catheter 50 is acquired. In one aspect, the electroanatomical mapping devices 50 utilize an electroanatomical mapping system to find the coordinates of the electroanatomical data. The mapping system is able to identify the X, Y, and Z coordinates of the EP catheter 50 within the heart 15, and can then place the electroanatomical data to the coordinates. For example, these data can be collected using an electroanatomical mapping system such as the ENSITE NAVX TM NAVX TM Carto TM (Webster Biosense System). However, other systems that are able to provide such information can also be used.

[0062] Figure 3An example is shown of a current electroanatomical map for a heart model. The heart model is limited to the geometry of the right atrium (RA) and the left atrium (LA). The image on the left is presented in orthogonal views, with a right anterior oblique (RAO) view on the left and a long axis oblique (LAO) view on the right. The distal end of an electrophysiology catheter is visualized within this geometry. A first (1st) catheter with four electrodes is positioned at a normal conduction site, at the His location. A second catheter with 10 electrodes is positioned in the coronary sinus. A third catheter with 4 electrodes is only visualized in the LAO projection, as it is advanced through the tricuspid valve and into the right ventricle. Finally, a radiofrequency (RF) ablation catheter with 4 electrodes is positioned at a site of abnormal electrical tissue. A sphere (shown in red) marks the site at which the RF lesion is placed. Figure 3 The central bottom target heart projection indicates the total force (TF) applied to the heart tissue by the ablation catheter.

[0063] While the above describes an electroanatomical mapping device 50 and electroanatomical data associated with cardiac surgery, it is understood that the device and data can be associated with other systems and organs found in the human body.

[0064] As the electroanatomical data is collected and mapped as described above, the VR / AR device 60 is provided by the computing device 30 with a 3D model 65 of the patient's heart 15 (see Figure 4 , including the placement of the EP catheter 50 (see Figure 5B ). In one aspect, the VR / AR device 60 can include a true virtual reality (VR) device (i.e., a device that fully immerses the operator in a created environment) or an augmented reality (AR) device (i.e., an operator can have images or models displayed virtually in a virtual space, but is still able to see and interact with the real environment). In these ways, the VR / AR device 60 can be a wearable device, including but not limited to the Microsoft HOLOLENS TM (i.e., an AR device) and the OCULUS RIFT TM (i.e., a VR device). The VR / AR device 60 can include sensors such as motion sensors (e.g., accelerometers, gyroscopes, or inertial measurement units), audio sensors, eye and gaze tracking sensors, and / or electronic displays, among other components. In another aspect, the VR / AR device 60 can provide a projected holographic display that includes the 3D model 65. The VR / AR device 60 can be communicatively coupled to the HMD via a wireless exchange protocol or via a wired connection. In at least some aspects, the use of an AR device can be advantageous as it allows the operator to see and interact with the patient in real time while viewing and interacting with the 3D model 65 and deriving the benefits thereof, contributing to a safer patient experience.

[0065] In these aspects, the 3D model 65 within the holographic image can appear dull or semi-transparent depending on the location of the EP catheter, or fully transparent to enable unobstructed viewing of the EP catheter location. In the transparent view, the operator can use operator-enabled cues received by at least one sensor of the VR / AR device to change the transparency of the heart wall, allowing for easy visualization of the catheter in the heart during diagnosis and / or surgery. Portions of the catheter can also be represented in any view. Additionally, the VR / AR device 60 can also allow the operator to manipulate the position, orientation, and size of the heart, as well as create slices to be viewed. Furthermore, the operator can switch between views and data displays without using hands, so the operator can maintain sterility throughout the procedure.

[0066] By way of non-limiting example, the VR / AR system 100 can use head and / or eye tracking technology to receive input commands (e.g., user inputs) from the operator without requiring the operator to physically touch an input device (e.g., the VR / AR device 60) using the operator's hands. In some embodiments, the input device is physically connected to the VR / AR device 60. In other embodiments, the input device is separate from the VR / AR device 60 and communicatively coupled to the computing device 30. For example, the input device is a Microsoft KINECT® TM The input device can include imaging sensors (e.g., cameras), illumination sources for the imaging sensors, motion sensors, depth sensors, and other components. Based on the sensor data, the input device can capture gestures and perform operator gesture detection. In one aspect, operator-enabled inputs can be derived from modifying existing operator-enabled inputs provided by the VR / AR device 60.

[0067] Additionally, the electroanatomical mapping device 50 can also be used in conjunction with the VR / AR device 60 to produce a planned map gathered from collected electroanatomical data in preparation for an ablation procedure. In another aspect, the VR / AR device 60 can display virtual calipers (e.g., rulers) to allow the operator to make real-time accurate measurements in the virtual environment. This feature allows the operator to make measurements in various locations, for example, in milliseconds when measuring electrograms, and in millimeters in the heart geometry.

[0068] Other electroanatomical information can be displayed or communicated to the operator. For example, the electroanatomical information can be represented numerically on the 3D model 65 of the heart (e.g., color-coded regions on the heart to indicate electrical activity and force data at the time of application by the EP catheter), visually displayed to the operator (e.g., a table showing relevant force data and electrical activity), or aurally. In one aspect, the aural approach can be used to inform the operator of the force applied by the EP catheter when activated during an ablation procedure. In this case, the aural response can be proportional to the force at the time of application. The aural signal relates to the force applied by the ablation catheter. The stronger the force applied, the more frequent and high-pitched the tone to the operator. This aural feature is only present in force-sensing catheters. An example of a force-sensing catheter is the TACTICATH® TM (St Jude Medical), which provides feedback to the operator to indicate how much force (e.g., measured in grams) the catheter tip is applying to the tissue.

[0069] Figure 5A 、 Figure 5B and Figure 5C shows an example diagram of a holographic image displayed by the VR / AR device 60 according to one aspect of the present disclosure. Figure 5A shows the open main menu of the 3D model 65 (virtual heart model). The transparency of the 3D heart model 65 is increased in this view to allow the operator to quickly visualize the precise catheter locations in one or more planes without having to change the orientation of the model. In various aspects, the computing device 30 of the VR / AR system can segment the 3D model into sub-units to modify the holographic display by rendering a portion of the 3D model 65 as transparent and / or invisible to facilitate selection of a portion of the 3D model 65. Non-limiting examples of sub-units of the 3D model 65 that can be segmented by the computing device 30 include the left and right atria, the left and right ventricles, one or more valves, one or more arteries and veins associated with the heart, and any other relevant cardiac structure.

[0070] Figure 5B shows the 3D model 65 displayed in the rear projection with the catheter locations open, where a cut plane feature is used to cut out a portion of the heart so that the intracardiac catheter locations can be visualized. In this view, four catheters (circled) can be seen in the coronary sinus (in the atrioventricular groove where the left atrium separates from the left ventricle; toward the left side of the screen), the high right atrium (in the right upper chamber, near the junction of the right atrium / superior vena cava), the right ventricular apex (in the right lower chamber, pointing toward the apex of the heart), and the normal His conduction system (toward the center or key of the heart). In some aspects, the computing device 30 includes additional components in the displayed 3D model 65 based on the operator-controlled zoom level (e.g., zoom in / out) of the holographic display. For example, at a larger zoom level (i.e., zoomed-in image), the computing device 30 includesFigure 5B One or more of the four conduits shown in FIG. 1. In another aspect, at lower levels of zoom (i.e., zoomed-out images), the computing device 30 does not include the four conduits, for example due to the resulting limited level of resolution of the displayed 3D model 65. In Figure 5B On the right in FIG. 1, a fluoroscopy screen has been opened in the virtual environment.

[0071] Figure 5C A 3D model 65 is shown oriented in a down-the-barrel view (i.e., the surgeon’s view) of the ventricle, with the atrium and great arteries virtually removed, such as a cross-sectional view. These images provide non-limiting examples of potential views that an operator can see using the VR / AR system 10, and should not be interpreted as the only views provided by the system 10 of the present disclosure.

[0072] In various aspects, the computing device 30 can include, but is not limited to, a notebook computer, a desktop computer, a tablet computer, a server with a connected display, and the like. According to one aspect, as shown in FIG. 1, the computing device 30 can include a display 110, a processor 120, a memory 130, a network adapter 126, and a wireless interface controller (“W.I.”) 100. The display 110 can be any type of display known in the art, including but not limited to a liquid crystal display (“LCD”), a light-emitting diode (“LED”) display, an organic light-emitting diode (“OLED”) display, a plasma display, a cathode ray tube (“CRT”) display, and the like. The processor 120 can be any type of processor known in the art, including but not limited to a central processing unit (“CPU”), a microprocessor, a microcontroller, a digital signal processor (“DSP”), a graphics processing unit (“GPU”), and the like. The memory 130 can be any type of memory known in the art, including but not limited to a random access memory (“RAM”), a read-only memory (“ROM”), a dynamic RAM (“DRAM”), a synchronous dynamic RAM (“SDRAM”), a flash memory, and the like. The network adapter 126 can be any type of network adapter known in the art, including but not limited to a network interface card (“NIC”), a wireless network adapter, and the like. The wireless interface controller 100 can be any type of wireless interface controller known in the art, including but not limited to a Bluetooth® interface controller, a Wi-Fi® interface controller, and the like. Figure 6

[0073] ​The computing device 30 can have one or more software applications 104 to perform the methods discussed above. The computing device 30 includes system memory 108, which can store various applications 104, including applications that perform the functions described above, and an operating system 110. The system memory 108 can also include data 112 that is accessible by the various software applications 104. The system memory 108 can include random access memory (RAM) or read-only memory (ROM). The data 112 stored on the computing device 30 can be any type of retrievable data. The data can be stored in a variety of databases, including relational databases, including but not limited to Microsoft Access and SQL Server, MySQL, INGRES, DB2, INFORMIX, Oracle, PostgreSQL, Sybase 11, Linux Data Storage Components, and the like.

[0074] The computing device 30 can include various other computer-readable media, including storage 114. The storage 114 can be used to store computer code, computer-readable instructions, program modules, and other data 112 for the computing device 30, and the storage 114 can be used to back up or alternatively run the operating system 110 and / or other applications 104. The storage 114 can include a hard disk, various magnetic storage devices such as a tape cartridge or disk, a solid-state flash drive, or other optical storage, random access memory, and the like.

[0075] The computing device 30 can include a system bus 118 that connects the various components of the computing device 30 to the system memory 108 and the storage 114, as well as to each other. Other components of the computing device 30 can include one or more processors or processing units 120, a user interface (UI) 122, and one or more input / output interfaces 124. In addition, the computing device 30 includes a network adapter 126. Further, the computing device 30 can include a power source 128, including but not limited to a battery or external power source. Further, the computing device 30 can include a display adapter 226 and a display 228 (e.g., a monitor or screen). Additionally, input devices (e.g., a keyboard, mouse, joystick, etc.) can be used via the input / output interface 124. Further, the other 3D medical imaging device 40, electroanatomical mapping device 50, and VR / AR device 60 can also communicate with the computing device 30 via the input / output interface 124.

[0076] The VR / AR system 10 is configured to display a virtual 3D model 65 of the patient in front of the interventional physician (e.g., operator) during diagnosis and / or surgery. By using the 3D model 65 that reveals real-time electrophysiology, improvements in physician training, patient outcomes, and clinician collaboration will occur, as well as a reduction in radiation exposure rates for both the patient and the physician. In addition, the system 10 will also reduce the medical and economic burden on the patient who would otherwise be subjected to multiple procedures affected by poor visualization of their anatomy.

[0077] While the foregoing written description of the present disclosure enables a person skilled in the art to make and use the presently preferred mode thereof, those skilled in the art will appreciate and understand that variations, combinations, and equivalents of the particular embodiments, methods, and examples herein fall within the scope and spirit of the present disclosure. Accordingly, the present disclosure should not be limited to the above-described embodiments, methods, and examples, but rather all embodiments and methods within the scope and spirit of the present disclosure. To the extent necessary to understand or complete the present disclosure, all publications, patents, and patent applications referred to herein are expressly incorporated by reference herein to the same extent as if each individual publication, patent, or patent application was specifically and individually incorporated by reference.

[0078] Having thus described the exemplary embodiments of the present disclosure, those skilled in the art will appreciate and understand that the disclosure is merely exemplary and that various other substitutions, changes and modifications can be made within the scope and spirit of the disclosure. Accordingly, the present disclosure is not limited to the particular embodiments as shown herein.

Claims

1. A VR / AR visualization system, comprising: VR / AR devices, including holographic displays configured to display holographic images to an operator; as well as A computing device operatively coupled to the VR / AR device, the computing device including non-volatile memory and a processor, wherein the computing device is configured to: Receive at least one stored 3D image of the anatomical structure of an object, said at least one stored 3D image including at least one anatomical landmark of the anatomical structure of the object; Mapping data is received from at least one surgical instrument, the mapping data including at least one real-time 3D position of the at least one surgical instrument, spatial data of the anatomical structure of the object, and electrophysiological data of the anatomical structure of the object; Use the aforementioned calibration data to generate a 3D model; By utilizing the spatial data and matching at least one anatomical landmark of the object's anatomical structure with the 3D model, the 3D model and at least one stored 3D image of the object's anatomical structure are registered to a coordinate system; The holographic image is generated using the coordinate system, the holographic image including the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one stored 3D image of the anatomical structure of the object; and The holographic image is modified by attaching or removing the display of the electrophysiological data.

2. The system according to claim 1, wherein, The at least one 3D position of the at least one surgical instrument is received from at least one instrument position sensor of the surgical instrument system operatively coupled to the computing device.

3. The system according to claim 1, wherein, The at least one stored 3D image of the anatomical structure of the object was obtained using a medical imaging device selected from the following: CT scanner, MRI scanner, PET scanner, ultrasound imaging system, and electroanatomical mapping system.

4. The system according to claim 1, wherein, The VR / AR device further includes at least one sensor configured to detect cues generated by the operator.

5. The system according to claim 4, wherein, The prompts are selected from one or more of gestures, eye movements, voice comments, facial expressions, and head movements.

6. The system according to claim 1, wherein, The at least one stored 3D image of the object's anatomical structure includes a 3D image representing at least a portion of the object's internal tissues or internal organs.

7. The system according to claim 4, wherein, The computing device is further configured to modify the holographic image in response to a prompt generated by the operator, the modification including at least one of the following: scaling, rotating, translating, generating a section, adding a portion of the at least one stored 3D image, subtracting at least a portion of the at least one stored 3D image, and changing the rendering of the at least one stored 3D image.

8. The system according to claim 1, wherein, The computing device is further configured to update the holographic image to incorporate changes in the real-time 3D position of at least one of the received surgical instruments.

9. The system according to claim 1, wherein, The computing device is further configured to receive at least one additional dataset and to generate the holographic image to further include the at least one additional dataset. The at least one additional dataset is selected from at least one of the following: fluorescent perforated images of the anatomical structure of the object, a patient vital signs dataset, and a patient demographic dataset.

10. The system according to claim 9, in, The holographic image further includes one or more additional display elements selected from the 2D fluorescent perspective image and the data table, and The one or more additional display elements are positioned anywhere within the field of view of the holographic display as specified by the operator.

11. At least one non-transitory computer-readable storage medium for providing an operator with VR / AR visualization of three-dimensional medical images, said computer-readable storage medium having computer-executable instructions embodied thereon, wherein, The computer-executable instructions, when executed by at least one processor, cause the processor to: Receive at least one stored 3D image of the anatomical structure of an object, said at least one stored 3D image including at least one anatomical landmark of the anatomical structure of the object; Mapping data is received from at least one surgical instrument, the mapping data including at least one real-time 3D position of the at least one surgical instrument, spatial data of the anatomical structure of the object, and electrophysiological data of the anatomical structure of the object; Use the aforementioned calibration data to generate a 3D model; By utilizing the spatial data and matching at least one anatomical landmark of the object's anatomical structure with the 3D model, the 3D model and at least one stored 3D image of the object's anatomical structure are registered to a coordinate system; The coordinate system is used to display a holographic image to the operator, the holographic image including the at least one real-time 3D position of the at least one surgical instrument overlaid on the at least one stored 3D image of the anatomical structure of the object; as well as The holographic image is modified by attaching or removing the display of the electrophysiological data.

12. The at least one non-transitory computer-readable storage medium according to claim 11, wherein, The computer-executable instructions further enable the processor to: At least one cues generated by the operator are detected via at least one sensor of the VR / AR device; and The holographic image is modified in response to the at least one prompt.

13. The at least one non-transitory computer-readable storage medium according to claim 12, wherein, The computer-executable instructions further enable the processor to: The holographic image is modified in response to the at least one prompt by performing at least one modification, the at least one modification being selected from: scaling, rotating, translating, generating a cross section, adding a portion of the at least one stored 3D image, subtracting at least a portion of the at least one stored 3D image, and changing the rendering of the at least one stored 3D image.

14. The at least one non-transitory computer-readable storage medium according to claim 11, wherein, The computer-executable instructions further enable the processor to: Acquire stored medical image data using medical imaging devices selected from the following: CT scanner, MRI scanner, and PET scanner; and The at least one stored 3D image is generated using the stored medical image data.

15. The at least one non-transitory computer-readable storage medium according to claim 11, wherein, The computer-executable instructions further enable the processor to: Acquire stored medical image data using medical imaging devices selected from the following: CT scanners, MRI scanners and PET scanners, ultrasound imaging systems, and electroanatomical mapping systems.

16. The at least one non-transitory computer-readable storage medium according to claim 11, wherein, The computer-executable instructions further enable the processor to: The holographic image is updated to incorporate changes in the real-time 3D position of at least one of the received surgical instruments.

17. The at least one non-transitory computer-readable storage medium according to claim 11, wherein, The computer-executable instructions further enable the processor to: Receive at least one additional dataset; and The holographic image is generated to further include the at least one additional dataset. The at least one additional dataset is selected from at least one of the following: fluorescent perforated images of the anatomical structure of the object, a patient vital signs dataset, and a patient demographic dataset.

18. The at least one non-transitory computer-readable storage medium according to claim 17, in, The holographic image further includes one or more additional display elements selected from the 2D fluorescent perspective image and the data table, and The one or more additional display elements are positioned anywhere within the field of view of the holographic display as specified by the operator.

Images