Real-time fused holographic visualization and guidance for deployment of structural heart repair or replacement products

Abstract

The system 100 and method 200 for performing a structural heart repair or replacement procedure on a patient includes an augmented reality system 102, a tracked instrument 104, a first image acquisition system 108, and a computer system 106. The method includes acquiring a first holographic image dataset 122 from the patient. The computer system 106, in a next step, tracks the tracked instrument 104 using multiple sensors 115, 117, 119, 121 to provide a tracked instrument dataset 132. The method 200 also includes registering the first holographic image dataset 122 and the tracked instrument dataset 132 with the patient. The augmented reality system 102 then, in a next step, renders a first hologram 134 based on the first holographic image dataset 122 from the patient for viewing by a practitioner. The practitioner is thereby enabled to perform the procedure on the patient while viewing the patient and the first hologram 134 using the augmented reality system 102.

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 969,035, filed February 1, 2020. The entire disclosure of the above application is incorporated herein by reference.

[0002] The present disclosure relates to holographic augmented reality applications, and more particularly to medical applications employing holographic augmented reality. [Background technology]

[0003] This section provides background information related to the present disclosure that is not necessarily prior art.

[0004] Image-guided surgery has become standard practice for many different procedures, such as structural heart repair. Image-guided surgery visually correlates intraoperative data with preoperative data. The use of image-guided surgery has been shown to increase the safety and success of these procedures. However, there are a number of known difficulties that can arise during image-guided surgery.

[0005] For example, how intraoperative and preoperative data is presented to the practitioner can directly correlate to the practitioner's surgical performance. Typically, this information is presented on two-dimensional (2D) displays positioned around the patient. This undesirably shifts the practitioner's attention from the patient to the 2D display. It can also place additional strain on the practitioner's neck by forcing them to constantly look up and look at the 2D display while performing the procedure.

[0006] Additionally, determining the optimal angle of instrument insertion can be difficult due to how the intraoperative and preoperative data is displayed. As previously mentioned, this data is typically presented in 2D, meaning that the practitioner must mentally translate the instrument's position and trajectory relative to the data presented on the 2D display. This can undesirably lead to confusion and error as the position and trajectory do not translate well to 2D.

[0007] There is a continuing need for visualization, guidance, and navigation methods and systems for structural heart repair or replacement that involve holographic augmented reality and allow the practitioner to view the operational data and the patient in the same field of view. Desirably, the systems and methods would allow the practitioner to more easily determine the position and trajectory of instruments during the procedure. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] U.S. Patent Application Publication No. 2018 / 0303563 [Patent Document 2] U.S. Patent Application No. 17 / 110,991 [Patent Document 3] U.S. Patent Application No. 17 / 117,841 Summary of the Invention [Means for solving the problem]

[0009] In accordance with the present disclosure, it has surprisingly been discovered that visualization, guidance, and navigation methods and systems for structural heart repair or replacement involve holographic augmented reality, allowing the practitioner to view the operation data and the patient in the same field of view, allowing the practitioner to more easily determine instrument position and trajectory during the procedure.

[0010] In one embodiment, a method for performing a structural heart repair or replacement procedure on a patient includes providing an augmented reality system, a tracked instrument, a first image acquisition system, and a computer system. The computer system has a processor and a memory. The tracked instrument has a plurality of sensors. The first image acquisition system is configured to acquire a first holographic image dataset from the patient. The computer system is in communication with the augmented reality system, the tracked instrument, and the first image acquisition system. The method may include acquiring the first holographic image dataset from the patient by the first image acquisition system. The method may also include tracking the tracked instrument using the plurality of sensors by the computer system to provide the tracked instrument dataset. The method may further include registering the first holographic image dataset and the tracked instrument dataset with the patient by the computer system. The method may additionally include rendering, by the augmented reality system, a first hologram based on the first holographic image dataset from the patient for viewing by a practitioner. The method may also include performing, by a practitioner, a structural heart repair or replacement procedure on the patient while viewing the patient and the first hologram with the augmented reality system, whereby the practitioner employs the augmented reality system for at least one of visualization, guidance, and navigation of the tracked instrument during the structural heart repair or replacement procedure.

[0011] In a further embodiment, a system for performing a structural heart repair or replacement procedure on a patient includes an augmented reality system, a tracked instrument, a first image acquisition system, and a computer system. The tracked instrument has a plurality of sensors employed for detection of the location and orientation of the tracked instrument by the computer system. The first image acquisition system is configured to acquire a first holographic image dataset from the patient. The computer system has a processor and memory and is in communication with the augmented reality system, the tracked instrument, and the first image acquisition system. The computer system is configured with machine-readable instructions to track the tracked instrument using the plurality of sensors and to register the first holographic image dataset and the tracked instrument dataset with the patient to provide the tracked instrument dataset. The augmented reality system is configured to render a first hologram based on the first holographic image dataset from the patient for viewing by a practitioner. The practitioner is thereby enabled to perform the structural heart repair or replacement procedure on the patient while viewing the patient and the first hologram using the augmented reality system. In particular, practitioners employ augmented reality systems for visualization, guidance, and / or navigation of tracked instruments during structural heart repair or replacement procedures.

[0012] In an exemplary embodiment, the present disclosure enables a holographic display of the intended trajectory of a tracked instrument by using spatial computing, augmented reality, and artificial intelligence (AI) to generate a holographic light beam to mimic the intended trajectory of the tracked instrument. The system and method can be used with any augmented reality display, optionally using electromagnetic or optical tracking. This allows the holographic light beam to be adapted to any instrument by design, adjusted to any desired angle, and sized to accommodate any desired needle, catheter, or trocar size.

[0013] It should be appreciated that the present disclosure solves significant problems associated with the navigation, guidance, and positioning of structural heart repair or replacement products. Visualization and orientation of therapies and tools is enabled through holographic visualization and access to the heart via transapical or transaortic techniques or transfemoral approaches. The present disclosure provides comprehensive tools for planning, sizing, and / or pre-orienting implant or therapy delivery, percutaneous entry points, and real-time navigation and guidance by merging or fusing transesophageal echocardiography, transabdominal echocardiography, CT, or MRI images with data from wired and wireless electromagnetic or optical tracking systems.

[0014] The present disclosure also enables pre-planning of optimal angles for instrument or implant insertion. Known standards of care rely on the use of 2D image acquisition to achieve proper access, navigation, and delivery of therapies or devices. The holography of the present disclosure advantageously provides a real-time view by fusing pre-operative imaging with real-time echocardiography that is holographically displayed and precisely registered to the patient using electromagnetic or optical tracking. Holographic representation of cardiac gating may also enable medical practitioners to better understand the heart's full range of motion. Holographic fusion of CT images with intra-operative fluoroscopic imaging to model cardiac motion relative to the cardiac cycle is also contemplated.

[0015] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

[0016] The drawings described herein are for illustration purposes only and are not intended to limit the scope of the present disclosure in any way. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a schematic diagram of a holographic augmented reality visualization and guidance system for structural heart repair or replacement according to one embodiment of the present disclosure, further showing an augmented reality system, a tracked instrument, a computer system, a first image acquisition system, and a second image acquisition system in communication with each other via a computer network. [Figure 2] FIG. 2 is a schematic diagram of the tracked instrument shown in FIG. 1 according to one embodiment of the present disclosure. [Figure 3] 1 is a flowchart illustrating a method for performing a structural heart repair or replacement procedure, according to one embodiment of the present disclosure. [Figure 4] 4 is another flowchart illustrating additional steps of the method for performing the structural heart repair or replacement procedure shown in FIG. 3. [Figure 5] 4 is yet another flowchart illustrating additional steps of the method for performing the structural heart repair or replacement procedure shown in FIG. 3. DETAILED DESCRIPTION OF THE INVENTION

[0018] The following description of technology is merely illustrative of the nature of the subject matter, manufacture, and use of one or more inventions and is not intended to limit the scope, application, or use of any particular invention claimed in this application, or in other applications that may have been filed claiming priority to this application, or in patents issued therefrom. With respect to the methods disclosed, the order of steps presented is exemplary in nature, unless otherwise disclosed, whereby the order of steps may be varied in various embodiments, including instances where certain steps may be performed simultaneously.

[0019] I. Definition Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0020] As used herein, the terms "a" and "an" indicate the presence of "at least one" of that item, and where possible, multiple such items may be present. Unless expressly indicated otherwise, all numerical quantities in this description should be understood to be modified by the word "about," and all geometric and spatial descriptors, in describing the broadest scope of the technology, should be understood to be modified by the word "substantially." When applied to a numerical value, "about" indicates that the calculation or measurement allows for some slight imprecision in the value (with some degree of approach to the precision of the value, approximately or reasonably close to the value, nearly). If for some reason the imprecision provided by "about" and / or "substantially" is not otherwise understood in the art using this ordinary meaning, then "about" and / or "substantially," as used herein, at least indicate the variation that may result from ordinary methods of measurement or using such parameters.

[0021] Although the open-ended term "comprising" is used herein to describe and claim embodiments of the present technology as synonymous with non-limiting terms such as including, containing, or having, embodiments may alternatively be described using more restrictive terms such as "consisting of" or "consisting essentially of." Thus, for a given embodiment reciting a material, component, or process step, the present technology also specifically includes embodiments that consist of or consist essentially of such material, component, or process step, excluding (to consist of) additional materials, components, or processes, and excluding (to consist essentially of) additional materials, components, or processes that affect the critical nature of the embodiment, even if such additional materials, components, or processes are not explicitly recited in this application. For example, a recitation of a process reciting elements A, B, and C specifically contemplates an embodiment consisting of, and consisting essentially of, A, B, and C, excluding element D, which may be recited in the art even though element D is not expressly described as excluded herein.

[0022] As referred to herein, the disclosure of ranges includes the endpoints, unless otherwise specified, and includes all separate values ​​and further subranges within the entire range. Thus, for example, a range "from A to B" or "from about A to about B" includes A and B. The disclosure of a value and range of values ​​for a particular parameter (such as an amount, weight percentage, etc.) does not exclude other values ​​and ranges of values ​​useful herein. It is contemplated that two or more specific exemplified values ​​for a given parameter may define endpoints for a range of values ​​that may be claimed for the parameter. For example, if parameter X is exemplified herein as having a value A and also exemplified as having a value Z, it is contemplated that parameter X may have a range of values ​​from about A to about Z. Similarly, the disclosure of two or more ranges of values ​​for a parameter (whether such ranges are nested, overlapping, or separate) is contemplated to encompass all possible combinations of ranges for values ​​that may be claimed using the endpoints of the disclosed ranges. For example, if a parameter X is exemplified herein as having a value in the range of 1 to 10, or 2 to 9, or 3 to 8, it is contemplated that the parameter X may have other ranges of values, including 1 to 9, 1 to 8, 1 to 3, 1 to 2, 2 to 10, 2 to 8, 2 to 3, 3 to 10, 3 to 9, etc.

[0023] When an element or layer is referred to as being "on," "engaged to," "connected to," or "coupled to" another element or layer, the element or layer may be directly on, engaged to, connected to, or bonded to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly engaged to," "directly connected to," or "directly coupled to" another element or layer, there may be no intervening elements or layers present. Other words used to describe relationships between elements should be construed in a similar manner (e.g., "between" vs. "directly between," "adjacent" vs. "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0024] Terms such as "first," "second," "third," and the like may be used herein to describe various elements, components, regions, layers, and / or sections, but these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may only be used to distinguish one element, component, region, layer, or section from another region, layer, or section. Terms such as "first," "second," and other numerical terms, when used herein, do not imply an arrangement or order unless clearly indicated by context. Thus, a first element, component, region, layer, or section discussed below could be referred to as a second element, component, region, layer, or section without departing from the teachings of the illustrative embodiments.

[0025] Spatially relative terms, such as "inner," "outer," "beneath," "below," "lower," "above," and "upper," may be used herein for ease of description to describe the relationship of one element or feature to another element or feature as depicted in the figures. Spatially relative terms may encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if a device in the figures were turned over, elements described as being "below" or "beneath" other elements or features would now be oriented "above" the other elements or features. Thus, the illustrative term "beneath" can encompass both an upper and lower orientation. A device may be otherwise oriented (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein should be interpreted accordingly.

[0026] As used herein, the term "percutaneous" refers to anything made, done, or performed through the skin.

[0027] As used herein, the term "percutaneous medical procedure" refers to accessing internal organs or tissues through needle puncture of the skin, rather than by using an open approach in which the internal organs or tissues are exposed (typically with a scalpel).

[0028] As used herein, the term "non-vascular," when used in conjunction with a "percutaneous medical procedure," refers to a medical procedure performed on any part of a subject's body separate from the vasculature that is accessed percutaneously. Examples of percutaneous medical procedures include biopsies, tissue ablation, cryotherapy procedures, brachytherapy procedures, endovascular procedures, drainage procedures, orthopedic procedures, pain management procedures, vertebroplasty procedures, pedicle / screw placement procedures, guidewire placement procedures, SI joint fusion procedures, training procedures, and the like.

[0029] As used herein, the term "endovascular," when used in conjunction with "percutaneous medical procedure," refers to a medical procedure performed on a blood vessel (or lymphatic system) that is accessed percutaneously. Examples of endovascular percutaneous medical procedures include aneurysm repair, stent-graft / placement, endovascular prosthesis placement, wire placement, catheterization, filter placement, angioplasty, and the like.

[0030] As used herein, the term "interventional device" or "tracked instrument" refers to a medical instrument used during a non-vascular percutaneous medical procedure.

[0031] As used herein, the term "tracking system" refers to anything used to observe one or more objects undergoing motion and provide a time-ordered sequence of tracking data (e.g., location data, orientation data, etc.) in a tracking coordinate system for further processing. Illustratively, the tracking system may be an electromagnetic tracking system capable of observing an interventional device equipped with a sensor coil as the interventional device moves through a patient's body.

[0032] As used herein, the term "tracking data" refers to information recorded by a tracking system relating to the observation of one or more objects undergoing motion.

[0033] As used herein, the term "tracking coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements specific to a particular tracking system. For example, the tracking coordinate system may be rotated, scaled, etc. from the standard 3D Cartesian coordinate system.

[0034] As used herein, the terms "head-mounted device" or "headset" or "HMD" refer to a display device configured to be worn on the head, with one or more display optics (including lenses) in front of one or more eyes. These terms may be referred to even more generally by the term "augmented reality system," although it should be recognized that the term "augmented reality system" is not limited to display devices configured to be worn on the head. In some cases, the head-mounted device may also include non-transitory memory and a processing unit. An example of a suitable head-mounted device is Microsoft HoloLens®.

[0035] As used herein, the terms "imaging system," "image acquisition device," "image acquisition system," and the like refer to technology that creates a visual representation of the inside of a patient's body. For example, an imaging system may be a computed tomography (CT) system, an X-ray fluoroscopy system, a magnetic resonance imaging (MRI) system, an ultrasound (US) system, etc.

[0036] As used herein, the term "coordinate system" or "augmented reality system coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements specific to a particular augmented reality system or image acquisition system to which it belongs. For example, the headset coordinate system may be rotated, scaled, etc. from the standard 3D Cartesian coordinate system.

[0037] As used herein, the terms "image data" or "image dataset" or "imaging data" refer to information recorded in 3D by an imaging system relating to the observation of the interior of a patient's body. For example, "image data" or "image dataset" may include processed 2D or 3D images or models, such as cross-sectional images, represented by data formatted according to the Digital Imaging and Communications in Medicine (DICOM) standard or other relevant imaging standards.

[0038] As used herein, the terms "imaging coordinate system" or "image acquisition system coordinate system" refer to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of points or other geometric elements specific to a particular imaging system. For example, the imaging coordinate system may be rotated, scaled, etc. from the standard 3D Cartesian coordinate system.

[0039] As used herein, the terms "hologram," "holographic," "holographic projection," or "holographic representation" refer to a computer-generated image projected onto the lenses of a headset. Generally, holograms may be synthetically generated (as in augmented reality (AR)) and are not related to physical reality.

[0040] As used herein, the term "physical" refers to something that is real. Something that is physical is not holographic (or computer-generated).

[0041] As used herein, the terms "two-dimensional" or "2D" refer to something that is expressed in two physical dimensions.

[0042] As used herein, the terms "three-dimensional" or "3D" refer to something that is expressed in three physical dimensions. Elements that are "4D" (e.g., 3D + time and / or motion dimensions) are included in the definition of three-dimensional or 3D.

[0043] As used herein, the term "integrated" can refer to two things being linked or coordinated. For example, a coil sensor can be integrated with an interventional device.

[0044] As used herein, the term "degrees of freedom" or "DOF" refers to several independently varying factors. For example, a tracking system may have six degrees of freedom (or 6DOF), i.e., 3D point and three dimensions of rotation.

[0045] As used herein, the term "real-time" refers to the actual time during which a process or event occurs. In other words, a real-time event occurs live (within milliseconds so that results are immediately available as feedback). For example, a real-time event may be expressed within 100 milliseconds of the event occurring.

[0046] As used herein, the terms "subject" and "patient" are used interchangeably and can refer to any vertebrate animal.

[0047] As used herein, the term "registration" refers to the step of transforming tracking data and body image data into a common coordinate system and creating a holographic representation of images and information relative to the physical patient's body during a procedure, as further described, for example, in U.S. Patent Application Publication No. 2018 / 0303563 to West et al., and also in commonly owned U.S. Patent Application No. 17 / 110,991 to Black et al. and U.S. Patent Application No. 17 / 117,841 to Martin III et al., the entire disclosures of which are incorporated herein by reference.

[0048] II. Systems for Structural Heart Repair or Replacement As shown in FIG. 1 , a holographic augmented reality visualization and guidance system 100 for performing a structural cardiac repair or replacement procedure ( 200 , shown in FIG. 2 ) on a patient includes an augmented reality system 102, a tracked instrument 104, a computer system 106, and a first image acquisition system 108. In some instances, the holographic augmented reality visualization and guidance system 100 may further include a second image acquisition system 110. Each of the augmented reality system 102, the tracked instrument 104, the first image acquisition system 108, and the second image acquisition system 110 may be in selective or persistent communication with the computer system 106, for example, via a computer network 112. Other suitable instruments, tools, equipment, subsystems, etc. for use with the holographic augmented reality visualization and guidance system 100, as well as other networking means, including wired and wireless means of communication between components of the holographic augmented reality visualization and guidance system 100, may also be employed by those skilled in the art as desired.

[0049] 2 , the tracked instrument 104 is an interventional device that is sensorized so that both the location and orientation of the tracked instrument 104 can be determined by the computer system 106. In particular, the tracked instrument has an elongated body, such as a long flexible tube, with multiple sections 114, 116, 118, and 120 disposed along the length of the elongated body, each having one of multiple sensors 115, 117, 119, and 121. For example, the tracked instrument 104 may have a tip section 114, an upper section 116, a middle section 118, and a lower section 120. The tip sensor 115 may be disposed at the tip section 114 of the tracked instrument 104. The upper section sensor 117 may be disposed at the upper section 116 of the tracked instrument 104. The middle portion sensor 119 may be disposed at the middle portion 118 of the tracked instrument 104. The bottom portion sensor 121 may be disposed at the bottom portion 120 of the tracked instrument 104. Each of the sensors 115, 117, 119, 121 is in communication with or otherwise detectable by the computer system 106.

[0050] It should be appreciated that the tracking provided by the tip sensor 115 is particularly beneficial because it may be used by the practitioner as a preselected reference point for the tracked instrument 104. The preselected reference point is configured to be an anchoring point for a trajectory hologram (shown in FIG. 1 and described herein as “142”), such as a holographic light beam that may be generated by the augmented reality system 102. The holographic light beam may assist the practitioner in aligning and moving the tracked instrument 104 along a preferred path or trajectory, as described further herein. It should also be appreciated that one skilled in the art may select any number of preselected reference points within the scope of the present disclosure. In further examples, the preselected reference point may be adjusted in real time by the practitioner during a medical procedure and may alternatively be based on one or more of the other sensors 115, 117, 119, 121, as desired.

[0051] In one example, sensors 115, 117, 119, and 121 may be part of the computer system 106 and / or part of an electromagnetic (EM) tracking system that can be used by the computer system 106 to detect the location and orientation of the physical tracked instrument 104. For example, sensors 115, 117, 119, and 121 may include one or more sensor coils. The computer system 106 can detect the one or more sensor coils and provide tracking data (e.g., with six degrees of freedom) in response to the detection. For example, the tracking data can include real-time 3D position data and real-time 3D orientation data. The tracking system of the computer system 106 can also detect coil sensors that are not positioned on the physical interventional device (e.g., positioned on a fiducial marker or other imaging target).

[0052] Additionally, the plurality of position sensors 115, 117, 119, 121 are configured to assess various additional information about the tracked instrument 104, such as the angular velocity and acceleration of the tracked instrument 104. Non-limiting examples of the plurality of position sensors 115, 117, 119, 121 suitable for determining angular velocity and acceleration include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. Electromagnetic sensors, in particular, enable more accurate real-time object tracking of small objects without line-of-sight limitations.

[0053] Other suitable tracking systems are specifically contemplated, such as optical tracking systems used in conjunction with the augmented reality system 102 and the computer system 106. Embodiments are further contemplated in which the tracked instrument 104 may communicate wirelessly or by transmission through a wired connection with the augmented reality system 102 and the computer system 106. Those skilled in the art will also recognize that multiple position sensors 115, 117, 119, 121 of different types may be employed as desired.

[0054] The tracked instrument 104 may further include an implant or tool configured to be inserted into the patient's heart. Non-limiting examples of tracked instruments 104 and associated implants include needles, catheters, stents, mechanical heart valves, or biological heart valves. In other instances, the implants themselves may be sensorized, at least temporarily, during the procedure to facilitate their tracking.

[0055] In most specific instances, the tracked instrument 104 is a catheter configured for insertion of a cardiac implant, such as a replacement valve, into a patient's heart. In other instances, the tracked instrument 104 is a catheter configured for use in a cardiac ablation procedure. However, one skilled in the art may employ other suitable interventional devices for the tracked instrument 104 depending on the desired procedure within the scope of this disclosure.

[0056] 1 , the first image acquisition system 108 is configured to acquire a first holographic image dataset 122 from a patient. In particular, the first image acquisition system 108 may be configured to acquire the first holographic image dataset 122 from the patient in a pre-operative manner. In one embodiment, the first image acquisition system 108 is one of a magnetic resonance imaging (MRI) machine and a computed tomography (CT) machine. Other suitable types of instrumentation for the first image acquisition system 108 may also be employed, as desired.

[0057] Similarly, the second image acquisition system 110 is configured to acquire a second holographic image dataset 124 from the patient. In particular, the second image acquisition system 110 may be configured to acquire the second holographic image dataset 124 from the patient in an intraoperative manner, most particularly in real time as a procedure is being undertaken. In one embodiment, the second image acquisition system 110 is an ultrasound echocardiogram (ECG) imaging device. Most particularly, the second holographic image dataset 124 may be acquired by a predetermined modality including one of a transthoracic echocardiogram (TTE), a transesophageal echocardiogram (TEE), and an intracardiac echocardiogram (ICE). Other suitable types of instrumentation and modalities for the second image acquisition system 110 may be employed as desired.

[0058] Although the use of both the first image acquisition system 108 and the second image acquisition system 110 is shown and described herein, embodiments in which only one or the other of the first image acquisition system 108 and the second image acquisition system 110 are employed are considered to be within the scope of the present disclosure.

[0059] Continuing to refer to FIG. 1, the computer system 106 of the present disclosure includes at least one processor 126 and at least one memory 128 having tangible, non-transitory machine-readable instructions 130 stored therein.

[0060] The one or more processors 126 may perform functions related to the operation of the holographic augmented reality visualization and guidance system 100. The one or more processors 126 may be any type of general-purpose or special-purpose processor. In some cases, multiple processors 126 may be utilized according to other embodiments. Indeed, the one or more processors 126 may include, by way of non-limiting example, one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture.

[0061] Memory 128 may be one or more memories of any type suitable for the local application environment and may be implemented using any suitable volatile or non-volatile data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. For example, memory 128 may comprise any combination of random access memory (RAM), read-only memory (ROM), static storage such as a magnetic or optical disk, a hard disk drive (HDD), or any other type of non-transitory machine- or computer-readable medium. Instructions 128 stored in memory may include program instructions or computer program code that, when executed by one or more processors 126, enable holographic augmented reality visualization and guidance system 100 to perform the tasks described herein.

[0062] The machine-readable instructions 130 may include modules, which may be implemented as one or more of functional logic, hardware logic, electronic circuitry, software modules, etc. The modules may include one or more of an augmented reality system module, an image acquisition module, an instrument tracking module, an image dataset registration module, a hologram rendering module, an image registration module, a trajectory hologram rendering module, and / or other suitable modules, as desired.

[0063] The computer system 106 is in communication with the augmented reality system 102, the tracked instrument 104, the first image acquisition system 108, and the second image acquisition system 110, for example, via a network 112, and is configured by machine-readable instructions 130 to operate according to the method 200 described further herein. The computer system 106 may be separate from and remote from the augmented reality system 102, or may be provided with the augmented reality system 102 as a single integrated unit, if desired.

[0064] It should be appreciated that the network 112 of the holographic augmented reality visualization and guidance system 100 may include, by way of non-limiting example, a wireless access network such as LTE or 5G, a local area network (LAN), a wide area network (WAN) such as the Internet, or a wireless LAN (WLAN). It should be appreciated that this is not intended to be limiting, and that the scope of the present disclosure includes implementations in which one or more computing platforms of the holographic augmented reality visualization and guidance system 100 may be operatively linked via some other communications coupling. One or more computing platforms may be configured to communicate with a networked environment via wireless or wired connections. Additionally, in embodiments, one or more computing platforms may be configured to communicate directly with each other via wireless or wired connections. Examples of the one or more computing platforms may include, but are not limited to, a smartphone, a wearable device, a tablet, a laptop computer, a desktop computer, an Internet of Things (IoT) device, or other mobile device or a stationary device such as a standalone server, a networked server, or an array of servers.

[0065] In particular embodiments, the computer system 106 may be configured to track the tracked instrument 104 using multiple sensors 115, 117, 119, 121 to provide a tracked instrument dataset 132. The tracked instrument dataset 132 may be stored in the memory 128. In particular, the location and orientation of the tracked instrument 104 in physical space may be stored as the tracked instrument dataset 132, for example.

[0066] The computer system 106 may be further configured to register the first holographic image dataset 122 from the first image acquisition system 108 and the tracked instrument dataset 132 acquired by the computer system 106 with the patient, as also described further herein.

[0067] Continuing with reference to FIG. 1 , the augmented reality system 102 is configured to render multiple holograms 134, 136, 138, 140, and 142 in accordance with the method 200 of the present disclosure. In particular, the augmented reality system 102 may be a mixed reality (MR) display, such as MR smart glasses or a MR head-mounted display. Non-limiting examples of the augmented reality system 102 include Magic Leap One® or Microsoft Hololens®. It should be appreciated that other types of MR displays may be used for the augmented reality system 102 as long as they are capable of superimposing computer-generated images over real-world objects. Additionally, although the augmented reality system 102 is primarily described herein as a head-mounted display, it should be understood that other types of displays that are not head-mounted but are capable of generating holograms 134, 136, 138, and 140 and superimposing them over a view of the real world may be employed as needed.

[0068] In instances where the augmented reality system 102 does not include the computer system 100, it should be appreciated that the augmented reality system 102 may further include additional non-transitory memory and a processing unit (which may include one or more hardware processors) that may assist in rendering or generating the holograms 134, 136, 138, 140, 142. The augmented reality system 102 may also include a camera for recording one or more images, one or more image generation components for generating / displaying visualizations of the holograms 134, 136, 138, 140, 142, and / or other visualization and / or recording elements.

[0069] In yet a further example, it should be appreciated that the augmented reality system 102 may also include a plurality of position sensors 144. The plurality of position sensors 144 of the augmented reality system 102 are configured to determine various position information for the augmented reality system 102, such as an approximated position, orientation, angular velocity, and acceleration in three-dimensional (3D) space of the augmented reality system 102. In particular, it should be appreciated that this allows the holographic image to be accurately displayed in the practitioner's field of view during surgery.

[0070] Non-limiting examples of the plurality of position sensors 144 include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. Those skilled in the art should further recognize that different types and numbers of the plurality of position sensors 144 may be employed in the augmented reality system 102, as required by, for example, the procedure or situation in which the augmented reality system 102 is being used.

[0071] 1 , for example, the holograms 134, 136, 138, 140, 142 generated by the augmented reality system 102 may include a first hologram 134, a tracked instrument hologram 136, a second hologram 138, an animated hologram 140, and a trajectory hologram 142. The first hologram 134 generated by the augmented reality system 102 may be based on a first holographic image dataset 122 from the patient. The tracked instrument hologram 136 generated by the augmented reality system 102 may be based on the tracked instrument dataset 132. The second hologram 138 generated by the augmented reality system 102 may be based on a second holographic image dataset 124. The animated hologram 140 may be based on processing by the computer system 106 of the second holographic image dataset 124 to provide an animated hologram dataset 148, which is described further herein. The trajectory hologram 142 may be based on a trajectory data set 146, which may be selected either manually or automatically and stored in the memory 128 of the computer system 106, as further described herein.

[0072] In addition to rendering or generating the various holograms 134, 136, 138, 140, and 142, the augmented reality system 102 may be further configured to present multiple operational information or details to the practitioner. For example, the augmented reality system 102 may project multiple operational information onto a real-world object, such as a patient. The operational information may include, for example, real-time navigation instructions or guidance for a trajectory to be adopted. It should be appreciated that the augmented reality system 102 may project multiple operational information onto various real-world objects, such as the tracked instrument 104, as well as onto the various rendered holograms 134, 136, 138, 140, and 142, as desired.

[0073] Desirably, this generation of operational information or details allows the practitioner to simultaneously view the patient and multiple operational information in the same field of view. Also, generating operational information or details along with the various holograms 134, 136, 138, 140, 142 allows the practitioner to plan, size, or pre-orient the tracked instrument 104 during surgery.

[0074] 1, computer system 106 is in communication with augmented reality system 102 and tracked instrument 104. Computer system 106 is configured to store and generate operational information either through manual intervention by a practitioner or other medical professional or automatically based on machine-readable instructions 130 encoded on memory 128. For example, operational information may be generated in augmented reality system 102 in response to a sensor-determined position or orientation of tracked instrument 104, such as by an algorithm, artificial intelligence (AI) protocol, or other practitioner-input data or threshold.

[0075] Additionally, computer system 106 is further configured to allow the practitioner to selectively adjust the plurality of operational information in real time. For example, the practitioner may be able to adjust the position or orientation of orbital hologram 142. Additionally, the practitioner may be able to determine which of the plurality of operational data is actively presented to the practitioner. It should be appreciated that other settings and attributes of the plurality of operational information may be adjusted by the practitioner in real time within the scope of the present disclosure.

[0076] In particular, it should be appreciated that the augmented reality system 102 of the present disclosure advantageously enables a practitioner to perform the method 200 for structural heart repair or replacement on a patient while viewing the patient and first hologram 134 and, optionally, the instrument hologram 136 using the augmented reality system 102. Similarly, the practitioner is advantageously enabled to employ the augmented reality system 102 for at least one of visualization, guidance, and navigation of the tracked instrument 104 during the structural heart repair or replacement procedure, as further described herein with respect to the method 200 of the present disclosure.

[0077] III. Methods for Structural Heart Repair or Replacement 3 illustrates a flow diagram of an example method 200, according to one embodiment of the present disclosure. The method 200 may include a first step 202 of providing a holographic augmented reality visualization and guidance system 100 as described herein.

[0078] In a second step 204, the method 200 may then include acquiring, by the first image acquisition system 108, a first holographic image dataset 122 from the patient. The method 200 may include a third step 206 of tracking, by the computer system 106, the tracked instrument 104 using the multiple sensors 115, 117, 119, 121 to provide a tracked instrument dataset 132.

[0079] The method 200 may then include a fourth step 208 of registering, by the computer system 106, the first holographic image dataset 122 and the tracked instrument dataset 132 with the patient.

[0080] In a fifth step 210, the method 200 may then include rendering, by the augmented reality system 102, a first hologram 134 based on the first holographic image dataset 122 from the patient for viewing by the practitioner. Optionally, a tracked instrument hologram 136 may also be rendered in the fifth step 210, also based on the tracked instrument dataset 132, for viewing by the practitioner. It should be appreciated that when the instrument hologram 134 is shown, the practitioner may employ visualization of it to assist in positioning the real-world tracked instrument 104.

[0081] The method 200 may then include a sixth step 212 of the practitioner performing the structural heart repair or replacement procedure on the patient while viewing the patient and first hologram 134, and optionally the instrument hologram 136, using the augmented reality system 102. Under the sixth step 212, the practitioner employs the augmented reality system 102 for at least one of visualization, guidance, and navigation of the tracked instrument 104 during the structural heart repair or replacement procedure.

[0082] It should be appreciated that visualization, guidance, and / or navigation of the tracked instrument 104 during a structural heart repair or replacement procedure may advantageously also involve further steps that take into account intra-procedural movement of the patient during the procedure.

[0083] 4 , the method 200 may further include a seventh step 214 of acquiring a second holographic image dataset 124 from the patient by the second image acquisition system 110. In particular, the second holographic image dataset 124 may be intraoperative and acquired in real time during a structural heart repair or replacement procedure. Once the second holographic image dataset 124 is acquired, the method may further include an eighth step 216 of registering the second holographic image dataset 124 with the patient by the computer system 106. Following registration of the second holographic image dataset 124, the method may include a ninth step 218 of rendering a second hologram 138 based on the second holographic image dataset 124 from the patient by the augmented reality system 102.

[0084] The method 200 may then further include a tenth step 220 of selecting, either automatically or manually, by the computer system 106, e.g., according to machine-readable instructions 130 stored in the memory 128 of the computer system 106, a predetermined portion (not shown) of at least one of the first hologram 134 and the second hologram 138 to be animated. As a non-limiting example, the computer system 106 may also automatically or manually merge, stitch, or otherwise digitally combine the first hologram 134 and the second hologram 138 to provide a fused or composite hologram such that a practitioner may use the computer system 106 to select a predetermined portion of the composite hologram for animation. In one embodiment, the predetermined portion is associated with one of the patient's heart and chest. If the predetermined portion is associated with the patient's heart, the animated hologram 140 generated by the computer system 106 may depict the rhythmic movement of the heartbeat. If the predetermined portion relates to the patient's chest, the animated hologram 140 generated by the computer system 106 may depict the respiratory cycle movement of the chest.

[0085] In certain instances, the animation may be based on real-time imaging of the heart via a second image acquisition system 110, such as an ultrasound ECG, to allow the animated hologram 140 to beat in rhythm with the patient's heart in real time. In yet another instance, the animation may be based on a holographic representation of cardiac gating, for example, from a gated CT scan that measures the respiratory cycle relative to the start (atrial systole) and end (atrial diastole) of the cardiac cycle. This may allow for a more complete understanding of the full range of motion of the patient's heart.

[0086] 4 , the method may further include an eleventh step 222 of generating, by the computer system 106, an animated hologram dataset 148 for a predetermined portion of at least one of the first hologram 134 and the second hologram 138 based on the real-time acquired second holographic image dataset 124. Following generation of the animated hologram dataset 148, the method 200 may further include a twelfth step 224 of rendering, by the augmented reality system 102, an animated hologram 140 from the animated hologram dataset for viewing by a practitioner during the structural heart repair or replacement procedure.

[0087] Advantageously, by allowing the practitioner to view the animated hologram 140, either in addition to or instead of the steady or otherwise static forms of the first hologram 134 and the second hologram 138, the practitioner may insert the tracked instrument 104 and / or deploy the associated implant into the patient with more confidence compared to conventional procedures without the system 100.

[0088] It should be appreciated that visualization, guidance, and / or navigation of the tracked instrument 104 during a structural heart repair or replacement procedure may advantageously involve further steps that allow for pre-determination or planning and optimization of both the percutaneous entry point and the trajectory of the tracked instrument prior to and / or during the procedure.

[0089] 5, the method 200 may further include a thirteenth step 226 of using the computer system 106 to plan a predetermined trajectory of insertion of the tracked instrument 104 into the patient to provide the predetermined trajectory dataset 146. Once planning is complete, the method 200 may further include a fourteenth step 228 of rendering, by the augmented reality system 102, a trajectory hologram 142 based on the predetermined trajectory dataset 146.

[0090] A trajectory hologram 142 of the present disclosure may include, for example, holographic rays illustrating a predetermined trajectory of the tracked instrument 104. The holographic rays may be straight or curved, or may have one or more angles, to describe an optimal path for the tracked instrument 104. The trajectory hologram 142 may also be used to clearly identify intravascular landing points within the patient for the tracked instrument 104, such as percutaneous entry points on the patient and preferred landing zones in the patient's cardiac anatomy for implants to be deployed.

[0091] It should be appreciated that the overall size, shape, and orientation of the trajectory hologram 142 generated by the augmented reality system 102 may be based on operational information from the computer system 106, including pre-operative and intra-operative data. It should also be appreciated that the operational information may include additional data from other sensors in the operational area and other holographic projections 134, 136, 138, 140 being generated by the augmented reality system 102.

[0092] The pre-operative data may include, for example, information related to a patient obtained prior to a medical procedure using the first holographic image acquisition system 108. Non-limiting examples of pre-operative data include still images or recordings from a transesophageal echocardiogram, a transabdominal echocardiogram, a transthoracic echocardiogram, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, or an X-ray. It should be appreciated that the pre-operative data may include information from other diagnostic medical procedures, as desired.

[0093] The intra-operative data may include, for example, patient-related information obtained in real time during a medical procedure using the second holographic image acquisition system 110. For example, the diagnostic medical procedures listed above may be performed simultaneously with the current medical procedure.

[0094] In further embodiments, the plurality of operational information includes fused pre-operative and intra-operative data. The fused pre-operative and intra-operative data merges the pre-operative data with the intra-operative data in a manner that presents a more concise and approximate image and animation to the practitioner. In some cases, the fusion is performed manually. In other cases, the fusion is performed by the computer system 106, for example, using at least one of a plurality of algorithms described in the machine-readable instructions 130 or via artificial intelligence (AI).

[0095] As explained above, in certain embodiments, the holographic beam may be anchored on a preselected reference point on the tracked instrument 104. In yet further instances, the intended trajectory may be adjusted in real time by the practitioner via the computer system 106, for example, to address unforeseen complications that may arise during the procedure.

[0096] It is believed that the trajectory hologram 142, along with other holographic projections, may minimize the risk of complications associated with transapical approach procedures. For example, the overall size of the heart, artery, or vein incision may be minimized because the practitioner can be more precise with the intended trajectory of the tracked instrument 104 via the trajectory hologram 142, such as a holographic beam.

[0097] Additionally, the trajectory hologram 142 is believed to allow the practitioner to more easily find the optimal approach angle for valve implantation or paravalvular leak (PVL) closure, and allowing the practitioner to more easily find the optimal approach angle helps the practitioner avoid critical structures such as lung tissue, coronary arteries, and the left anterior descending artery.

[0098] For example, a holographic representation of a real-time intraoperative scan may be overlaid with a holographic representation of a preoperative scan. The fused preoperative and intraoperative data may further include a holographic fusion of a CT scan image with intraoperative fluoroscopic imaging to model the heart's motion relative to the cardiac cycle. Additionally, the fused preoperative and intraoperative data may further include an overlay that alerts the practitioner to sensitive areas in the patient's body that should not be touched by the tracked instrument 104. It should be appreciated that different applications of the fused preoperative and intraoperative data may be employed by those skilled in the art within the scope of this disclosure.

[0099] In yet another embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 may be configured to predict the shape of an implant, such as a valve, after the implant has been deployed by the tracked instrument 104. The predicted shape of the implant may also be visualized, for example, in the form of a hologram further generated by the augmented reality system 102.

[0100] In yet a further embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 may be configured to facilitate coaxial deployment, or in other words, centering of the valve within the endovascular structure, using the tracked instrument 104. The augmented reality system 102 may be employed to generate "error bars" or coloring (e.g., "green" for acceptable and "red" for unacceptable) to guide the practitioner in the coaxial deployment during the procedure.

[0101] In yet additional embodiments, the computer system 106 of the holographic augmented reality visualization and guidance system 100 may be employed to predict remodeling of vascular or cardiac structures that is expected to result from the deployed location of the implant over time. In particular, the computer system 106 may estimate or predict how the heart will be removed over time in a particular placement, thereby enabling planning of the placement in a manner that minimizes possible remodeling over time.

[0102] In yet another embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 may be used to aid in prosthesis or implant size selection prior to completion of the procedure. Employing the holographic augmented reality visualization and guidance system 100 to select appropriate sizing may minimize the possibility of patient-prosthesis mismatch (PPM), which can sometimes occur when the implanted prosthetic valve is either too small or too large for the patient.

[0103] Advantageously, the holographic augmented reality visualization and guidance system 100 and method 200 for structural heart repair or replacement allows the practitioner to simultaneously view the operational data and the patient in the same field of view via the augmented reality system 102. Additionally, the holographic light beam allows the practitioner to easily determine the intended trajectory of the tracked instrument 104.

[0104] It should also be appreciated that method 200 may allow a practitioner to customize how much of the key operational information is shown in augmented reality system 102. Additionally, the practitioner may customize settings and attributes of the operational information using, for example, computer system 106. Method 200 further allows a practitioner to perform instrument insertion at any desired angle and without the need for additional disposable physical instrument guides.

[0105] It should further be appreciated that the system 100 and method 200 of the present disclosure are particularly well adapted for use in patients requiring mechanical aortic and mitral valve prostheses or who require repeat endocardial ablation procedures.

[0106] While certain representative examples and details have been set forth for the purpose of illustrating the present invention, it will be apparent to those skilled in the art that various modifications can be made without departing from the scope of the present disclosure, as further set forth in the appended claims below.

Claims

1. 1. A system for performing a structural heart repair or replacement procedure on a patient, comprising: an augmented reality system; a tracked instrument having a plurality of sensors; a first image acquisition system configured to acquire a first image data set for generating a first hologram; a second image acquisition system configured to acquire a second image data set for generating a second hologram; a computer system having a processor and a memory, the computer system in communication with the augmented reality system, the tracked instrument, the first image acquisition system, and the second image acquisition system; tracking the tracked instrument using the plurality of sensors to provide a tracked instrument dataset; registering a first image data set with the patient to generate the first hologram; registering a second image data set with the patient to generate the second hologram; generating an animated hologram data set for a predetermined portion of at least one of the first hologram and the second hologram based on the second image data set acquired in real time; planning a predetermined trajectory for insertion of the tracked instrument into the patient to provide a predetermined trajectory data set; a computer system configured with machine-readable instructions to perform Equipped with the augmented reality system renders the first hologram on a display of the augmented reality system based on the first image data set for viewing by a practitioner; the augmented reality system renders the second hologram based on the second image data set on a display of the augmented reality system for viewing by a practitioner; the augmented reality system combining the first hologram and the second hologram to render a combined hologram on a display of the augmented reality system for viewing by a practitioner; the augmented reality system combining animated hologram data sets for predetermined portions of the first and second holograms with the combined hologram to render an animated hologram on a display of the augmented reality system for viewing by a practitioner; the augmented reality system renders a trajectory hologram on a display of the augmented reality system for viewing by a practitioner based on the predetermined trajectory data set, the trajectory hologram including successive holographic light rays projected from a tip sensor of the tracked instrument and extending outward along a path in space to depict a predicted future trajectory of the tracked instrument, the holographic light rays visually representing light rays emanating from the tip sensor and illustrating alignment and movement of the tracked instrument along the trajectory hologram, the tip sensor being a preselected reference point for the tracked instrument, the preselected reference point being an anchoring point for the successive holographic light rays; whereby the practitioner is enabled to perform the structural heart repair or replacement procedure on the patient while viewing the patient and the first hologram using the augmented reality system, and employ the augmented reality system for at least one of visualization, guidance, and navigation of the tracked instrument during the structural heart repair or replacement procedure; the first hologram, the second hologram, the combined hologram, the animated hologram, and the trajectory hologram are synthetically generated within an augmented reality system. system.

2. The system of claim 1 , wherein the first image acquisition system is one of a magnetic resonance imaging (MRI) machine and a computed tomography (CT) machine.

3. The system described in claim 2, wherein the first image dataset is pre-operative.

4. 2. The system of claim 1, wherein the second image data set is acquired by a predetermined modality comprising one of a transthoracic echocardiogram (TTE), a transesophageal echocardiogram (TEE), and an intracardiac echocardiogram (ICE).

5. 10. The system of claim 1, wherein the computer system is further configured to generate an animated hologram data set for a predetermined portion of at least one of the first computer-generated image and the second computer-generated image based on the second image data set acquired in real time.

6. 6. The system of claim 5, wherein the augmented reality system is further configured to render animated computer-generated images from the animated hologram dataset onto a headset lens for viewing by the practitioner during the structural heart repair or replacement procedure.

7. 7. The system of claim 6, wherein the computer system is further configured to select the predetermined portion of at least one of the first computer-generated image and the second computer-generated image to be animated.

8. The system of claim 7 , wherein the predetermined portion is associated with one of the patient's heart and chest.

9. 9. The system of claim 8, wherein the predetermined portion relates to the patient's heart and the animated hologram depicts the beating rhythmic motion of the heart.

10. 9. The system of claim 8, wherein the predetermined portion relates to the chest of the patient and the animated hologram depicts respiratory cycle motion of the chest.

11. The system of claim 1 , wherein the tracked instrument has an elongated body with a tip portion, an upper portion, a lower portion, and a middle portion.

12. The system of claim 11 , wherein the plurality of sensors includes a tip sensor disposed at the tip portion of the tracked instrument.

13. The system of claim 12 , wherein the plurality of sensors includes an upper portion sensor disposed in the upper portion, a lower portion sensor disposed in the lower portion, and a middle portion sensor disposed in the middle portion.

14. The system of claim 1 , wherein the augmented reality system is further configured to render a computer-generated image of a trajectory in a headset lens based on the predetermined trajectory data set.

15. 10. The system of claim 1, wherein the second image acquisition system is configured to acquire the second image dataset from the patient, the second image dataset being intra-operative and acquired in real time during the structural heart repair or replacement procedure.

16. The system of claim 15 , wherein the second image acquisition system is an echocardiogram (ECG) imaging device.

Images