Real-time fused holographic visualization and guidance for deployment of structural heart repair or replacement products

Abstract

A system (100) and method (200) for performing a structural heart repair or replacement procedure on a patient includes an augmented reality system (102), a tracked instrument (104), a first image acquisition system (108), and a computer system (106). The method includes a step of acquiring a first holographic image data set (122) from the patient. In a next step, the computer system (106) tracks the tracked instrument (104) using a plurality of sensors (115, 117, 119, 121) to provide a tracked instrument data set (132). The method (200) also includes a step of registering the first holographic image data set (122), the tracked instrument data set (132), and the patient. Then, in a next step, the augmented reality system (102) renders a first holographic map (134) based on the first holographic image data set (122) from the patient for viewing by a practitioner. Thus, the practitioner is allowed to perform the procedure on the patient while viewing the patient and the first holographic map (134) using the augmented reality system (102).

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 62 / 969,035, filed February 1, 2020. The entire disclosure of the aforementioned application is incorporated herein by reference. Technical Field

[0003] This disclosure relates to augmented reality (AR) applications, particularly medical applications using augmented reality. Background Technology

[0004] This section provides background information relevant to this disclosure, which is not necessarily prior art.

[0005] Image-guided surgery has become standard practice in many different procedures, such as structural cardiac repair. Image-guided surgery visually correlates intraoperative data with preoperative data. The use of image-guided surgery has been shown to improve the safety and success rate of these procedures. However, many known challenges arise in image-guided surgery.

[0006] For example, how intraoperative and preoperative data are presented to the practitioner can be directly related to the practitioner's surgical performance. Typically, this information is displayed on a two-dimensional (2D) monitor located around the patient. Undesirably, this diverts the practitioner's attention from the patient to the 2D monitor. Furthermore, because the practitioner must constantly look up at the 2D monitor during surgery, this puts additional stress on the practitioner's neck.

[0007] Furthermore, determining the optimal instrument insertion angle can be difficult due to the way intraoperative and preoperative data are displayed. As mentioned earlier, this data is typically displayed in 2D, meaning practitioners must mentally translate the instrument's position and trajectory into data displayed on a 2D monitor. Since position and trajectory cannot be readily translated into 2D, this can unintentionally lead to confusion and errors.

[0008] There is a persistent need for a visualization, guidance, and navigation method and system involving holographic augmented reality for structural cardiac repair or replacement, which would enable practitioners to view procedural data and the patient within the same field of view. It is anticipated that this system and method would allow practitioners to more easily determine the position and trajectory of instruments during the procedure. Summary of the Invention

[0009] Consistent with this disclosure, a visual, guidance, and navigation system and method involving holographic augmented reality for structural cardiac repair or replacement has been surprisingly discovered, which enables practitioners to view operational data and the patient in the same field of view and makes it easier for practitioners to determine the position and trajectory of instruments during the procedure.

[0010] In one embodiment, a method for performing structural heart repair or replacement surgery on a patient includes providing an augmented reality system, a tracked instrument, a first image acquisition system, and a computer system. The computer system has a processor and memory. The tracked instrument has multiple sensors. The first image acquisition system is configured to acquire a first holographic image dataset from the patient. The computer system communicates with the augmented reality system, the tracked instrument, and the first image acquisition system. The method may include acquiring the first holographic image dataset from the patient via the first image acquisition system. The method may also include tracking the tracked instrument using multiple sensors via the computer system to provide a tracked instrument dataset. The method may further include registering the first holographic image dataset, the tracked instrument dataset, and the patient via the computer system. The method may further include rendering a first hologram based on the first holographic image dataset from the patient using the augmented reality system for viewing by a practitioner. The method may further include performing structural heart repair or replacement surgery on the patient while the practitioner views the patient and the first hologram using the augmented reality system. Thus, in structural heart repair or replacement surgery, the practitioner uses the augmented reality system for at least one of visualization, guidance, and navigation of the tracked instrument.

[0011] In another embodiment, a system for performing structural heart repair or replacement surgery on a patient includes an augmented reality system, a tracked instrument, a first image acquisition system, and a computer system. The tracked instrument has multiple sensors for detecting the position and orientation of the tracked instrument via the computer system. The first image acquisition system is configured to acquire a first holographic image dataset from the patient. The computer system has a processor and memory and communicates with the augmented reality system, the tracked instrument, and the first image acquisition system. The computer system is configured via machine-readable instructions to: track the tracked instrument using the multiple sensors to provide a tracked instrument dataset; and register the first holographic image dataset, the tracked instrument dataset, and the patient. The augmented reality system is configured to render a first hologram based on the first holographic image dataset from the patient for a practitioner to view. Therefore, the practitioner is allowed to perform structural heart repair or replacement surgery on the patient while viewing the patient and the first hologram using the augmented reality system. In particular, during structural heart repair or replacement surgery, the practitioner uses the augmented reality system for at least one of visualization, guidance, and navigation of the tracked instrument.

[0012] In one exemplary embodiment, this disclosure allows for the holographic display of the expected trajectory of a tracked instrument by generating holographic rays using spatial computing, augmented reality, and artificial intelligence (AI). The system and method can be used with any augmented reality display and optionally employ electromagnetic or optical tracking. This enables the holographic rays to be designed to adapt to any instrument, adjusted to any desired angle, and sized to fit any desired needle, catheter, or cannula size.

[0013] It should be understood that this disclosure addresses significant problems related to the navigation, guidance, and positioning of structural cardiac repair or replacement products. It allows for the visualization and positioning of treatments and tools through holographic visualization and access to the heart via transapical, transaortic, or transfemoral approaches. This disclosure provides a comprehensive tool for planning, calibrating, and pre-determining delivery to implants or treatments, percutaneous access points, and real-time navigation and guidance by incorporating or fusing images from transesophageal echocardiography, transabdominal echocardiography, computed tomography (CT), or magnetic resonance imaging (MRI) with data from wired and radio electromagnetic or optical tracking systems.

[0014] This disclosure also allows for the pre-planning of the optimal angle for instrument or implant placement. Known standards of care rely on the use of 2D image acquisition to achieve proper access, guidance, and delivery of treatments or devices. The holography of this disclosure advantageously provides a real-time view by fusing preoperative imaging with real-time echocardiography, which is displayed holographically and accurately registered for the patient using electromagnetic or optical tracking. Holographic representation of cardiac gating can also allow interventional practitioners a deeper understanding of the entire range of cardiac motion. Holographic fusion of CT images with intraoperative fluorescence imaging is also considered to simulate cardiac motion associated with the cardiac circulation.

[0015] Further applicability will become apparent from the description provided herein. It should be understood that the descriptions and specific examples are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0016] The accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure in any way.

[0017] Figure 1This is a schematic diagram of a holographic augmented reality visualization and guidance system for structural heart repair or replacement according to an embodiment of the present disclosure, and further illustrates an augmented reality system, a tracked instrument, a computer system, a first image acquisition system, and a second image acquisition system communicating with each other via a computer network;

[0018] Figure 2 According to one embodiment of this disclosure Figure 1 A schematic diagram of the tracked instrument shown;

[0019] Figure 3 This is a flowchart illustrating a method for performing a structural heart repair or replacement surgery according to an embodiment of the present disclosure;

[0020] Figure 4 It is shown Figure 3 Another flowchart illustrating the additional steps of a method for performing structural heart repair or replacement surgery; and

[0021] Figure 5 It is shown Figure 3 This is another flowchart illustrating the additional steps of a method for performing structural heart repair or replacement surgery. Detailed Implementation

[0022] The following description of the technology is merely exemplary in relation to the subject matter, manufacture, and use of one or more inventions, and is not intended to limit the scope, application, or use of any particular invention claimed in this application or any other application that may be filed claiming priority to this application or any patent published thereby. Regarding the disclosed methods, unless otherwise disclosed, the order of the proposed steps is essentially exemplary, and therefore, the order of the steps may differ in various embodiments, including cases where certain steps may be performed simultaneously.

[0023] I. Definition

[0024] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0025] As used herein, the terms “a” and “an” indicate the presence of “at least one” item; where possible, multiple such items may exist. Unless otherwise expressly stated, all numerical quantities in this specification shall be understood to be modified by the word “approximately”, and all geometric and spatial descriptors shall be understood to be modified by the word “substantially” when describing the broadest range of techniques. When applied to numerical values, “approximately” means that the calculated or measured value allows for some slight imprecision (some close to the exact value; approximate or reasonably close to the value; nearly). If for some reason the imprecision provided by “approximately” and / or “substantially” is not understood in this general sense in the art, then “approximately” and / or “substantially” as used herein at least indicates a variation that can be caused by common methods of measuring or using these parameters.

[0026] Although the open-ended term "comprising" is used herein as a synonym for non-limiting terms such as including, comprising, or having to describe and declare embodiments of the present technology, alternatively, more restrictive terms (e.g., "consisting of" or "substantially composed of") may be used to describe embodiments. Therefore, for any given embodiment that describes materials, components, or process steps, the present technology also specifically includes embodiments that consist of or substantially consist of such materials, components, or process steps, excluding additional materials, components, or processes (consisting of) and not including additional materials, components, or processes (substantially composed of) that affect important characteristics of the present embodiment, even if such additional materials, components, or process steps are not expressly described herein. For example, a description of processes describing elements A, B, and C specifically contemplates embodiments that consist of and substantially consist of A, B, and C, excluding element D, which may be described in the art, even if element D is not expressly described herein as excluded.

[0027] As stated herein, unless otherwise specified, the disclosure of a range includes endpoints and encompasses all distinct values ​​throughout the range and further subdivisions. Thus, a range such as “from A to B” or “from approximately A to approximately B” includes both A and B. The disclosure of values ​​and ranges of values ​​for a particular parameter (such as quantity, weight percentage, etc.) does not exclude other values ​​and ranges of values ​​useful herein. It is conceivable that two or more specific exemplary values ​​of a given parameter can define endpoints of the range of values ​​that can be declared for that parameter. For example, if parameter X is exemplified herein as having the value A and also exemplified as having the value Z, it is conceivable that parameter X can have a range of values ​​from approximately A to approximately Z. Similarly, it is conceivable that the disclosure of two or more ranges of values ​​for a parameter (whether these ranges are nested, overlapping, or distinct) includes all possible combinations of ranges of values ​​that can be declared using the endpoints of the disclosed range. For example, if parameter X is exemplified in this document as having a value in the range of 1-10, 2-9, or 3-8, it is also conceivable that parameter X could have other ranges of values ​​including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, etc.

[0028] When an element or layer is referred to as "on another element or layer," "joined to," "connected to," or "coupled to" another element or layer, it may be directly on, joined to, connected to, or coupled to another element or layer, or there may be intermediate elements or layers present. Conversely, when an element is referred to as "directly on another element or layer," "directly joined to," "directly connected to," or "directly coupled to" another element or layer, there may be no intermediate elements or layers present. Other terms used to describe relationships between elements should be interpreted in a similar manner (e.g., "between" vs. "directly between," "adjacent" vs. "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the related listed items.

[0029] Although the terms first, second, third, etc., may be used herein to describe various elements, components, regions, layers, and / or portions, these elements, components, regions, layers, and / or portions should not be limited by these terms. These terms are only used to distinguish one element, component, region, layer, or portion from another. Unless the context clearly indicates otherwise, terms such as “first,” “second,” and other numerical terms used herein do not imply sequence or order. Therefore, a first element, component, region, layer, or portion discussed below may be referred to as a second element, component, region, layer, or portion without departing from the teachings of the exemplary embodiments.

[0030] For ease of description, this document uses spatial relative terms such as “inside,” “outside,” “below,” “below,” “above,” and “upper” to describe the relationship between one element or feature and another, as shown in the figure. In addition to the orientation shown in the figure, spatial relative terms are also intended to encompass different orientations of the device during use or operation. For example, if the device in the figure is flipped, an element described as “below other elements or features” or “below other elements or features” would be oriented “above” other elements or features. Therefore, the example term “below” can encompass both above and below orientations. The device may be oriented in other ways (rotated 90 degrees or otherwise), and the spatial relative descriptors used herein can be interpreted accordingly.

[0031] As used in this article, the term "transdermal" refers to something that is done, completed, or achieved through the skin.

[0032] As used in this article, the term "percutaneous medical procedure" refers to accessing internal organs or tissues through needle punctures of the skin rather than through an open approach that exposes the internal organs or tissues using a scalpel (usually a surgical knife).

[0033] As used herein, the term "non-vascular" when used in conjunction with "percutaneous medical procedure" refers to a medical procedure performed on any part of the subject's body other than the percutaneously accessed vascular system. Examples of percutaneous medical procedures may include biopsy, tissue ablation, cryotherapy, brachytherapy, endovascular surgery, drainage procedures, orthopedic surgery, pain management procedures, vertebroplasty, pedicle / screw placement, guidewire placement, iliac joint fixation, and rehabilitation procedures.

[0034] As used herein, the term "endovascular" when used in conjunction with "percutaneous medical procedure" refers to a medical procedure performed on a blood vessel (or lymphatic system) that is accessed percutaneously. Examples of endovascular percutaneous medical procedures may include aneurysm repair, stent grafting / placement, endovascular prosthesis placement, wire placement, catheter insertion, filter placement, angioplasty, etc.

[0035] As used in this article, the terms "interventional device" or "tracked instrument" refer to medical instruments used in nonvascular percutaneous medical procedures.

[0036] As used herein, the term "tracking system" refers to something used to observe one or more moving objects and to provide a timely and ordered sequence of tracking data (e.g., position data, orientation data, etc.) in a tracking coordinate system for further processing. For example, a tracking system could be an electromagnetic tracking system that observes an interventional device equipped with sensor coils as it moves within a patient's body.

[0037] As used in this article, the term "tracking data" refers to information recorded by a tracking system that relates to the observation of one or more moving objects.

[0038] As used herein, the term "tracking coordinate system" refers to a three-dimensional (3D) Cartesian coordinate system that uses one or more numbers to determine the position of points or other geometric elements specific to a particular tracking system. For example, the tracking coordinate system can be rotated, scaled, etc., relative to a standard 3D Cartesian coordinate system.

[0039] As used herein, the terms "head-mounted device," "head-mounted viewer," or "helmet-mounted display (HMD)" refer to a display device configured to be worn on the head, having one or more display optics (including lenses) in front of one or more eyes. These terms may be used even more generally as the term "augmented reality system," although it should be understood that the term "augmented reality system" is not limited to a display device configured to be worn on the head. In some cases, head-mounted devices may also include non-transitory memory and processing units. An example of a suitable head-mounted device is Microsoft...

[0040] As used herein, the terms “imaging system,” “image acquisition device,” and “image acquisition system” refer to technologies that create a visual representation of the inside of a patient’s body. For example, an imaging system can be a computed tomography (CT) system, a fluorescence examination system, a magnetic resonance imaging (MRI) system, an ultrasound (US) system, etc.

[0041] As used herein, the term "coordinate system" or "augmented reality system coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the position of points or other geometric elements specific to a particular augmented reality system or image acquisition system to which it belongs. For example, a head-mounted viewer coordinate system can be rotated, scaled, etc., relative to a standard 3D Cartesian coordinate system.

[0042] As used herein, the terms “image data” or “image dataset” or “imaging data” refer to information recorded in 3D by an imaging system that relates to the observation of the inside of a patient’s body. For example, “image data” or “image dataset” may include processed two-dimensional or three-dimensional images or models such as tomographic images, for example, data formatted according to the Digital Imaging and Communications in Medicine (DICOM) standard or other relevant imaging standards.

[0043] As used herein, the term "imaging coordinate system" or "image acquisition system coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the position of points or other geometric elements specific to a particular imaging system. For example, the imaging coordinate system can be rotated and scaled relative to a standard 3D Cartesian coordinate system.

[0044] As used herein, the terms “hologram,” “holography,” “holographic projection,” or “holographic representation” refer to a computer-generated image projected onto a lens of a head-mounted viewer. Typically, holograms can be synthesized (in augmented reality (AR)) and are independent of physical reality.

[0045] As used in this article, the term "physical" refers to real things. Physical things are not holographic (or computer-generated).

[0046] As used in this article, the term “two-dimensional” or “2D” refers to something represented in two physical dimensions.

[0047] As used herein, the terms “three-dimensional” or “3D” refer to something represented by three physical dimensions. Elements of “four-dimensional (4D)” (e.g., 3D plus time and / or motion dimensions) will be included in the definition of three-dimensional or 3D.

[0048] As used in this article, the term "integration" can refer to two things that are linked or coordinated. For example, a coil sensor can be integrated with an interventional device.

[0049] As used in this paper, the term "degrees of freedom (DOF)" or "DOF" refers to a number of independent, variable factors. For example, a tracking system may have six degrees of freedom (or 6DOF), one 3D point, and three rotational dimensions.

[0050] As used in this article, the term "real-time" refers to the actual time when a process or event occurs. In other words, a real-time event is completed in real time (within milliseconds, so the result can be used as feedback immediately). For example, a real-time event can be represented within 100 milliseconds of the event occurring.

[0051] As used herein, the terms “subject” and “patient” are used interchangeably and refer to any vertebrate organism.

[0052] As used herein, the term “registration” refers to the steps of converting tracking data and body image data into a common coordinate system and creating a holographic display of images and information related to the patient’s body during surgery, as further described, for example, in U.S. Patent Application Publication No. 2018 / 0303563 by West et al., and U.S. Patent Application Serial No. 17 / 110,991 jointly owned by the applicants of Black and Martin III, the entire disclosures of which are incorporated herein by reference.

[0053] II. Systems for structural heart repair or replacement

[0054] like Figure 1 As shown, it is used for patients undergoing structural heart repair or replacement surgery (200, such as...). Figure 2 The holographic augmented reality visualization and guidance system 100 (shown) includes an augmented reality system 102, a tracked instrument 104, a computer system 106, and a first image acquisition system 108. In some examples, the holographic augmented reality visualization and guidance system 100 may also include a second image acquisition system 110. Each of the augmented reality system 102, the tracked instrument 104, the first image acquisition system 108, and the second image acquisition system 110 may communicate selectively or permanently with the computer system 106, for example, via a computer network 112. Other suitable instruments, tools, devices, subsystems, and other network devices (including wired and wireless communication devices between components of the holographic augmented reality visualization and guidance system 100) for use with the holographic augmented reality visualization and guidance system 100 may also be employed by those skilled in the art as needed.

[0055] refer to Figure 2The tracked instrument 104 is an interventional device that is sensed so that its position and orientation can be determined by a computer system 106. Specifically, the tracked instrument may have an elongated body (e.g., a long flexible tube) and multiple sections 114, 116, 118, and 120 arranged along its length, each section having one of multiple sensors 115, 117, 119, and 121 in sequence. For example, the tracked instrument 104 may have a tip 114, a top 116, a middle section 118, and a bottom 120. The tip sensor 115 may be located at the tip 114 of the tracked instrument 104. The top sensor 117 may be located at the top 116 of the tracked instrument 104. The middle sensor 119 may be located at the middle 118 of the tracked instrument 104. The bottom sensor 121 may be located at the bottom 120 of the tracked instrument 104. Each of the sensors 115, 117, 119, and 121 communicates with or is otherwise detectable by the computer system 106.

[0056] It should be understood that the tracking provided by the tip sensor 115 is particularly advantageous because practitioners can use the tip sensor 115 as a pre-selected reference point for the tracked instrument 104. The pre-selected reference point is configured as a trajectory hologram (e.g., Figure 1 Anchor points (shown and described herein as "142"), such as holographic rays generated by augmented reality system 102, can be used. As further described herein, holographic rays can assist practitioners in aligning and moving the tracked instrument 104 along a preferred path or trajectory. It should be understood that those skilled in the art can also select any number of pre-selected reference points within the scope of this disclosure. In further embodiments, the pre-selected reference points can be adjusted in real time by the practitioner during medical procedures and can alternatively be based on one or more of other sensors 115, 117, 119, 121 as needed.

[0057] In some examples, sensors 115, 117, 119, and 121 may be part of an electromagnetic (EM) tracking system, which may be part of and / or used by computer system 106 to detect the position and orientation of the physically tracked instrument 104. For example, sensors 115, 117, 119, and 121 may include one or more sensor coils. Computer system 106 may detect one or more sensor coils and provide tracking data (e.g., having six degrees of freedom) in response to detection. For example, the tracking data may include real-time 3D position data and real-time 3D orientation data. The tracking system of computer system 106 may also detect coil sensors that are not located on a physical intervention device (e.g., on a reference marker or other imaging target).

[0058] In addition, multiple position sensors 115, 117, 119, and 121 are configured to evaluate various additional information about the tracked instrument 104, such as its angular velocity and acceleration. Non-limiting examples of the multiple position sensors 115, 117, 119, and 121 suitable for determining angular velocity and acceleration include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. Notably, electromagnetic sensors enable more accurate real-time object tracking of small objects, without being limited by line-of-sight.

[0059] Other suitable tracking systems are considered, such as optical tracking systems used in conjunction with augmented reality system 102 and computer system 106. Embodiments in which the tracked instrument 104 can communicate with augmented reality system 102 and computer system 106 via wireless transmission or via wired connection are also considered. It should also be understood that those skilled in the art may use multiple position sensors 115, 117, 119, 121 of different types as needed.

[0060] The tracked instrument 104 may also include an implant or tool configured for insertion into a patient's heart. Non-limiting examples of the tracked instrument 104 and associated implants include needles, catheters, stents, mechanical heart valves, or biological heart valves. In other examples, the implant itself may be sensed, at least temporarily, during the procedure to facilitate its tracking.

[0061] In most specific examples, the tracked instrument 104 is a catheter configured for inserting a cardiac implant (e.g., a replacement valve) into a patient's heart. In other examples, the tracked instrument 104 is a catheter configured for cardiac ablation procedures. However, those skilled in the art can use other suitable interventional devices for the tracked instrument 104 within the scope of this disclosure, depending on the required procedure.

[0062] Refer again Figure 1 The first image acquisition system 108 is configured to acquire a first holographic image dataset 122 from the patient. Specifically, the first image acquisition system 108 may be configured to acquire the first holographic image dataset 122 from the patient preoperatively. In some embodiments, the first image acquisition system 108 is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device. Other suitable types of instruments for the first image acquisition system 108 may also be used as needed.

[0063] Similarly, the second image acquisition system 110 is configured to acquire a second holographic image dataset 124 from the patient. Specifically, the second image acquisition system 110 may be configured to acquire the second holographic image dataset 124 from the patient in a real-time manner, particularly during surgery. In some embodiments, the second image acquisition system 110 is an echocardiogram (ECG) imaging device. Most notably, the second holographic image dataset 124 can be acquired via a predetermined mode, which includes one of transthoracic echocardiography (TTE), transesophageal echocardiography (TEE), and intracardiac echocardiography (ICE). Other suitable types of instruments and modes for the second image acquisition system 110 may also be used as needed.

[0064] Although the use of both the first image acquisition system 108 and the second image acquisition system 110 is shown and described herein, embodiments using only one or the other of the first image acquisition system 108 and the second image acquisition system 110 are considered to be within the scope of this disclosure.

[0065] Continue to refer to Figure 1 The computer system 106 of this disclosure has at least one processor 126 and at least one memory 128 storing tangible non-transitory machine-readable instructions 130.

[0066] One or more processors 126 can perform functions related to the operation of the holographic augmented reality visualization and guidance system 100. One or more processors 126 can be any type of general-purpose or special-purpose processor. In some cases, according to other embodiments, multiple processors 126 may be used. In fact, as a non-limiting example, one or more processors 126 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture.

[0067] Memory 128 may be one or more memories and may be of any type suitable for the local application environment, and may be implemented using any suitable volatile or non-volatile data storage technology such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. For example, memory 128 may consist of any combination of random access memory (RAM), read-only memory (ROM), static storage such as a disk or optical disk, a hard disk drive (HDD), or any other type of non-transitory machine or computer-readable medium. Instructions stored in memory 128 may include program instructions or computer program code that, when executed by one or more processors 126, enable the holographic augmented reality visualization and guidance system 100 to perform the tasks described herein.

[0068] Machine-readable instructions 130 may include modules. These modules may be implemented as one or more of functional logic, hardware logic, electronic circuits, and software modules. As needed, modules may include one or more of augmented reality system modules, image acquisition modules, instrument tracking modules, image dataset registration modules, hologram rendering modules, image registration modules, trajectory hologram rendering modules, and / or other suitable modules.

[0069] Computer system 106 communicates, for example, via network 112 with augmented reality system 102, tracked instrument 104, first image acquisition system 108, and second image acquisition system 110, and is configured by machine-readable instructions 130 to operate according to method 200 as further described herein. Computer system 106 may be provided separately and isolated from augmented reality system 102, or may be provided together with augmented reality system 102 as a single integrated unit as needed.

[0070] It should be understood that, as a non-limiting example, the network 112 of the holographic augmented reality visualization and guidance system 100 may include a wireless access network such as Long Term Evolution (LTE) or 5th Generation Mobile Communication Technology (5G), a local area network (LAN), a wide area network (WAN) such as the Internet, or a wireless local area network (WLAN). It should be understood that this is not intended to be limiting, and the scope of this disclosure includes implementations of one or more computing platforms of the holographic augmented reality visualization and guidance system 100 that can be operatively linked through some other communication coupling. One or more computing platforms may be configured to communicate with the network environment via wireless or wired connections. Furthermore, in one embodiment, one or more computing platforms may be configured to communicate directly with each other via wireless or wired connections. Examples of one or more computing platforms may include, but are not limited to, smartphones, wearable devices, tablets, laptops, desktop computers, Internet of Things (IoT) devices, or other mobile or fixed devices such as standalone servers, web servers, or server arrays.

[0071] In a particular embodiment, computer system 106 may be configured to track tracked instrument 104 using multiple sensors 115, 117, 119, 121 to provide tracked instrument dataset 132. Tracked instrument dataset 132 may be stored on memory 128. Specifically, for example, the location and orientation of tracked instrument 104 in physical space may be stored as tracked instrument dataset 132.

[0072] The computer system 106 may also be configured to register a first holographic image dataset 122 from the first image acquisition system 108, a tracked instrument dataset 132 obtained by the computer system 106, and the patient, as further described herein.

[0073] Continue to refer to Figure 1 The augmented reality system 102 is configured to render a plurality of holograms 134, 136, 138, 140, and 142 according to the method 200 of this disclosure. Specifically, the augmented reality system 102 may be a mixed reality (MR) display, such as MR smart glasses or an MR head-mounted display. Non-limiting examples of the augmented reality system 102 include Magic Leap. or Microsoft It should be understood that other types of MR displays can be used in augmented reality system 102, as long as the MR display is capable of overlaying computer-generated images onto real-world objects. Furthermore, although this document primarily describes augmented reality system 102 as a head-mounted display, it should be understood that other types of displays, which are not head-mounted but are capable of generating holograms 134, 136, 138, 140 and overlaying them onto a real-world view, may also be used as needed.

[0074] It should be understood that, in the absence of computer system 100, augmented reality system 102 may further include additional non-transitory memory and a processing unit (which may include one or more hardware processors) to assist in rendering or generating holograms 134, 136, 138, 140, 142. Augmented reality system 102 may also include a camera for recording one or more images, one or more image generation components for generating / displaying visualizations of holograms 134, 136, 138, 140, 142, and / or other visualization and / or recording elements.

[0075] In a further example, it should be understood that the augmented reality system 102 may also include a plurality of position sensors 144. The plurality of position sensors 144 of the augmented reality system 102 are configured to determine various positional information of the augmented reality system 102, such as approximate position in three-dimensional (3D) space, orientation, angular velocity, and acceleration of the augmented reality system 102. In particular, it should be understood that this enables the holographic image to be accurately displayed in the practitioner's field of vision during operation.

[0076] Non-limiting examples of multiple position sensors 144 include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. It should also be understood that those skilled in the art may use different types and numbers of multiple position sensors 144 of the augmented reality system 102, for example, as needed for the process or circumstances of using the augmented reality system 102.

[0077] like Figure 1As shown, for example, holograms 134, 136, 138, 140, and 142 generated by augmented reality system 102 may include a first hologram 134, a tracked instrument hologram 136, a second hologram 138, an animated hologram 140, and a trajectory hologram 142. The first hologram 134 generated by augmented reality system 102 may be based on a first holographic image dataset 122 from a patient. The tracked instrument hologram 136 generated by augmented reality system 102 may be based on the tracked instrument dataset 132. The second hologram 138 generated by augmented reality system 102 may be based on a second holographic image dataset 124. As further described herein, the animated hologram 140 may be based on the processing of the second holographic image dataset 124 by computer system 106 to provide the animated hologram dataset 148. As further described herein, the trajectory hologram 142 may be based on the trajectory dataset 146, which may be manually or automatically selected and stored in the memory 128 of computer system 106.

[0078] In addition to rendering or generating various holograms 134, 136, 138, 140, and 142, the augmented reality system 102 can be further configured to display multiple operational information or details to the practitioner. For example, the augmented reality system 102 can project multiple operational information onto a real-world object (e.g., a patient). For example, the operational information may include real-time navigation instructions or guidance on a chosen trajectory. It should be understood that the augmented reality system 102 can project multiple operational information onto various real-world objects (e.g., the tracked instrument 104) and various rendered holograms 134, 136, 138, 140, and 142 as needed.

[0079] The expectation is that the generation of this operational information or details will allow practitioners to view the patient and multiple operational information simultaneously within the same field of view. Furthermore, the generation of operational information or details, as well as various holograms 134, 136, 138, 140, and 142, will enable practitioners to plan, calibrate, or pre-orient the tracked instrument 104 during operation.

[0080] like Figure 1 As shown, computer system 106 communicates with augmented reality system 102 and the tracked instrument 104. Computer system 106 is configured to store and generate multiple operational information based on machine-readable instructions 130 encoded into memory 128, either through fully manual intervention by a practitioner or other medical professional or automatically. For example, in augmented reality system 102, multiple operational information can be generated based on the position or orientation of the tracked instrument 104 determined by sensors, such as through algorithms, artificial intelligence (AI) protocols, or other practitioner-input data or thresholds.

[0081] Furthermore, the computer system 106 is configured to enable the practitioner to selectively adjust multiple pieces of operational information in real time. For example, the practitioner may be able to adjust the position or orientation of the trajectory hologram 142. Additionally, the practitioner may be able to determine which of the multiple pieces of operational data is being actively displayed to them. It should be understood that, within the scope of this disclosure, the practitioner may adjust other settings and attributes of the multiple pieces of operational information in real time.

[0082] In particular, it should be understood that the augmented reality system 102 of this disclosure advantageously enables practitioners to perform structural heart repair or replacement procedures 200 on a patient while viewing the patient and the first hologram 134 and optionally the instrument hologram 136. Similarly, it advantageously enables practitioners to use the augmented reality system 102 during structural heart repair or replacement surgery to visualize, guide, and navigate at least one of the tracked instruments 104, as further described herein with respect to the method 200 of this disclosure.

[0083] III. Structural Heart Repair or Replacement Methods

[0084] Figure 3 An example flowchart of a method 200 according to an embodiment of the present disclosure is shown. Method 200 may include a first step 202, namely providing a holographic augmented reality visualization and guidance system 100 as described herein.

[0085] In the second step 204, method 200 may include acquiring a first holographic image dataset 122 from the patient via the first image acquisition system 108. Method 200 may include a third step 206, namely, using a computer system 106 to track the tracked instrument 104 using multiple sensors 115, 117, 119, 121 to provide a tracked instrument dataset 132.

[0086] Then, method 200 may include a fourth step 208, namely registering the first holographic image dataset 122, the tracked instrument dataset 132, and the patient through computer system 106.

[0087] In step 210, method 200 may include rendering a first hologram 134 based on a first holographic image dataset 122 from the patient via augmented reality system 102 for a practitioner to view. Optionally, a tracked instrument hologram 136 may also be rendered in step 210 based on a tracked instrument dataset 132 for a practitioner to view. With the instrument hologram 134 displayed, it should be understood that the practitioner may apply the visualization of the hologram to help locate the real-world tracked instrument 104.

[0088] Then, method 200 may include a sixth step 212, in which a practitioner performs structural heart repair or replacement surgery on the patient while viewing the patient and the first hologram 134 and optionally the instrument hologram 136 using the augmented reality system 102. In the sixth step 212, the practitioner uses the augmented reality system 102 to visualize, guide, and navigate at least one of the tracked instrument 104 during the structural heart repair or replacement surgery.

[0089] It should be understood that, in structural heart repair or replacement surgery, visualization, guidance, and navigation of at least one of the tracked instruments 104 may also advantageously involve further steps in interpreting the patient’s intraoperative movement during the procedure.

[0090] In one example, such as Figure 4 As shown, method 200 may further include a seventh step 214, namely, acquiring a second holographic image dataset 124 from the patient via a second image acquisition system 110. Specifically, the second holographic image dataset 124 may be intraoperative and may be acquired in real time during structural heart repair or replacement surgery. After acquiring the second holographic image dataset 124, the method may further include an eighth step 216, namely, registering the second holographic image dataset 124 and the patient via a computer system 106. After registering the second holographic image dataset 124, the method may include a ninth step 218, namely, rendering a second hologram 138 based on the second holographic image dataset 124 from the patient via an augmented reality system 102.

[0091] Then, method 200 may further include a tenth step 220, namely, automatically or manually selecting a predetermined portion (not shown) of at least one of the first hologram 134 and the second hologram 138 by computer system 106, for example, according to machine-readable instructions 130 stored in memory 128 of computer system 106, to create an animation. As a non-limiting example, computer system 106 may also automatically or manually merge, stitch, or otherwise digitally combine the first hologram 134 and the second hologram 138 to provide a fused or combined hologram, so that a practitioner can utilize computer system 106 to select predetermined portions of the combined hologram for animation. In some embodiments, the predetermined portion is associated with one of the patient's heart and chest. When the predetermined portion is associated with the patient's heart, the animated hologram 140 generated by computer system 106 may depict the rhythmic movement of the heartbeat. When the predetermined portion is associated with the patient's chest, the animated hologram 140 generated by computer system 106 may depict the respiratory circulatory movement of the chest.

[0092] In a specific example, the animation could be based on real-time imaging of the heart (e.g., ultrasound ECG) via a second image acquisition system 110, allowing the animated hologram 140 to rhythmically beat in real time with the patient's heart. In another example, the animation could be based on a cardiac-gated holographic representation, for example, from a gated CT scan measuring the respiratory cycle relative to the start (atrial systole) and end (atrial diastole) of the cardiac cycle. This could allow for a more thorough understanding of the entire range of motion of the patient's heart.

[0093] Further reference Figure 4 The method may further include an eleventh step 222, namely, generating an animated hologram dataset 148 based on a second holographic image dataset 124 acquired in real time by a computer system 106, representing a predetermined portion of at least one of the first hologram 134 and the second hologram 138. After generating the animated hologram dataset 148, the method 200 may further include a twelfth step 224, namely, rendering the animated hologram 140 according to the animated hologram dataset by an augmented reality system 102 for viewing by practitioners during structural heart repair or replacement surgery.

[0094] Advantageously, by enabling practitioners to view the animated hologram 140 in a static or other static form attached to or replacing the first hologram 134 and the second hologram 138, practitioners can have greater confidence in inserting the tracking instrument 104 into the patient and / or deploying related implants compared to conventional surgery without the system 100.

[0095] It should be understood that at least one of the visualization, guidance, and navigation of the tracked instrument 104 during structural heart repair or replacement surgery may advantageously involve further steps that allow the percutaneous entry point and trajectory of the tracked instrument to be predetermined or planned and optimized before and / or during the surgery.

[0096] In another example, such as Figure 5 As shown, method 200 may further include a thirteenth step 226, namely, using computer system 106 to plan a predetermined trajectory for the tracked instrument 104 to be inserted into the patient, in order to provide a predetermined trajectory dataset 146. After the planning is completed, method 200 may further include a fourteenth step 228, namely, rendering a trajectory hologram 142 based on the predetermined trajectory dataset 146 using augmented reality system 102.

[0097] The trajectory hologram 142 of this disclosure may, for example, include holographic rays illustrating a predetermined trajectory of the tracked instrument 104. The holographic rays may be linear or curved, or may have one or more angles, and may depict the optimal path of the tracked instrument 104. The trajectory hologram 142 can also be used to clearly identify the percutaneous entry point of the tracked instrument 104 in the patient and the intravascular landing point within the patient's body, such as a preferred landing area with the patient's cardiac structure for implant deployment.

[0098] It should be understood that the overall size, shape, and orientation of the trajectory hologram 142 generated by the augmented reality system 102 may be based on operational information from the computer system 106, including preoperative and intraoperative data. It should also be understood that the operational information may include additional data from other sensors in the operating area, as well as other holographic projections 134, 136, 138, and 140 generated by the augmented reality system 102.

[0099] Preoperative data may include patient-related information acquired prior to medical procedures, such as using the first holographic image acquisition system 108. Non-limiting examples of preoperative data include still images or records from transesophageal echocardiography, transabdominal echocardiography, transthoracic echocardiography, computed tomography (CT) scans, magnetic resonance imaging (MRI), or X-rays. It should be understood that, as needed, preoperative data may include information from other diagnostic medical procedures.

[0100] Intraoperative data can include patient-related information acquired in real time during medical procedures, such as using a second holographic imaging system 110. For example, the diagnostic procedures listed above can be performed simultaneously with the current medical procedure.

[0101] In a further embodiment, the operational information includes fused preoperative and intraoperative data. The fused preoperative and intraoperative data are combined in such a way that a more concise and approximate image and animation are presented to the practitioner. In some cases, the fusion is performed manually. In other cases, the fusion is performed by computer system 106, for example, using at least one of the multiple algorithms described in machine-readable instructions 130, or via artificial intelligence (AI).

[0102] As described above, in some embodiments, the holographic ray can be anchored to a pre-selected reference point on the tracked instrument 104. In a further example, the practitioner can also adjust the expected trajectory in real time via computer system 106, for example, to address any unforeseen complications that may arise during the procedure.

[0103] It is believed that the trajectory hologram 142 and other holographic projections can minimize the risk of complications associated with transapical approach surgery. For example, the overall size of the cardiac, arterial, or venous incision can be minimized because practitioners can more accurately grasp the expected trajectory of the tracked instrument 104 through the trajectory hologram 142 (e.g., holographic rays).

[0104] Furthermore, it is believed that the trajectory hologram 142 makes it easier for practitioners to find the optimal approach angle for valve implantation or paravalvular leak (PVL) closure. In addition, making it easier for practitioners to find the optimal approach angle helps them avoid critical structures such as lung tissue, coronary arteries, and the left anterior descending artery.

[0105] For example, a holographic display of a real-time intraoperative scan can be overlaid with a holographic display of a preoperative scan. Furthermore, the fused preoperative and intraoperative data may also include holographic fusion of CT scan images and intraoperative fluorescence imaging to model cardiac motion associated with the cardiac circulation. Additionally, the fused preoperative and intraoperative data may include a covering layer for warning practitioners of sensitive areas within the patient's body that should not be touched by the tracked instrument 104. It should be understood that, within the scope of this disclosure, those skilled in the art can use the fused preoperative and intraoperative data for various applications.

[0106] In another embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 can be configured to predict the shape of an implant (e.g., a valve) after the implant is deployed by the tracked instrument 104. For example, the predicted shape of the implant can also be visualized in the form of a hologram further generated by the augmented reality system 102.

[0107] In yet another embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 may be configured to facilitate coaxial deployment with the tracked instrument 104, i.e., to center the valve within the intravascular structure. The augmented reality system 102 may be used to generate “error bars” or colorings (e.g., “green” for acceptable, “red” for unacceptable) to guide practitioners in coaxial deployment during surgery.

[0108] In a further embodiment, the computer system 106 of the holographic augmented reality visualization and guidance system 100 can be used to predict the remodeling of intravascular or cardiac structures over time caused by the expected placement of the implant. In particular, the computer system 106 can anticipate or predict how the heart will be removed over time in a specific placement manner, thereby allowing placement to be planned in a way that minimizes remodeling that may occur over time.

[0109] In other embodiments, the computer system 106 of the holographic augmented reality visualization and guidance system 100 can be used to assist in selecting the size of a prosthesis or implant before the completion of surgery. Using the holographic augmented reality visualization and guidance system 100 to select an appropriate size can minimize the chance of patient-prosthesis mismatch (PPM), which would otherwise occur when the implanted prosthetic valve is too small or too large for the patient.

[0110] Advantageously, the holographic augmented reality visualization and guidance system 100 and method 200 for structural cardiac repair or replacement enable practitioners to simultaneously view operational data and the patient within the same field of view via the augmented reality system 102. Furthermore, holographic ray tracing allows practitioners to easily determine the intended trajectory of the tracked instrument 104.

[0111] It should also be understood that method 200 allows practitioners to customize how much critical operational information is displayed in the augmented reality system 102. Furthermore, practitioners can, for example, use computer system 106 to customize the settings and attributes of the operational information. Method 200 further enables practitioners to insert instruments at any desired angle without requiring additional disposable physical instrument guides.

[0112] It should be further understood that the system 100 and method 200 of this disclosure are particularly suitable for patients who require mechanical aortic valve and mitral valve prostheses, or patients who require endocardial ablation surgery again.

[0113] While certain representative embodiments and details have been shown for the purposes of illustrating the invention, it will be apparent to those skilled in the art that various changes can be made without departing from the scope of this disclosure, which is further described in the appended claims.

Claims

1. A system for performing a structural heart repair or replacement procedure on a patient, comprising: an augmented reality system; a tracked instrument having a plurality of sensors; a first image acquisition system configured to acquire a first holographic image data set from the patient; a second image acquisition system configured for acquiring a second holographic image data set from the patient; and a computer system having a processor and a memory in communication with the augmented reality system, the tracked instrument, and the first image acquisition system and configured by machine readable instructions to: track the tracked instrument using the plurality of sensors to provide a tracked instrument data set; register the first holographic image data set and the patient, and plan a predetermined trajectory of insertion to provide a predetermined trajectory data set; wherein the augmented reality system is configured to: render a first holographic image based on the first holographic image data set from the patient for viewing by a practitioner; render a trajectory holographic image based on the predetermined trajectory data set for viewing by the practitioner; wherein the trajectory holographic image includes a holographic light ray that delineates a holographic extension of a sensor of the tracked instrument based on a location of a tip of the sensor, and that shows an alignment and movement of the tracked instrument along the trajectory holographic image, the holographic extension being one of a curvilinear path and an angled path, wherein the sensor is configured as a preselected reference point of the tracked instrument, the preselected reference point being configured as an anchor point of the trajectory holographic image; and wherein the practitioner is allowed to perform the structural heart repair or replacement procedure on the patient while viewing the patient and the first holographic image using the augmented reality system, and to use the augmented reality system for at least one of visualization, guidance, and navigation of the tracked instrument during the structural heart repair or replacement procedure. the first image acquisition system is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device.

2. The system of claim 1, wherein, the first holographic image data set from the patient is pre-operative.

3. The system of claim 2, wherein, the second holographic image data set is acquired by a predetermined modality including one of a transthoracic echocardiogram (TTE), a transesophageal echocardiogram (TEE), and an intracardiac echocardiogram (ICE).

4. The system of claim 1, wherein, the computer system is further configured to generate an animated holographic data set of a predetermined portion of at least one of a first computer generated image and a second computer generated image based on the second holographic image data set acquired in real time.

5. The system of claim 1, wherein, the augmented reality system is further configured to render an animated computer generated image projected onto a lens of a head mounted device in accordance with the animated holographic data set for viewing by the practitioner during the structural heart repair or replacement procedure.

6. The system of claim 5, wherein, the computer system is further configured to select the predetermined portion of at least one of the first computer generated image and the second computer generated image to animate.

7. The system of claim 6, wherein, the predetermined portion is associated with one of a heart and a chest of the patient.

8. The system of claim 7, wherein, ​ 9. The system of claim 8, wherein, The predetermined portion is associated with a heart of the patient, and the animated hologram depicts beating rhythmic motion of the heart.

10. The system of claim 8, wherein, The predetermined portion is associated with a chest of the patient, and the animated hologram depicts breathing cyclic motion of the chest.

11. The system of claim 1, wherein, The tracked instrument has an elongated body having a tip, a top, a bottom, and a middle.

12. The system of claim 11, wherein, The plurality of sensors includes a tip sensor arranged at the tip of the tracked instrument.

13. The system of claim 12, wherein, The plurality of sensors includes a top sensor arranged at the top, a bottom sensor arranged at the bottom, and a middle sensor arranged at the middle.

14. The system of claim 1, wherein, The augmented reality system is further configured to render a trajectory computer-generated image projected onto a lens of a head-mounted device based on the predetermined trajectory data set.

15. The system of claim 1, wherein, The second image acquisition system is configured to acquire the second holographic image data set from the patient, the second holographic image data set being intraoperative and acquired in real-time during a structural heart repair or replacement procedure.

16. The system of claim 15, wherein, The computer system is configured to register the second holographic image data set with the patient.

17. The system of claim 16, wherein, The augmented reality system is configured to render a second hologram based on the second holographic image data set from the patient.

18. The system of claim 17, wherein, The second image acquisition system is an echocardiogram (ECG) imaging device.

Images