Augmented reality solutions for interrupting, converting and enhancing cardiovascular surgery and / or procedural mapping navigation and procedural diagnostics

Abstract

The augmented reality system includes an augmented reality display system; and a processing circuit system in communication with the augmented reality display system. The processing circuit system is configured to receive data associated with an anatomical feature of a subject from a mapping system; receive an indication of a location of a medical device within the anatomical feature of the subject from a navigation system; and display at least one of the data associated with the anatomical feature of the subject and the indication of the location of the medical device within the anatomical feature of the subject and a virtual organ object via the augmented reality display system, which overlays a real-world environment viewable by a user via the augmented reality display system.

Description

Technical Field

[0001] The present invention relates to methods and systems for providing augmented / mixed reality solutions for enhancing cardiovascular surgery and / or procedure mapping, navigation and diagnosis. Background Art

[0002] Cardiac arrhythmias, such as ventricular tachycardia and atrial fibrillation, present in the heart disrupt the normal rhythm and efficiency of the heart. These arrhythmias can be treated using ablation techniques such as radiofrequency (RF), ultrasound, microwave, pulsed electric field, cryoablation, and other forms of ablation. In the case of ablation techniques involving the delivery of electrical therapy, a catheter can be used to deliver the ablation energy, or direct visualization can be used in conjunction with surgical ablation tools.

[0003] For example, procedures such as pulmonary vein isolation (PVI) are often used to treat atrial fibrillation. This procedure may involve the use of a cryodevice, such as a cryoballoon catheter, which is positioned at the ostium of a pulmonary vein (PV) such that any blood flow exiting the PV into the left atrium (LA) is completely blocked. In some procedures, once in place, the cryodevice may be activated for a sufficient duration to create the desired lesion within the myocardial tissue at the PV-LA junction, such as the PV ostium. If a cryoballoon is used as the therapeutic element of the cryodevice, the cryoballoon is typically inflated using a fluid or gas coolant so that the balloon can create a circumferential lesion around the ostium and / or sinus of the PV for interrupting abnormal electrical signals exiting the PV.

[0004] The success of the procedure depends largely on the quality of the lesion(s) created during the procedure and whether the cryoballoon has completely occluded the PV. For example, a complete circumferential lesion is only created when the cryoballoon has completely occluded the PV. Incomplete occlusion allows blood to flow from the treated PV, through the cryoballoon, and into the left atrium of the heart. This flow of warm blood may prevent the cryoballoon from reaching a temperature low enough to create permanent lesions in the target tissue. The creation of reversible lesions may not be sufficient to achieve electrical isolation, and as a result, atrial fibrillation may re-emerge. Additionally, even if the PV is completely occluded, suboptimal operation of the cryoablation system may result in a cryoballoon temperature that is not low enough or is not applied for a sufficient amount of time to create permanent lesions in the target tissue.

[0005] Current methods of evaluating or monitoring PV obstruction include fluoroscopic imaging of a radiopaque contrast agent injected into the PV from a device. If the PV ostium has not been completely blocked by a device (such as a cryoballoon catheter), some of the contrast agent may flow from the PV into the left atrium. In that case, the device can be repositioned and more contrast agent injected into the PV. Unfortunately, this method exposes the patient to potentially large doses of contrast agent and exposes the patient, physician, and support staff to radiation. Likewise, fluoroscopic imaging techniques used to evaluate PV obstruction require the use of X-rays, which are known to be harmful to the patient, and require the physician to wear heavy lead protective clothing, which limits the physician's physical movement during the procedure and may cause physical and orthopedic stress to the physician over time.

[0006] Alternatively, pressure measurement at the distal end of the obstruction site can be used to assess the obstruction before initiating cryogen injection. Other methods may involve using a temperature sensor to determine the temperature within the cryoballoon, and to associate the measured temperature with the predicted thickness of ice created in the tissue in contact with the cryoballoon. Other methods include, for example, pressure wedges, cooled saline, impedance, Doppler echocardiography, etc. However, data from these types of sensor readings typically involve traditional monitors (e.g., desktop monitors). In addition, multiple operators and monitors are typically required, which may increase the cost and complexity of the workflow of the procedure and make the workflow more inefficient. That is, the prior art requires the physician to remove his or her sight from the patient during manipulation and operation of the catheter to view the procedure diagnostic data and ensure that the anatomical navigation is correct. This may result in a higher risk of damage to adjacent anatomical features, such as the phrenic nerve and vagus plexus, adjacent coronary arteries or esophageal walls. Some physicians may choose to abandon the use of traditional monitors, or not view the information on the monitor as frequently as they prefer, so as to avoid frequently removing their sight from the patient during the procedure.

[0007] In addition to PVI, adjacent anatomical structures can be ablated, such as the posterior wall, left atrial appendage, top of the left atrium, and mitral isthmus line to ensure complete electrical isolation of the left atrium. Confident visualization and navigation to these anatomical structures is critical for confident ablation without damaging adjacent structures. Such ablation can be achieved using any balloon such as a cryoballoon, RF balloon, laser balloon, etc. Additionally, these linear lesions can be created using a linear catheter such as an Arctic Line cryolinear catheter, an RF linear catheter, a laser linear catheter, or point by point RF or cryoablation.

[0008] When performing various types of invasive and minimally invasive medical procedures, a clear line of sight to the target anatomical structure can greatly enhance the safety and efficacy of such procedures. Visualization of the target anatomical structure during the procedure may be limited due to technical limitations of current systems and the caution of physicians in trying to minimize the risk of complications. In addition, the cost of current navigation and mapping systems is very expensive, which limits the accessibility of these systems for procedures and for on-site and remote training (e.g., remote guidance). Procedural diagnostics play an important role in helping physicians implement treatment. Such diagnostic information (e.g., sensor data, mapping, navigation data, biometrics, disease state data, medical device positioning indicators, feedback on treatment effectiveness, etc.) can greatly contribute to the quality of patient outcomes. Unfortunately, this diagnostic information is not easy to access and requires multiple monitors and systems.

[0009] Augmented reality devices blend computer-generated information with a user's view of the physical world (i.e., the real world). Said another way, augmented reality devices augment computer-displayed information to a user's view of the real-world environment so as to enhance the user's situational awareness within the physical world using the computer-displayed information. However, existing augmented reality devices are quite limited in their use and have been primarily limited to personal use rather than for medical imaging and procedures.

[0010] Accordingly, in view of the limitations noted above, it would be desirable to provide systems, devices, and methods for providing augmented / mixed reality solutions for enhancing cardiovascular procedural and / or surgical mapping, navigation, and diagnosis. Summary of the invention

[0011] The present invention advantageously provides methods and systems for providing augmented reality solutions for enhancing cardiovascular surgical mapping, navigation, and procedural diagnosis. Specifically, one aspect of the present invention includes an augmented reality system having: an augmented reality display system; and a processing circuit system that communicates with the augmented reality display system. The processing circuit system is configured to receive data associated with an anatomical feature of a subject from a mapping system; receive an indication of a location of a medical device within the anatomical feature of the subject from a navigation system; and display at least one of the data associated with the anatomical feature of the subject and an indication of the location of the medical device within the anatomical feature of the subject and a virtual organ object via the augmented reality display system, the data associated with the anatomical feature of the subject and at least one of the indication of the location of the medical device within the anatomical feature of the subject and the virtual organ object overlaying a real world environment viewable by a user via the augmented reality display system.

[0012] In this and other aspects, embodiments of the present invention include a virtual organ object based on a three-dimensional model of an anatomical feature of a subject, the anatomical feature of the subject including at least one of at least a portion of a heart and at least a portion of a vascular system. The data includes electrogram data; and at least one of the data associated with the anatomical feature of the subject and an indication of the location of a medical device, the virtual organ object, and the real world environment are simultaneously viewable by a user via an augmented reality display system within the user's field of view. Further, the navigation system is configured to continuously monitor the location of the medical device within the anatomical feature of the subject during the surgical procedure; and the processing circuit system is configured to continuously receive the monitored location from the navigation system during the surgical procedure and display the indication of the monitored location on the virtual organ object. The processing circuit system is further configured to continuously update the indication of the monitored location on the virtual organ object during the surgical procedure. The navigation system is also configured to deliver alternating current to a plurality of external electrodes and direct a plurality of current sources in at least an X plane, a Y plane, and a Z plane for determining the location of the medical device within the anatomical feature of the subject.

[0013] At least one of the navigation system and the mapping system includes at least one medical sensor; and the processing circuit system is configured to receive data associated with the at least one medical sensor and display an indication of the data via the augmented reality display system. The at least one medical sensor is configured to sense an electrical parameter of the heart; and the processing circuit system is configured to display an indication of the electrical parameter via the augmented reality display system. The at least one medical sensor includes at least one of a temperature sensor, a pressure sensor, and an electrode. The processing circuit system is configured to receive an indication of at least one of an obstruction of a pulmonary vein and a parameter associated with an injury from the at least one sensor; and the processing circuit system is configured to display an indication of at least one of an obstruction of a pulmonary vein and a parameter associated with an injury via the augmented reality display system. The indication of at least one of an obstruction of a pulmonary vein and a parameter associated with an injury is based on at least one of an impedance measurement and a temperature measurement from the at least one sensor. Mesh collisions between a virtual mesh of the patient's anatomy and a virtual mesh of an ablation tool can also be used to provide feedback about the quality of the obstruction before, during, and after ablation.

[0014] According to another aspect, a method for augmenting a surgical procedure is provided. The method includes receiving, by an augmented reality device including an augmented reality display system, data associated with an anatomical feature of a subject and an indication of a location of a medical device within the anatomical feature of the subject; and displaying, via the augmented reality display system, at least one of the data and the indication of the location of the medical device and a virtual organ object, the data and at least one of the indication of the location of the medical device and the virtual organ object overlaying a real world environment viewable by a user via the augmented reality display system. The receiving and displaying are performed during a surgical procedure, wherein the user is a surgeon and the real world environment includes at least a portion of a patient's body in an operating room. At least one of the data and the indication of the location, the virtual organ object, and at least a portion of the patient's body are simultaneously viewable by the surgeon via the augmented reality display system during the surgical procedure.

[0015] According to this aspect, in some embodiments, the virtual organ object is based on a three-dimensional model of an anatomical feature of a subject, the anatomical feature of the subject including at least one of at least a portion of a heart and at least a portion of a vascular system. The augmented reality device continuously receives a location of a medical device within the anatomical feature of the subject during a surgical procedure, and continuously updates an indication of the location of the medical device on the virtual organ object during the surgical procedure. The augmented reality device receives data from at least one medical sensor and displays an indication of the data from the at least one medical sensor via an augmented reality display system. An indication of at least one of an obstruction of a pulmonary vein and a parameter associated with the injury is received by the augmented reality device from the at least one sensor; and the indication is displayed via the augmented reality display system.

[0016] According to yet another aspect, a method for augmenting a surgical procedure is provided. The method includes: receiving, by an augmented reality device including an augmented reality display system, an indication of a location of a medical device within an anatomical feature of a subject; displaying, via the augmented reality display system, a virtual organ object and the indication of the location of the medical device within the anatomical feature of the subject, the virtual organ object and the indication of the location of the medical device within the anatomical feature of the subject overlaying a real world environment viewable by a user via the augmented reality display system; and continuously updating the indication of the location of the medical device on the virtual organ object during the surgical procedure.

[0017] According to this aspect, in some embodiments, the virtual organ object is based on a three-dimensional model of an anatomical feature of the subject, the anatomical feature of the subject including at least one of at least a portion of a heart and at least a portion of a vascular system. Electrogram data associated with the anatomical feature of the subject is received by an augmented reality device; and the virtual organ object and the electrogram data are displayed via an augmented reality display system, the virtual organ object and the electrogram data overlaying a real world environment viewable by a user via the augmented reality display system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] A more complete understanding of the present invention, together with attendant advantages and features thereof, will be more readily appreciated by reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A rear perspective view of an exemplary augmented reality device is shown in accordance with the principles of the present disclosure;

[0020] Figure 2 According to the embodiment of the present disclosure, Figure 1 A block diagram view of components of an augmented reality device;

[0021] Figure 3 An exemplary medical system is shown according to the present disclosure;

[0022] Figure 4 According to the present disclosure, Figure 3 Exemplary medical devices within a medical system;

[0023] Figure 5 An exemplary mapping system at least partially disposed on a patient is shown according to an embodiment of the present disclosure;

[0024] Figure 6a-6b According to an embodiment of the present disclosure, Figure 5 Exemplary mapping of a heart of a patient of;

[0025] Figure 7 An exemplary method is shown according to an embodiment of the present disclosure;

[0026] Figure 8 According to an embodiment of the present disclosure, Figure 5 a cross-sectional view of a patient's heart;

[0027] Figure 9a and Figure 9b Partial and complete occlusion of a pulmonary vein are shown, respectively, according to an embodiment of the present disclosure;

[0028] Fig.10 According to an embodiment of the present disclosure, Figure 1 Virtual organ objects presented by augmented reality devices;

[0029] Fig.11 According to an embodiment of the present disclosure, a physician Figure 1 an augmented reality device (not shown) to simultaneously view an augmented reality view of a real-world patient lying on an operating table and a free-floating virtual heart object in the operating room; and

[0030] Fig.12 According to an embodiment of the present disclosure, a top view is shown. Fig.11 A physician's augmented reality view of a real-world patient with a virtual organ object attached to the real-world patient's heart during a surgical procedure. DETAILED DESCRIPTION

[0031] The present disclosure describes systems, devices, and methods for providing augmented reality solutions for enhancing cardiovascular surgical mapping, navigation, and procedural diagnostics. The most commonly performed surgical procedures, such as those used to treat cardiac conditions, present challenges for physicians. For example, for heart failure procedures requiring transvenous access, these challenges include the inability to visualize the branches of venous structures during positioning of the treatment device (e.g., when performing coronary sinus catheterization) and difficulty in real-time assessment of cardiac lead placement. In addition, for heart failure procedures requiring substernal access, these challenges include the need for a complete thoracotomy, rib removal, and / or lung deflation, and the lack of tools suitable for navigating to the lead delivery location. The systems, devices, and methods described herein not only overcome these procedural challenges, but also provide additional benefits, such as meaningfully integrating information from various biometric, therapeutic, navigation, and diagnostic systems into a single user-friendly system, as well as facilitating minimally invasive and noninvasive surgery, device positioning and placement, pre-procedural planning to assess disease state (e.g., annulus eccentricity, degree of calcification, cardiac output, etc.), and other benefits.

[0032] Before describing in detail exemplary embodiments of the present disclosure, it should be noted that the terms "augmented reality" and "mixed reality" are used interchangeably and are intended to indicate a device, method, and / or system in which data, information, virtual images or objects, graphics, etc., or other sensor-generated or computer-generated actionable decision-making information is provided to an observer by a source (such as a display source, for example, a wearable device or a holographic projector, etc.) within a real-world environment that can be simultaneously observed by the observer. In some embodiments of the present disclosure described herein, "mixed reality" may further indicate a device, method, and / or system in which real-time data from one or more information sources is communicated, transmitted, streamed, and / or presented / rendered to an observer within a real-world environment. In yet other embodiments of the present disclosure, "mixed reality" and "augmented reality" may further indicate a device, method, and / or system in which computer-generated information can be presented to an observer in a manner that interacts with or is otherwise bound to at least a portion of the real-world environment.

[0033] Referring now to the drawings, wherein like reference numerals refer to like elements, embodiments of an augmented reality device according to the principles of the present invention are shown in FIG. Figure 1 is shown in and is generally designated as "100." Figure 3 and Figure 5 Embodiments of a medical system 10 and a mapping system 500 in communication with an augmented reality device 100 are shown, respectively.

[0034] The following description will begin with an introduction to the augmented reality device 100, and various exemplary components of the medical system 10 and the mapping system 500, followed by a description of exemplary methods of using and / or implementing the devices and systems 100, 10, 500 according to the principles of the present disclosure.

[0035] Augmented reality devices

[0036] exist Figure 1 1. An exemplary augmented reality device 100 is shown in FIG. 1 as a wearable augmented reality device in the form of glasses 102. The glasses 102 include a pair of lenses 104, 106 and a frame 108, and a head support member 110 extending from the frame 108.

[0037] The head support member 110 is configured to support the augmented reality device 100 on the head of the wearer / user. The exemplary head support member 110 is shown in the form of a headband; however, alternative embodiments may be in the form of, for example, a pair of side arms that are configured to rest on the ears of the wearer like traditional glasses. The head support member 110 or another portion of the augmented reality device 100 may be adapted as an augmented reality device housing for housing augmented reality device electrical and / or optical components, such as, for example, processing circuitry, sensors, cameras, network interfaces, etc., as described below with reference to Figure 2 The block diagram of will be described in more detail. Alternative embodiments may be in the form of contact lenses that wirelessly communicate and display information generated from the subsystems to build a mixed reality environment. Alternative embodiments may use spatial computing to project digital information into the physical world using cameras, mirrors, light refractors, and stages. This will allow information from multiple sources to be displayed and integrated without the user having to wear anything on their head or in their eyes.

[0038] The frame 108 and head support member 110 may be formed from solid rigid polymer, plastic, and / or metal structures, or may be formed as hollow structures of similar materials to allow for wiring throughout the eyeglasses 102 and interconnection of internal electrical components.

[0039] The lenses 104, 106 may be sufficiently transparent to allow a user to see through the lenses 104, 106, such as lenses conventionally provided for non-computing glasses. Said another way, the lenses 104, 106 may be considered see-through lenses 104, 106. In some embodiments, the lenses 104, 106 may be prescription lenses. In some embodiments, the lenses 104, 106 may be slightly tinted. Although, Figure 1 Two lenses 104 , 106 are shown, but it is understood that the augmented reality device 100 may include one lens or more than two lenses 104 , 106 .

[0040] Now mainly see Figure 2 , a brief reference Figure 1 , the augmented reality device 100 may include computing components and optical components housed within or otherwise supported by or included in the head support member 110. The computing components and optical components may be coupled to the frame 108 and / or may be housed within the head support member 110. It is understood that any one or more of the computing components and optical components may be provided on other portions of the glasses 102, or may be remotely located, communicating with the glasses 102 wirelessly or by wire via a communication link.

[0041] Despite Figure 1The exemplary augmented reality device depicted in FIG. 5 is shown as a wearable augmented reality device, but it is contemplated that some embodiments of the present disclosure may also be implemented in non-wearable augmented reality devices or systems.

[0042] Figure 2 An example block diagram of example components of augmented reality device 100 is shown. Example augmented reality device 100 may include display system 200, cardiac augmented reality module 202, camera 204, sensor 206, user input interface 208, processor 210, memory 212, and network interface 214.

[0043] The display system 200 may include a microdisplay that, when worn, may be arranged to be in close proximity to the wearer's eyes. The microdisplay may be arranged between the lenses 104, 106 and the wearer's eyes. The microdisplay may be a transparent or translucent display. The microdisplay may be a matrix display, such as a light emitting diode (LED) display, a liquid crystal display (LCD), and the like. It is understood that any applicable display may be used in the embodiments of the present disclosure, the precursor being that it allows the wearer or user of the glasses to view the image as a virtual object (e.g., a holographic image), which enhances the view of the physical / real world environment that the wearer (or user) can view via the display system 200, for example, by both the real world and the virtual world observable through the lenses 104, 106. The display driver may provide an interface between the processor 210 and the display system 200 for driving the display system 200. The display system 200, in conjunction with a display driver (not shown), may be configured to display a virtual organ object (e.g., 1000) that overlays the user's view of the real world through the lenses 104, 106.

[0044] Various techniques for displaying virtual objects using head mounted augmented reality devices or other non-wearable augmented reality devices are known and can be used with embodiments of the present invention. In one embodiment, display system 200 can communicate with processor 210 and can include projector 201 that projects images on a surface (such as a microdisplay). Processor 210 can process image data, such as, for example, a three-dimensional model of heart 800 (see Figure 8 ), which image data is then fed into projector 201 for projecting an image on a microdisplay or other display surface. Projector 201 may be configured to project the image on an internal display surface of the microdisplay, thereby making the image viewable by the user as an image overlaid on the user's view of the real world environment through lenses 104, 106. Display system 200 may also include a light projection system, and may include a reflective coating that reflects light projected onto the reflective surface directly to the user's retina.

[0045] In another embodiment, the display system 200 can be an optical see-through head mounted display that places an optical combiner directly in front of the wearer's eyes. The optical combiner is partially transparent to allow the user to view the real world through the combiner. The combiner is also partially reflective to allow the user to view virtual objects that bounce off the combiner. In yet another embodiment, the display system 200 can use a waveguide to display virtual objects that overlay the user's real world field of view.

[0046] In yet another embodiment, the augmented reality display system may be formed as a smart contact that can be worn directly on the wearer's eyes. In yet another embodiment, the augmented reality display system may include non-wearable elements, such as a non-wearable projector that uses, for example, spatial computing, reflectors, light refractors, cameras, etc. to display information (e.g., digital information, virtual objects, etc.) within a real-world environment.

[0047] Other types of augmented reality display system technologies may also be used with embodiments of the present invention. In one embodiment, the augmented display device 100 used in embodiments of the present invention is a Microsoft Hololens TM , which uses waveguides to overlay holographic images on the user's real-world view. In yet other embodiments, the augmented reality device 100 used in the embodiments of the present invention may include other known augmented reality devices, preferably head-mounted augmented reality devices. The components that implement the display elements of virtual objects through the augmented reality device 100 may be collectively referred to herein as an "augmented reality display system."

[0048] The cardiac augmented reality module 202 may be configured to facilitate overlaying a virtual cardiac image on a real-world view through the lenses 104, 106 according to known augmented reality techniques for overlaying images on a real-world view. In further embodiments, the cardiac augmented reality module 202 may be responsible for performing the following reference Figure 7 The flowchart depicted in the figure describes some or all of the process steps.

[0049] The cardiac augmented reality module 202 may be implemented as an executable instruction set resident in the augmented reality device 100 and executed by the device 100. In one implementation, the cardiac augmented reality module 202 may be one or more programs stored on a computer or machine-readable medium. In an exemplary implementation, the cardiac augmented reality module 202 is stored in the memory 212. The cardiac augmented reality module 202 may be a standalone software application or form part of a software application that performs additional tasks related to the augmented reality device 100. Likewise, although the cardiac augmented reality module 202 is implemented in software according to an implementation of the present invention; such functionality may also be performed via dedicated hardware or firmware, or via a combination of hardware, firmware, and / or software.

[0050] The camera 204 includes a camera lens and may be configured to capture still images as well as video. In one embodiment, the augmented reality device 100 may include more than one camera 204. The camera lens may face forward (i.e., away from the wearer / user's face when the augmented reality device 100 is in use) to capture still images and video of at least a portion of the real world as perceived by the user wearing the glasses 102. The images captured by the camera 204 may be stored in the memory 212 for use in displaying a virtual organ object (e.g., 1000) overlaid on the real world view perceived by the user through the lenses 104, 106. In other embodiments, the augmented reality device 100 may include more than one camera 204 for, for example, capturing a wide field of view of the real world.

[0051] The sensor 206 may be disposed on and / or within the head support member 110 and may be configured to sense or detect multiple aspects of the real world environment, such as, for example, the orientation of the user's head. The augmented reality device 100 may include more than one sensor 206. In one embodiment, the sensor 206 may be, for example, an accelerometer or a proximity sensor, or other sensors known in the art. The sensor 206 can, for example, detect when the user has taken off the glasses 102 to power off the augmented reality device 100 or to place the augmented reality device 100 in a sleep mode during non-use. In other embodiments, the augmented reality device 100 may include multiple sensors 206 to track, for example, the wearer's gaze and sense hand movements and gestures used by the user input interface 208. Preferably, the augmented reality device 100 includes numerous sensors to facilitate sensing the physical world in which the user is located, and to detect the user itself and more specifically his / her movements, gaze, head orientation, etc. Such sensors 206 may allow the augmented reality device 100 to combine real-world data with virtual image data in order to provide a more immersive and more accurate augmented experience for the physician and / or allow the virtual image to interact with portions of the real world.

[0052] The user input interface 208 is configured to allow a user to provide input to the augmented reality device 100, such as, for example, user selections, commands, etc. In one embodiment, the user input interface includes a finger-operable touchpad disposed on the head support member 110, which is capable of detecting at least one of the position, pressure, and movement of a finger via devices and methods well known in the art (such as, capacitive sensing, resistive sensing, etc.). For example, a user can use his or her finger to tap the touchpad to initiate or disable a software application responsible for generating and displaying a cardiac image as an augmented reality image according to the techniques described herein. In another embodiment, the augmented reality device 100 may include a sensor 206, which is configured to sense a user's gesture as a user input according to methods and devices known in the field of augmented reality devices. Other user input devices may be integrated into the augmented reality device 100, such as a microphone for voice recognition commands, a physical or virtual keyboard, a motion sensor, a gaze tracking camera, a pointing device, and / or other user input methods and devices (not shown). For example, in some embodiments, a microphone may be communicatively coupled to a processor configured to analyze and recognize human utterance that may correspond to voice commands that may be recognized by the processor to automatically command certain medical features (e.g., turning on and off ablation performed by an ablation treatment device).

[0053] The processor 210 may be, for example, a central processing unit (CPU), a controller, a microcontroller, a microprocessor (including a "general purpose" microprocessor or a special purpose microprocessor). The processor 210 executes code stored in the memory 212 and / or a non-volatile storage device to facilitate the operation of the augmented reality device 100. The processor 210 may provide processing capabilities for executing an operating system, running various applications, and providing processing for one or more of the techniques, functions, and / or methods described herein. As used herein, the terms "processor" and "processing circuit system" are intended to broadly include a single processor or multiple processors and other processing circuit systems that provide processing for one or more of the techniques, functions, and / or methods described herein. The augmented reality device 100 preferably includes one or more CPUs and one or more graphics processing units (GPUs) to facilitate processing of large amounts of image data.

[0054] The memory 212 associated with the augmented reality device 100 may be, for example, one or more buffers, registers, flash memory, or random access memory (RAM). The memory 212 may also include a non-volatile storage device. The non-volatile storage device may represent any suitable storage medium, such as a hard drive or a non-volatile memory, such as a flash memory, etc.

[0055] The network interface 214 may include one or more network interface cards (NICs) that provide the augmented reality device 100 with a connection to, for example, a personal area network (PAN) such as, The network interface 214 can facilitate communication over one or more wired and / or wireless network connections. The network interface 214 can facilitate communication between the augmented reality device 100, the medical system 10, the mapping system 500, and other devices and systems (such as a navigation system). For example, preoperative image data (such as a computed tomography (CT) scan of a patient) can be wirelessly transmitted to the augmented reality device 100 via the network interface 214 for use in generating a virtual organ object based on the CT scan.

[0056] Medical system

[0057] Now mainly see Figure 3 , an exemplary system for treating tissue is shown. The system can be considered as a medical system 10, and can be used to treat, ablate, diagnose and / or map one or more target tissue areas. The medical system 10 can generally include a treatment device 12, such as a treatment catheter, having one or more treatment or ablation electrodes 16 for delivering ablative or therapeutic energy and a console 18 including various system controls. The medical system 10 can be adapted for use with radio frequency (RF) energy, phased RF energy and / or pulsed field ablation (PFA) energy (delivered as square waves and / or sine waves), and can be additionally adapted to use other energy modalities for ablation or treatment, such as cryoablation and cryotherapy, ultrasound energy, laser energy, microwave energy, thermal balloon treatment or ablation or other modalities or combinations thereof. A description of an exemplary PFA energy delivery technique is described in U.S. Serial No. 15 / 228,406, the application of which is incorporated herein by reference in its entirety.

[0058] The treatment device 12 may generally include a handle 22, an elongated body 24 having a distal portion 26 and a proximal portion 28, one or more treatment elements, such as a non-expandable electrode 16 (e.g., as may be used on a focal catheter), an expandable or non-expandable electrode array (e.g., an expandable array having one or more carrier arms that carry a plurality of treatment / ablation electrodes 16), or an inflatable balloon 32 that may carry one or more treatment / ablation electrodes 16 (e.g., Figure 3 ). Although the cryoballoon 32 is Figure 31 is shown as having one treatment / ablation electrode belt 16, but it will be understood that any number or configuration of electrodes may be used. Moreover, in some embodiments, the treatment device 12 may be used only for cryoablation or cryotherapy, in which case the cryoballoon 32 may not carry the treatment / ablation electrode 16. Further, the treatment device 12 may have a longitudinal axis 34 and one or more mapping electrodes 36. It will be understood that although in Figure 3 The exemplary medical treatment device 12 depicted in FIG. 1 is shown as a cryoballoon catheter, but other embodiments of the present disclosure may include other types of medical treatment devices. That is, the embodiments of the present disclosure are not limited to any particular medical treatment device, as many different types of medical treatment devices are well known. Said in yet another way, Figure 3 The medical devices depicted in FIG. 5 are shown as exemplary medical devices and are not intended to limit the scope of the present disclosure.

[0059] The treatment device 12 may include one or more lumens. For example, the treatment device 12 may include one or more lumens for electrical wiring, manipulation elements, etc. In addition to using RF energy to treat tissue, the medical system 10 may be used for cryotherapy procedures, in which the tissue is thermally affected by the circulation of a coolant within the treatment element. For example, one or more treatment elements may include a cryoballoon 32, which may optionally have a plurality of treatment or ablation electrodes 16 (e.g., a plurality of electrodes 16) for ablating tissue. Figure 3 ), and the treatment device 12 may include one or more mapping electrodes 36. As a non-limiting example, the mapping electrode 36 may be positioned on the elongated body 24 proximal to the cryoballoon 32. In this case, the treatment device 12 may also include, for example, a fluid injection chamber and a coolant recovery chamber, wherein the fluid injection chamber is in fluid communication with the coolant supply reservoir 38, and the coolant recovery chamber is in fluid communication with the coolant recovery reservoir or collection system. Further, the coolant recovery chamber may be in communication with a vacuum device to facilitate the discharge of fluid (e.g., diffused coolant) from the cooling balloon 32. Therefore, it will be understood that references herein to delivering energy to tissue also include removing heat from tissue through a cryotherapy procedure.

[0060] The medical device 12 may further include one or more sensors for monitoring operating parameters in the medical device 12, including, for example, pressure, temperature, flow rate, volume, etc., in addition to monitoring, recording, or otherwise communicating measurements or conditions of the surrounding environment at the distal portion of the medical device 12. In some embodiments, the medical system 10 may include other medical sensors, such as, for example, magnetic sensors, radio frequency identification devices (RFID), impedance sensors, contact force sensors, etc. The sensor(s) may communicate with a control unit associated with the console 18 for initiating or triggering one or more alarms or coolant delivery modifications during operation of the medical device 12. One or more valves, controllers, etc. may communicate with the sensor(s) to provide controlled dispersion or circulation of fluid through the cavity / fluid path of the medical device 12.

[0061] One or more pressure sensors 39 (see Figure 4 ) may be coupled to a distal portion of the treatment device 12. One or more pressure sensors 39 may be used to facilitate placement of the treatment device 12 within the patient, and may further provide monitoring of engagement between the treatment device 12 and a designated tissue region during a procedure. The one or more pressure sensors 39 may be positioned on an exterior surface of the elongated body 24. The pressure sensors 39 may be adjacent to one another to form a single pressure sensing region, or they may be positioned at discrete locations along the elongated body 24. The one or more pressure sensors 39 may communicate with a control unit associated with the console 18.

[0062] Information from the one or more pressure sensors 39 may be transmitted to the console 18 and / or the augmented reality device 100, and the information may be used to create or modify the virtual organ object 1000 (see Figure 10-12 ) to provide procedural guidance or feedback to the physician. In some embodiments, information from one or more of the sensors described throughout this document (which may be provided on the medical device 12 or otherwise communicate with the augmented reality device 100, such as sensors for temperature, pressure, impedance, force, ECG, etc.) may also be captured, analyzed, and displayed via the augmented reality display 200 according to the principles described in the present disclosure. In one embodiment of the present disclosure, during a procedure, intraoperative pressure sensor data corresponding to one or more pressure sensors on the medical device 12 may be accessible by the user / physician of the augmented reality device 100 to provide the physician with easily accessible positioning data of the medical device 12. Such easily accessible real-time data may help the physician manipulate the medical device 12.

[0063] The console 18 may be in electrical communication with the treatment device 12 and, if used for cryotherapy, in fluid communication with the treatment device 12 and may include one or more fluid reservoirs (e.g., a coolant supply reservoir 38), an energy generator 40, a switch system 41, and a computer 42 having a display 44, and may further include various other displays, screens, user input controls, keyboards, buttons, valves, conduits, connectors, power supplies, processors, and computers for adjusting and monitoring system parameters. As used herein, the term "computer" may refer to any programmed or programmable data processing unit having a processing circuit system including a memory and a processor, the memory communicating with the processor, the memory having one or more instructions or algorithms that, when executed by the processor, configure the processor to perform the calculations, analyses, and techniques discussed herein. For example, the data processing unit and / or control unit may include an augmented reality device 100, a dedicated internal circuit system, a user control device, and the like.

[0064] As a non-limiting example, the medical system 10 may include Generator (Medtronic, Inc.) as the energy generator 40, but The generator may also record data from the device, and thus may also be referred to as a "computer". Further, the energy generator 40 may include one or more displays 46, user input devices, controllers, data storage units, etc. The computer (multiple) 42, the energy generator 40, and / or the console 18 may include one or more processing units that are electrically connected to one or more electrodes 16, 36 and one or more fluid valves (e.g., if the system is configured for cryotherapy). As discussed above, each processing unit 48 may have a processing circuit system that includes a memory and a processor, wherein the processing circuit system is configured to receive data from the treatment device 12, process the data, and communicate the data to the navigation system 50. In addition, although the energy generator 40 and the navigation system 50 are shown as being outside the console 18, all or part of these components may alternatively be positioned within the console 18, within the augmented reality device 100, and / or integrated with other components of the computer 42 and the console 18.

[0065] As a non-limiting example, the navigation system 50 may be a LOCALISA TM system (Medtronic, Inc., Minneapolis, Minnesota) or similar systems. For example, LOCALISA TMThe system is a system for locating a catheter or lead in the heart and displaying the position graphically on a display 52. ​​The navigation system 50 may include an energy generator configured to deliver AC energy from three alternating current (AC) sources to an external electrode patch 54 at three separate frequencies (e.g., 30.27kHz, 30.70kHz, and 31.15kHz) in the region of 30kHz. In some embodiments, such a system may be adapted or reconfigured to deliver AC power to the external electrode patch 54 at a single frequency. The external electrode patch 54 may be part of the navigation system 50, which is used to orient the three current sources orthogonally in the X, Y, and Z planes, but it will be understood that other system components may be used to generate X, Y, and Z plane current sources. In each plane, a voltage continuity is created from one electrode patch 54 in a pair to the other electrode patch 54 in the pair, thereby causing the healing / ablation electrode 16 in the plane to act as a pointer (wiper) of a potentiometer. The voltage measured by the treatment / ablation electrode(s) (16) and / or mapping electrode(s) 36 will indicate the position of the treatment device electrode in the plane. A filter can be used to select one of the three frequencies, and the voltage for each plane can be obtained. Thus, the three-dimensional position of the electrode can be determined. However, in another embodiment of the system disclosed herein, a single frequency can be time-division multiplexed into separate portions of the X-plane, Y-plane, and Z-plane to determine the position of the device.

[0066] In a preferred embodiment, the above reference Figure 3 The display 52 of the navigation system 50, the display 44 of the console 18, the display 46 of the energy generator 40, and various other displays, screens, user input controls, keyboards, buttons, etc. discussed in the overall medical system 10 of FIG. 1 may be omitted, replaced, or otherwise rendered unnecessary to the physician / user by the augmented reality display system 200 of the augmented reality device 100. That is, advantageously, various embodiments of the present disclosure may not require numerous conventional displays / monitors associated with existing medical systems.

[0067] Now mainly see Figure 4, showing an enlarged view of the distal portion of an embodiment of the medical device 12. In one embodiment, the medical device 12 is a cryoballoon catheter. The medical device 12 may include a cryoballoon 32 and one or more distal electrodes 30 and one or more proximal electrodes 31. If the electrodes 30, 31 are not configured to measure temperature, the medical device 12 may further include a reference electrode (not shown) and one or more thermocouples 37. The electrodes 30, 31 may be made of a conductive material suitable for sensing impedance and, optionally, temperature. The electrodes 30, 31 and the thermocouples 37 may be coupled to, adhered to, or disposed around the distal portion of the medical device 12, or integrated with or otherwise positioned around the distal portion of the medical device 12.

[0068] The thermocouple 37 may be positioned a distance from the distal electrode 30. For example, the thermocouple 37 may be positioned approximately 2 mm distal to the distal electrode 30. Temperature monitoring may provide an additional and / or redundant means for assessing the quality of freezing and the spread of freezing in the tissue, which in turn provides an indication of the quality of the injury. As a non-limiting example, the cryoballoon 32 may have a diameter of approximately 23 mm to approximately 28 mm.

[0069] The distal electrode 30 may be positioned proximal to the cryoballoon 32, and the proximal electrode 31 may be positioned proximal to the cryoballoon 32, such as on a distal portion of the elongated body 24. For example, the distal electrode 30 may be adjacent to or may be in abutment with the distal end of the cryoballoon 32. However, the proximal electrode 31 may alternatively be positioned on a sheath or a separate catheter. The proximal electrode 31 may be slightly larger than the distal electrode 30 and may serve as a neutral electrode in a bipolar impedance circuit or reference electrode. The larger size of the proximal electrode 31 may minimize the impedance drop at the electrode 31, making the circuit more sensitive to changes at the distal electrode 30. Because the electrode 31 is proximal to the cryoballoon 32, the electrode 31 may be more sensitive to changes in occlusion because the direct electrical path through the blood pool is eliminated. Figure 4 The placement of electrodes 30, 31 shown in the figure may additionally allow the medical device 12 to be integrated with electrode potential navigation systems, such as NavX, CARTO 3, Rhythmia, and LOCALISA. TM .

[0070] In addition to sensing impedance (and thereby sensing lesion mass and PV occlusion, as discussed herein), the electrodes 30, 31 may also be configured for mapping cardiac tissue relative to adjacent tissue (e.g., recording a cardiac electrogram). In a non-limiting embodiment, additional discrete electrodes (not shown) may be radially arranged in a distal housing coupled to a distal portion of the cryoballoon 32, with each additional discrete electrode protruding from the housing (e.g., dome-shaped) to facilitate local tissue depolarization for tissue mapping. Additionally or alternatively, the additional electrodes may be used for electrical impedance tomography to visualize (i.e., "see") ice formation from cryoablation. Information from the thermocouple 37 and / or the electrodes 30, 31 may be transmitted to the console 18 and / or augmented reality device 100, and the information may be used to create or modify the virtual organ object 1000 (see Figure 10-12 ) to provide the physician with device position information and / or procedural guidance or feedback.

[0071] In some embodiments, the medical system 10 may also include a video camera, which may be disposed, for example, at a portion of the medical device 12. The video camera may be configured to wirelessly transmit a video signal to the augmented reality device 100 for display via the augmented reality display system 200. The video may advantageously provide real-time visual feedback to a user of the augmented reality device 100 during, for example, a medical or surgical procedure.

[0072] Mapping system

[0073] Now mainly see Figure 5 and Figure 6a-6b , an exemplary mapping system 500 that communicates with the medical system 10 and / or augmented reality device 100 will be described. As used herein, the term "mapping" is intended to indicate one or more voltage recordings, such as millivolt-level voltages, which are referred to as intracardiac electrograms, performed by a cardiac electrophysiological mapping and / or recording system (such as, the mapping system 500). In addition, the term "mapping" may also be used herein to indicate the analysis of such intracardiac EGM detections of depolarization wavefronts moving through the myocardium. As described below, the mapping system 500 may also display such wavefronts moving on a three-dimensional (3D) presentation of the surface of the heart.

[0074] An exemplary mapping system 500 may include an array of electrodes 502 disposed on a support surface 508 that can be applied to a body surface of a patient 506, such as, for example, a support surface (e.g., a vest) that can engage a body surface of the patient (specifically, a chest region). ECG information from these electrodes can be superimposed on the surface of a virtual heart model to provide physicians with information about the electrical activity of the heart. This information can be presented before, during, and after ablation to demonstrate the therapeutic efficacy based on changes in the electrical patterns of the heart. In other embodiments, the array of electrodes 502 can be attached directly to the patient's skin. In yet other embodiments, the support surface 508 can be formed as other types of non-wearable support surfaces, but should be conductive. In one embodiment, electrical signals can be provided by the electrodes 502 for monitoring and recording the electrical activity of the patient's heart 800. Thus, non-invasive electrophysiological mapping can be performed on the patient's heart 800. In further embodiments, electrical activity may be recorded and analyzed by a processing system 504, such as a control and analysis system, which may collect data, store the data in memory, analyze the data, and present the EGM data on a display, such as, for example, an augmented reality display system 200. Other known types of mapping systems may be used with embodiments of the present disclosure.

[0075] In one embodiment, the mapping system 500 may further include other sensors disposed on the support surface 508, such as, for example, accelerometers or other position or movement sensors for detecting position and movement and helping to align the patient's heart position, respiration, body position, etc., which may change during the medical procedure.

[0076] It will be understood that although the term "patient's heart" is used throughout this document, other embodiments of the disclosure described herein may include methods and devices associated with different organs or anatomical features (instead of the heart, or in addition to the heart). For example, in some embodiments, data from accelerometers, PN stimulation, esophageal temperature probes, etc. may also be displayed via the augmented reality system of various embodiments of the present disclosure, for example, for safety purposes or for monitoring patients in other ways. In addition, various embodiments of the present disclosure may include methods and devices associated with a subject, and the subject may not be considered a patient, such as, for example, a corpse or an animal, depending on the embodiment for research or teaching purposes. Accordingly, as used herein, the term "subject" is intended to broadly include patients as well as non-patient subjects.

[0077] Methods for providing augmented reality solutions for enhancing cardiovascular procedural and / or surgical mapping, navigation, and diagnosis

[0078] See now Figure 7The flowchart shown in Figure 1-Figure 6b as well as Figure 8-Figure 12 , describes the use of the principles of the present disclosure Figure 1-Figure 2 Augmented reality device 100, Figure 3-Figure 4 The medical system 10 and the navigation system 50 and Figure 4 An exemplary method of the mapping system 500 is provided.

[0079] It should be understood that although the terms "navigation system" and "mapping system" are referred to separately herein, some embodiments of the present disclosure may be viewed as a single system having multiple components or subsystems that provide navigation and mapping functions (as described herein). Accordingly, as used herein, the term "navigation system" is intended to broadly encompass a navigation system separate from a mapping system, as well as components or subsystems within an overall system that performs the navigation techniques and functions described herein (e.g., the navigation techniques described with reference to the exemplary navigation system 50). Similarly, the term "mapping system" is intended to broadly encompass a mapping system separate from a navigation system, as well as components or subsystems within an overall system that performs the mapping techniques and functions described herein (e.g., the mapping techniques described with reference to the exemplary mapping system 300).

[0080] Furthermore, for purposes of clarity, the present invention will be described in the context of a cardiac ablation procedure for treating atrial fibrillation. Figure 7 however, it should be understood that other embodiments of the present disclosure may be used to enhance other types of medical procedures, including but not limited to the installation of: cardiac monitors / pacemakers, mitral valve replacement structures, cardiac stents, left atrial appendage closure devices, and other cardiac devices used in both invasive and minimally invasive procedures.

[0081] although Figure 7 A specific order of executing functional logic blocks is shown, but the order of executing the blocks may be changed relative to the order shown. Similarly, two or more blocks shown in succession may be executed simultaneously or partially simultaneously. For simplicity, some blocks may also be omitted. Moreover, some blocks are only exemplary steps in an exemplary implementation, but are not required to comply with the present invention.

[0082] The method may begin at step 700 and immediately proceed to step 702, where anatomical and / or electrical maps are generated for use in assisting a physician 1002 (see Fig.10 ) for preoperative planning and for intraoperative guidance (e.g., visualization during surgical procedures and treatment evaluation).

[0083] In one embodiment, processing system 504 of mapping system 500 may receive and analyze electrogram data and may present an electrical map corresponding to the EGM. As used herein, the term "electrogram data" is intended to broadly encompass data from an electrogram as well as data obtained from analysis of an electrogram or otherwise derived from an electrogram. In another embodiment, processing system 504 (or another computing device in communication with processing system 504, such as, for example, computing device 100 or 300) may create a 3D model of the patient's heart 800 and present the EGM data as moving on a cardiac surface of the 3D model of heart 800, e.g. Figure 6a-6b as shown in . Figure 6a and Figure 6b Exemplary presentations 600a, 600b each show EGM data moving over portions of the cardiac surface of a patient's heart 800.

[0084] In one embodiment, the 3D rendering 600a or 600b may be based on preoperative image data of the patient's heart 800, such as, for example, image data associated with magnetic resonance imaging (MRI), X-ray, ultrasound, fluoroscopy, electrocardiogram, and / or computed tomography (CT) scans of the patient's heart 800. It should be noted that such image data is typically acquired prior to a surgical procedure in order to determine whether the surgical procedure is necessary. Techniques and devices for providing preoperative image data are well known in the art and are therefore not described in detail herein.

[0085] In one embodiment, the augmented reality device 100, or more specifically one or more processors 210, may create a 3D model based on the received preoperative image data. In another embodiment, a separate computing device may create a 3D model, and the 3D model may be communicated to the augmented reality device 100 or otherwise provided to the augmented reality device 100. In one embodiment, the 3D model may be created by, for example, a software application that can apply various 3D modeling techniques to preoperative image data. The 3D modeling techniques may include, for example, segmenting the preoperative image data to identify (multiple) regions of interest, followed by mesh refinement of the (multiple) regions of interest, and further 3D refinement techniques, such as repairing the mesh, correcting errors, smoothing the surface, etc. Applications and techniques for converting medical imaging data to 3D models are known in the medical industry for use, for example, in 3D printing replacement structures or model anatomical parts. These and other known applications and techniques for creating 3D models may be used in embodiments of the present invention to create 3D models used by the augmented reality device 100.

[0086] Alternatively, the 3D rendering 600a or 600b may be based on a 3D model of a general heart to reduce the high costs associated with existing imaging techniques. In one embodiment of the present disclosure, the 3D model of the patient's heart 800 may be rendered as a 3D virtual organ object 1000 (see Fig.10 ) and may include associated vascular systems. The 3D virtual organ object 1000 may be augmented and displayed via the augmented reality display system 200 of the augmented reality device 100 instead of a traditional monitor. In yet further embodiments, the 3D virtual organ object 1000 may use EGM data showing electrical properties (e.g., see Figure 6a-6b ) is further enhanced, such as a depolarized wavefront moving on the surface of the 3D virtual organ object 1000.

[0087] In a preferred embodiment, the 3D model may be a 3D model of the patient's heart 800 and should include a 3D model of the patient's region(s) of interest, such as at least one target treatment anatomy of the patient. The particular region of interest may be a region of the heart 800 that is relevant to the surgical procedure to be performed on the patient's heart 800. The target treatment anatomy may be considered to be one or more anatomical features of the patient intended to be treated, such as, for example, an obstruction to the right lower pulmonary vein 804, the right upper pulmonary vein 806, the left lower pulmonary vein 808, and / or the left upper pulmonary vein 810 (see Figure 8 In one embodiment, the 3D model may be a 3D model of the patient's vascular system, or may include at least a portion of the patient's vascular system.

[0088] In one embodiment, the 3D model includes a cross-sectional view of the heart 800 showing the region(s) of interest, such as in Figure 8 Alternatively, the 3D model may be a surface model of the surface of the patient's heart 800, such as Figure 6a and Figure 6b 3D renderings 600a, 600b depicted in . In one embodiment, a user may input (into the augmented reality device 100 or into another computing system in communication with the augmented reality device 100) a selected view (cross-sectional view or surface view) and / or surgical procedure prior to creating the 3D model. Accordingly, a 3D model may be created to include a selected view or multiple views that may be relevant to the selected surgical procedure. In another embodiment of the system dedicated to a particular type of surgical procedure, the selected view(s) may be a default view. In a further embodiment, the 3D model may include multiple cross-sectional views of the heart 800 from which the surgeon / physician 1002 may select during preoperative planning and during the procedure itself to view in real time via the augmented reality device 100.

[0089] In some embodiments, a neural network algorithm using artificial intelligence may be configured to accept multiple different modalities of scans, such as MRI, CT, ultrasound, echo, etc., and may automatically segment and reconstruct the anatomical structures depicted in such scans to render virtual objects in 3D as a single 3D image of the anatomical structure using data from each of the multiple different modalities. The 3D reconstruction may be automatically unfolded and registered to the patient in mixed reality space. In some embodiments, the system may be configured to use automatic visual recognition techniques to determine blood volume and identify rigid structures for defining anatomical boundaries and landmarks of the 3D image.

[0090] Identify at least one treatment location

[0091] In step 704, at least one treatment location is identified. After the 3D virtual organ object 1000 and / or electrogram data has been rendered (preferably rendered on the 3D virtual organ object via the augmented reality display system 200 (as discussed above)), at least one treatment location may be identified on the virtual organ object 1000. In one embodiment, the treatment location may be the pulmonary veins 804, 806, 808, 810 (see Figure 8 ) which treatment locations may be intended to receive treatment in the form of ablation, for example. Other treatment locations may be identified in other embodiments, depending on the procedure, such as, for example, lead placement locations, left atrial appendage, and mitral valve. By being able to visualize the patient's heart 800 in a 3D environment that includes associated data (such as, for example, the electrical activity of the patient's heart 800), the physician 1002 is able to better identify treatment locations for the procedure. Such identification may be performed in the preoperative planning phase and / or may also be performed during the procedure. In various embodiments, the presentation of virtual organ objects 1000 and / or electrogram data during the procedure may provide the physician 1002 with a 3D visual reference within his / her field of view as an immediately accessible visual guide via the augmented reality display system 200. Advantageously, in such embodiments, the physician 1002 may not need to rely on existing conventional monitors that require the physician 1002 to frequently look away from the patient 506 in the operating room 1100 (such as Figure 10 to Figure 12 ) away.

[0092] A virtual organ object 1000 may be displayed by the augmented reality device 100 overlaid with a real world environment such as an operating room 1100 as viewed by a user 1002 through the lenses 104, 106 of the augmented reality device 100. Said another way, the real world environment 1100 may be viewed simultaneously by the user 1002 of the augmented reality device 100 through the augmented reality display system 200. For example, Figure 10-11A virtual organ object 1000 is shown floating in free space, and Fig.12 A virtual organ object 1000 is shown attached to and overlaying a patient's heart 800 .

[0093] In some embodiments, the virtual organ object 1000 may overlay a portion of the user's real-world field of view (e.g., see Fig.12 , where the patient's chest is covered by the virtual organ object 1000). In other embodiments, the virtual organ object 1000 may be semi-transparent, thereby allowing the portion of the user's real world view that would be covered to still be viewable by the user 1002 under the virtual organ object 1000. As used herein, the term "covering" is intended to broadly encompass both embodiments: where portions of the real world are covered, and where such portions can still be viewable by the user under the semi-transparent virtual object.

[0094] Furthermore, the terms "real-world environment" and "real-world objects" are intended to refer to the physical world; that is, a physical environment and the physical objects therein. An example of a real-world environment is an operating room 1100. Of course, as the user 1002 moves around, the real-world environment viewable through the lenses 104, 106 will also change. That is, as the user 1002 moves around, real-world objects may move into or out of the user's 1002 field of view.

[0095] Accordingly, the virtual organ object 1000 may be configured to move with the movement of the user 1002, thereby keeping the virtual organ object 1000 continuously within the field of view of the user 1002 as the user 1002 moves around his / her environment. Alternatively, in some embodiments, the augmented reality device 100 may be configured to attach (also referred to as "register" in the AR field) the virtual organ object 1000 to a physical object, such as a patient's anatomical structure (e.g., heart or chest), such as Fig.12 When the virtual organ object 1000 is attached / registered to the patient's anatomical structure 1200, the movement of the user 1102 away from the patient's anatomical structure 1200 (e.g., the physician 1002 leaves the operating room 1100) causes the virtual organ object 1000 to move away from the field of view of the user 1002. That is, in order to provide the look and feel of a virtual object interacting with a real-world object, the augmented reality device 100 can be configured to attach the virtual object to the real-world object. This can enhance the immersive experience of the physician 1002 with the virtual organ object 1000.

[0096] In some embodiments, the augmented reality device 100 may be configured to allow the user 1002 to provide user input selecting between the attachment mode and the non-attachment mode. In further embodiments, the augmented reality device 100 may be configured to allow the user 1002 to provide user input selecting a real-world object to attach the virtual organ object 1000. In still other embodiments, the augmented reality device 100 may be configured to provide these features as default non-selectable features.

[0097] As used herein, the terms "surgeon" and "physician" are used interchangeably and are intended to refer to a medical practitioner. In some embodiments, one or more of the medical practitioners observing or participating in a medical procedure according to an embodiment of the present disclosure may be an avatar of a real-world medical practitioner viewable via the augmented reality display system 200. Further, the terms "operating room" and "medical procedure room" may be used interchangeably and are intended to refer to an area or room in which a medical / surgical procedure (e.g., PV occlusion, ablation, LAA closure, etc.) is performed or is to be performed, as the case may be.

[0098] Various known techniques for registering virtual objects may be used in conjunction with embodiments of the present invention. For example, in one embodiment, an outward-facing video camera 204 disposed on or embedded within the augmented reality device 100 may capture a video of a view of a user's real-world environment. The video data may be analyzed by the processor 210 to identify real-world objects in the user's field of view. Accordingly, the virtual organ object 1000 may be presented based on the position of the identified real-world object (e.g., the patient's anatomy 1200). The movement of the real-world object may also be tracked by the camera 204 and the processor 210, thereby allowing the movement of the virtual organ object 1000 to be mapped to the movement of the real-world object. Embodiments of the present disclosure may use existing augmented reality devices, such as, for example, Microsoft's HoloLens TM Aspects of the present disclosure may include software code (or other computer executable instructions, e.g., the cardiac augmented reality module 202 ) configured to use the augmented reality device 100 to identify a real-world patient, overlay a virtual organ object 1000 created based on preoperative image data of the patient, and / or register the virtual organ object 1000 to the patient's anatomy 1200 .

[0099] In addition, data from sensors of the medical system 10, navigation system 50, and / or mapping system 500 may be displayed by the augmented reality device 100 within the field of view of the physician 1002 and overlaid with the real world environment. During the procedure, data such as temperature readings, pressures, electrical properties, etc. may be displayed in real time by the augmented reality device 100 via the augmented reality display system 200 to assist the physician in providing the physician with an ongoing assessment of the efficacy of the procedure, thereby enabling the physician to adjust his / her approach in real time if necessary.

[0100] In one embodiment, the sensor may be considered an intracardiac sensor (i.e., within the patient's heart), such as, for example, the treatment / ablation electrode 16. In further embodiments, the sensor may be configured to sense electrical parameters of the heart 800, such as, for example, impedance measurements or voltage measurements (e.g., electrograms) associated with the patient's heartbeat. The sensor data may be displayed in real time to the physician 1002 via the augmented reality display system 200. For example, during a surgical procedure, the electrical mapping data 1102 may be displayed and overlaid on the real world environment 1100, such as Fig.11 and / or Figure 6a-6b This can provide physician 1002 with real-time data on the electrical activity of the patient's heart 800 within physician 1002's operating field of view.

[0101] In other embodiments, the medical sensor(s) may include a temperature sensor (e.g., a thermocouple 37) or one or more pressure sensors 39. Such sensor data may allow the physician 1002 to make an informed assessment of the position of a treatment device and / or the quality of the treatment, such as, for example, the treatment device 12. For example, cryoablation requires extremely low temperatures in order to freeze tissue. Temperature data from the thermocouple 37 indicating a temperature value below a predetermined temperature value expected for cryoablation may alert the physician 1002 that a lesion may not be properly formed, or that the treatment device 12 may not be properly positioned to completely block the PV, as will be explained in more detail below. In embodiments where an indication of the sensor data is displayed via a display system 200 (which may also be referred to as an augmented reality display system 200), the physician 200 may not need to look away from the patient to receive the warning. In another embodiment, the sensor may be a sensor external to the patient, such as, for example, an electrode patch 54.

[0102] Navigate to and heal at least one healing location

[0103] In step 706, the treatment device 12 is navigated to the treatment location (e.g., PV 804, 806, 808, and / or 810) prior to delivering or initiating treatment during the surgical procedure. In an exemplary embodiment, the treatment element, such as the cryoballoon 32, is navigated toward the right inferior pulmonary vein 804, as shown in FIG. Figure 8 and Figure 9a-9b for use in a pulmonary vein ablation procedure (which may also be referred to as a pulmonary vein isolation (PVI) procedure). It will be appreciated that Figure 9a-9b The cryoballoon 32 shown in is exemplary only, and cryoballoons of other shapes and configurations may be used in conjunction with embodiments of the present invention, such as, for example, cryoballoons with adjustable diameters or non-spherical cryoballoons. Likewise, as used herein, the terms "PV tissue" or "pulmonary vein tissue" may include the PV ostium, PV sinus, LA wall tissue, and / or tissue at the junction between the LA and PV, and are not limited to tissue within the PV. In fact, ablation of tissue within the PV may be undesirable. The inflated balloon 32 may be navigated to the pulmonary vein (PV) ostium for occlusion of the PV, or blocking blood flow from the PV from entering the left atrium (LA) 812 of the heart. PV occlusion not only serves to position the balloon 32 to create a circumferential lesion around the PV ostium, but also prevents warm blood from flowing through the portions of the cryoballoon 32 that are in contact with the target tissue, thereby enhancing the ability of the cryoballoon 32 to reach a sufficiently low temperature to create a permanent and circumferential cryoablative lesion on or in the target tissue. If the PV is not completely occluded (see Figure 9a ), the blood flow through the cryoballoon 32 may have the effect of raising the temperature of the cryoballoon 32, which may result in reversible damage to or in the target tissue. The blocked blood in the PV may be referred to as "stagnant" blood, while the blood in the LA may be referred to as "flowing" blood, because blood may still enter the LA from the other three PVs that are not blocked by the medical device 12. The medical system 10 allows the physician 1002 to assess the blockage in real time and without the use of fluorescent contrast dyes.

[0104] In step 708, the treatment site, such as PV 804, can be treated by, for example, ablation of PV tissue. Electrodes 30 and / or 31 can make continuous impedance and temperature measurements during device placement and subsequent treatment by cryoablation. Impedance can increase as at least a portion of the cryoballoon 32 is inserted into the PV, which can indicate complete occlusion (see Figure 9b ) or partially blocked (see Figure 9a). The magnitude of the impedance increase can be used to determine whether there is a complete or partial blockage, and therefore can be used to determine whether permanent damage is being formed. For example, a larger magnitude can indicate a complete blockage, while a smaller magnitude can indicate a partial blockage. A complete blockage can indicate the formation of permanent damage caused by an ablation procedure. In some embodiments, tactile feedback can be provided by, for example, an augmented reality device or a medical tool to provide tactile feedback to the physician indicating various medical aspects, such as, for example, indicating a complete blockage or other useful information. Tactile feedback can provide an alternative to visual feedback for certain aspects of a medical procedure.

[0105] Evaluation and Treatment

[0106] In step 710, an evaluation of treatment may be performed during and / or after treatment based on measurements recorded by electrodes 30 and / or 31, thermocouples 37, pressure sensor 39, and / or other sensors. For example, if the impedance and / or temperature measurements indicate that the PV is not permanently ablated and / or is less than completely blocked, the treatment device 12 may be repositioned until the evaluation of the impedance and / or temperature measurements indicates complete PV blockage. For example, one or more processors (such as processor 210 of augmented reality device 100) may be programmed to receive and process sensor data from one or more electrodes and / or thermocouples, and to generate a warning or visual indication to the user indicating that the treatment device 12 should be repositioned to achieve complete PA blockage or that the treatment device 12 has been optimally positioned. Advantageously, such indications and / or warnings may be displayed via augmented reality display system 200, or otherwise output by augmented reality device 100. This may provide physician 1002 with real-time feedback of the navigation of medical device 12 within patient 506 during the procedure and within the physician's 1002 operating field of view. Accordingly, the physician may not need to look away from patient 506 during the procedure in order to view navigation and other procedural diagnostic data.

[0107] In one embodiment, the navigation system 50 (e.g., the LOCALISA™ system) is configured to continuously monitor the position / location of the medical device 12 within the patient's heart 800 during the procedure. The processor(s) 210 of the augmented reality device 100 may be configured to continuously receive the monitored position from the navigation system 50. In a further embodiment, the processor(s) 210 of the augmented reality device 100 may be further configured to display a visual indication 1104 of the monitored position of the medical device 12 on the virtual organ object 1000, for example, Fig.11. The visual indication 1104 can be any form of visual computer displayable graphical indication, such as, for example, a flashing "X", an arrow, a circle or other shape, etc. The visual indication 1104 of the monitored position of the medical device 12 is preferably continuously updated in real time during the procedure to provide a real-time navigation tool for the physician 1002. As used herein, the terms "continuously" and "continuous" are intended to broadly encompass periodic updates at time intervals that are close enough to be at least substantially continuous from a human perspective. As is known in the computing arts, any data transferred between computing devices must occur at some time interval (no matter how short) corresponding to the internal clock speed of the processor.

[0108] In some embodiments of the present disclosure, the display of information from, for example, a cryo-console (e.g., console 18) may be replicated or presented exclusively in an augmented reality space by, for example, an augmented reality device 100 via a display system 200. In further embodiments, temperature, pressure, ECG, impedance, radio signals, or any other information communicated to the cryo-console from a therapeutic device (e.g., medical device 12) may be communicated in an augmented reality environment via, for example, an augmented reality device 100. Such information may be collected or received and recorded or stored from a cryo-console, a navigation system, an EP recording system, an electrical mapping system, or other auxiliary subsystems, and may be aggregated and automatically analyzed using known transfer functions, or may be fed into an autonomous learning computer system configured to derive its own conclusions about trends, patterns, and relationships between data collected over time and, in some embodiments, data collected from numerous programs. In a preferred embodiment, program data from multiple systems and subsystems (including those discussed herein) may be used by an autonomous learning computer system for informing and displaying useful information (via, for example, an augmented reality device 100) within a mixed reality environment. In some embodiments, such information from the multiple systems and subsystems may be automatically uploaded to a cloud-based server, where the information may be automatically processed and analyzed using one or more artificial intelligence (AI) algorithms. In some embodiments, the AI ​​algorithms may be used by a processing device in communication with a cloud-based server, such as, for example, a self-learning computer system, to aggregate and analyze information from multiple programs at one or more remote locations to identify trends, patterns, relationships, etc., and formulate recommendations or conclusions that may be displayed via the augmented reality display system 200 for use by the user.

[0109] The process may end at step 712 .

[0110] It will be appreciated by those skilled in the art that the present invention is not limited to the contents specifically shown and described herein above. In addition, unless otherwise mentioned above, it should be noted that all drawings are not to scale. Various modifications and variations are possible under the inspiration of the above teachings without departing from the scope and spirit of the present invention, which is limited only by the appended claims.

[0111] In the above description, it should be understood that the terms used herein are only for the purpose of describing a specific embodiment, and are not intended to be limited. As used herein, the term "a / an" is defined as one or more than one. As used herein, the term "multiple" is defined as two or more than two. As used herein, the term "another" is defined as at least the second or more. As used herein, the term "includes" and / or "has" is defined as including (i.e., open semantics). As used herein, the term "coupled" is defined as connected, although not necessarily directly connected or mechanically connected. The term "provided" is defined herein with its broad meaning, for example, in a one-time or a time period, in whole or in multiple parts, it is made to realize physical existence / become physical existence, make it available and / or supply to someone or something. Further, unless otherwise specified, terms such as "first", "second", "third", etc. are only used for descriptive purposes, and should not be interpreted as indicating or implying relative importance.

[0112] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly defined and limited, terms such as "installed", "coupled", "connected" should be interpreted broadly, for example, it may be fixedly connected, or may be detachably connected, or integrally connected; it may be mechanically connected, or may be electrically connected; it may be directly connected. Or it may be indirectly connected via an intermediate medium. As used herein, the term "about" or "approximately" applies to all numerical values, whether or not explicitly indicated. These terms generally refer to a range of values ​​that a person skilled in the art would consider to be equivalent to the value described (i.e., having the same function or result). In many instances, these terms may include numbers rounded to the nearest significant figure. As used herein, the terms "program", "software application", etc. are defined as sequences of instructions designed to be executed on a computer system. "Program", "computer program" or "software application" may include subroutines, functions, programs, object methods, object implementations, executable applications, applets, small service programs, source code, object code, shared libraries / dynamically loaded libraries and / or other sequences of instructions designed to be executed on a computer system. Those skilled in the art can understand the specific meanings of the above terms in the embodiments of the present invention according to specific circumstances.

Claims

1. An augmented reality system, comprising: Augmented reality display system; as well as a processing circuit system in communication with the augmented reality display system, the processing circuit system being configured to: receiving electrogram data associated with an anatomical feature of the subject from a mapping system; receiving from a navigation system an indication of a location of a medical device within an anatomical feature of the subject; and displaying, via the augmented reality display system, the electrogram data associated with an anatomical feature of the subject, the indication of the location of the medical device within the anatomical feature of the subject, and a virtual organ object, the virtual organ object, the data associated with the anatomical feature of the subject, and the indication of the location of the medical device within the anatomical feature of the subject overlying a real world environment viewable by a user of the augmented reality display system, the processing circuitry being further configured to: receiving an indication of an obstruction of a pulmonary vein from at least one sensor; and displaying, via the augmented reality display system, an indication of an obstruction of the pulmonary vein; and Wherein the indication of obstruction of the pulmonary vein is based on temperature measurements from the at least one sensor, and mesh collisions between a virtual mesh of the subject's anatomical features and a virtual mesh of the therapeutic device are used to provide feedback regarding the quality of the obstruction.

2. The system according to claim 1, characterized in that The virtual organ object is based on a three-dimensional model of an anatomical feature of the subject, the anatomical feature of the subject comprising at least one from the group consisting of at least a portion of a heart and at least a portion of a vasculature.

3. The system according to claim 1 or 2, characterized in that: The virtual organ object, the electrogram data associated with the anatomical features of the subject and the indication of the position of the medical device, and the real world environment are simultaneously viewable by the user within the user's field of view via the augmented reality display system.

4. The system according to any one of claims 1-2, characterized in that: The navigation system is configured to continuously monitor the position of the medical device within an anatomical feature of the subject during a surgical procedure; and The processing circuitry is configured to continuously receive monitored positions from the navigation system during the surgical procedure and to display an indication of the monitored positions on the virtual organ object.

5. The system according to claim 4, characterized in that: The processing circuitry is further configured to continuously update the indication of the monitored position on the virtual organ object during the surgical procedure.

6. The system according to any one of claims 1-2, characterized in that: The navigation system is configured to deliver alternating current to a plurality of external electrodes and to direct a plurality of current sources in at least an X-plane, a Y-plane, and a Z-plane for determining the position of the medical device within an anatomical feature of the subject.

7. The system according to any one of claims 1-2, characterized in that: At least one from the group consisting of the navigation system and the mapping system includes at least one medical sensor; and The processing circuitry is configured to receive data associated with the at least one medical sensor and to display an indication of the data via the augmented reality display system.

8. The system according to claim 7, characterized in that: The at least one medical sensor is configured to sense an electrical parameter of the heart; and The processing circuitry is configured to display an indication of the electrical parameter via the augmented reality display system.

9. The system according to claim 7, characterized in that: The at least one medical sensor includes at least one of a temperature sensor, a pressure sensor, and an electrode.

10. The system according to any one of claims 1-2, characterized in that: The processing circuitry is further configured to receive an indication of a parameter associated with an injury from the at least one sensor; and The processing circuitry is also configured to display the indication of a parameter associated with the injury via the augmented reality display system.

11. The system according to claim 10, characterized in that: The indication of a parameter associated with the injury is based on at least one of an impedance measurement and a temperature measurement from the at least one sensor.

Images