Three-dimensional (3-D) mapping system for augmented reality (AR) environments

Abstract

A three-dimensional (3-D) mapping system for an augmented reality (AR) environment receives, over a period of time, a series of data packets containing coordinate data identifying coordinates of one or more portions of a 3-D model of an organ. The one or more portions collectively represent a portion of the 3-D model of the organ, rather than the entire 3-D model. The AR headset determines, for each respective portion of the 3-D model, a location at which an enhanced representation of the respective portion of the 3-D model is placed in the image of the AR environment. For each respective portion of the 3-D model, the AR headset superimposes an enhanced representation of the respective portion at a location that has been determined in the image of the AR environment. The AR head-mounted device presents an enhanced representation of the respective portion to the user for use in the medical procedure without presenting an enhanced representation of the entire 3-D model.

Description

[0001] Related Applications

[0002] This application claims priority to and the benefit of U.S. Provisional Application No. 63 / 512,460, filed on July 7, 2023, which is incorporated herein by reference in its entirety. Background Art

[0003] Augmented reality (AR) refers to a real-time view of a real-world physical environment modified by a computing device to enhance a person's perception of reality. In AR, elements of the real-world environment are "augmented" by computer-generated inputs such as sound, video, graphics, touch, Global Positioning System (GPS) data, etc. AR can be used to enhance and / or enrich an individual's experience of the real-world environment.

[0004] Augmented reality (AR) can be used to overlay virtual objects on a display of a captured (e.g., real-time) environment. An AR headset is a device worn on a user's head that has access to AR generation technology. An AR headset may have a camera for capturing video of the user's surroundings and may be configured to overlay one or more AR objects on a display of the environment captured by the camera.

[0005] AR is being used in a variety of different areas. For example, AR is being used to assist surgeons in performing non-invasive surgeries. One specific example involves using AR to assist in correcting arrhythmias. Arrhythmias, or irregular heartbeats, are problems with the heart rate or rhythm, where the heart beats too fast, too slow, or at an irregular rhythm. One way to treat arrhythmias is to apply one or more electrical pulses to specific areas of the heart, or to use cryoablation to freeze specific areas of the heart. This can be done by carefully placing one or more catheters into the heart, recording electrical activity, and identifying the part of the heart that is causing the arrhythmia.

[0006] One way to assist the surgeon in this procedure is to use an electroanatomical mapping system. An electroanatomical mapping system generates a point-by-point map of the interior of the heart and provides the surgeon with the physical boundaries of each part of the heart as well as a map of the electrical signals throughout the heart. A limitation of this system is that the mapping of electrical impulses to specific locations in the heart is inherently three-dimensional (3-D) in nature, while conventional display of the information is limited to a two-dimensional (2-D) display. Therefore, cardiac electrophysiologists require extensive training to map 2-D electrical recordings onto a 3-D human heart.

[0007] One solution to this problem has been disclosed in U.S. Patent No. 10,258,426, which teaches how to use a virtual reality VR / AR environment to assist surgeons in performing non-invasive surgery. In this patent, a hospital server computer renders a 3-D model of an organ (e.g., a heart). The 3-D model is transmitted to a VR / AR headset via a network. Once the VR / AR headset receives the 3-D model, the VR / AR headset begins the process of loading and displaying the model so as to provide the model in an AR environment. Once the model is loaded and displayed in the AR environment, the surgeon can view and interact with the model while performing surgery.

[0008] However, generating a 3-D model of a complex image (e.g., an image of a human heart) is a time-consuming process, and the model takes more than ten minutes to be fully rendered and displayed (i.e., superimposed) on the surgeon's image of the environment. This causes the surgeon to lose valuable time that could be better used to continue the surgery. In addition, due to the physical proximity of the server computer to the AR / VR headset, and because the complete 3-D model is transmitted to the VR / AR headset, the system cannot efficiently provide real-time AR services to users in different geographical locations.

[0009] Some embodiments of the present invention allow an AR head-mounted device to receive a series of data packets containing coordinate data of a 3-D model (e.g., a heart or other organ or body part) over a period of time. This allows the AR head-mounted device to generate an enhanced representation of the heart model in real time. For example, as data packets of portions of the 3-D model are received, the AR head-mounted device can generate an enhanced representation for each corresponding portion of the 3-D model. By generating portions of the 3-D model in real time (e.g., as data packets are received as part of a data stream), surgeons can perform operations related to medical surgery before receiving a complete heart model rendering. For example, while the 3-D model is being rendered, the surgeon may notice a lack of data points being generated. In this case, the surgeon can request more data points or perform another corrective operation. Since the 3-D model is streamed in a fragmented manner, corrective measures can be taken before the complete 3-D model is rendered. This allows surgeons to obtain a complete and accurate 3-D model rendering in a shorter time than the time required to wait for the complete 3-D model to be rendered in order to perform these corrective operations.

[0010] In addition, the AR platform can support AR sessions to enable multiple users to access the 3-D model in real time, even if those users are located in different geographic locations. For example, assume that the AR session is hosted by a first AR head-mounted device. In this case, the AR platform can provide a data packet of a portion of the 3-D model to one or more other AR head-mounted devices so that each corresponding AR head-mounted device can be used to generate an augmented representation of the received 3-D model portion. By providing multiple AR head-mounted devices with real-time (or near real-time) access to an AR session, surgeons in different geographic locations can provide a second opinion on the surgical technique performed by the surgeon hosting the AR session. SUMMARY OF THE INVENTION

[0011] In one aspect of the present invention, a device for augmented reality (AR) head-mounted devices includes: one or more cameras; a memory; and a processor communicatively connected to the memory. The processor is used to receive a series of data packets over a period of time, the data packets containing coordinate data identifying the coordinates of one or more parts of a three-dimensional (3-D) model of an organ. The one or more parts collectively represent a portion of the 3-D model of the organ, rather than the entire 3-D model. The processor is also used to determine, for each corresponding part of the 3-D model, where an enhanced representation of the corresponding part of the 3-D model is placed in an image of an AR environment captured by at least one of the one or more cameras. The processor is also used to overlay an enhanced representation of the corresponding part at a position that has been determined in the image of the AR environment for each corresponding part of the 3-D model. The processor is also used to present an enhanced representation of the corresponding part to a user for use in a medical procedure without presenting an enhanced representation of the entire 3-D model.

[0012] In one embodiment of the present invention, the processor is further configured to continue receiving a series of data packets, the data packets also including coordinate data identifying the coordinates of one or more remaining parts of the 3-D model of the organ. The processor is further configured to determine, for each corresponding remaining part of the 3-D model, a position at which an enhanced representation of the corresponding remaining part of the 3-D model is placed in the image of the AR environment. The processor is further configured to superimpose an enhanced representation of the corresponding remaining part at a position already determined in the image of the AR environment for each corresponding remaining part of the 3-D model. The processor is further configured to present the enhanced representation of the corresponding remaining part to the user for use in a medical procedure.

[0013] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the processor is further configured to receive image data identifying a 3-D model of the catheter. The processor is further configured to determine an initial position in the image of the AR environment at which an augmented representation of the 3-D model of the catheter is placed. The initial position corresponds to the position at which the physical catheter is inserted into the organ. The processor is further configured to superimpose the augmented representation of the 3-D model of the catheter at the initial position in the image of the AR environment. The initial position is updated based on the user moving the physical catheter to a different portion of the organ.

[0014] In another embodiment of the invention, the processor is further configured to receive electrical data indicative of electrical signals measured by the catheter in the organ. The electrical data includes values ​​corresponding to coordinate data values ​​of coordinate data identifying coordinates of a specific portion of the 3-D model. In this embodiment, the processor is further configured to superimpose an enhanced representation of the electrical data onto the enhanced representation of the specific portion of the 3-D model.

[0015] In another embodiment of the present invention, the AR head-mounted device is part of a mapping system. The mapping system also includes a mapping device and a remote server. The mapping device is configured to generate a 3-D model of the organ. The mapping device provides one or more parts of the 3-D model to the remote server so that the remote server streams the one or more parts to the AR head-mounted device within the time period. In another embodiment of the present invention, the mapping system also includes: one or more additional AR head-mounted devices including at least a second AR head-mounted device located at a geographical location different from the geographical location of the AR head-mounted device. When the part of the 3-D model is streamed to the AR device within the time period, the remote server will also stream the one or more parts to the second AR head-mounted device within the time period. The second AR head-mounted device is also used to superimpose the enhanced representation of each corresponding part of the 3-D model in an image of the AR environment displayed by the second AR head-mounted device. In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the processor is also used to receive image data of the 3-D model of the identification catheter. The processor is also used to determine an initial position at which to place the enhanced representation of the 3-D model of the catheter in the image of the AR environment. The initial position corresponds to the position at which the physical catheter is inserted into the organ. The processor is also used to superimpose the enhanced representation of the 3-D model of the catheter at the initial position in the image of the AR environment. The processor is also used to receive positioning data, the positioning data identifying an updated position of the enhanced representation of the 3-D model of the catheter, wherein the updated position is based on a user moving the enhanced representation of the 3-D model of the catheter in the image of the AR environment displayed on the second AR head-mounted device. The processor is also used to update the position of the enhanced representation of the 3-D model of the catheter from the initial position to the updated position in the image of the AR environment displayed by the AR head-mounted device.

[0016] In another embodiment of the present invention, the processor is further configured to receive instructions to manipulate the augmented representation of one or more portions of the superimposed 3-D model. The processor is further configured to manipulate the augmented representation by performing one or more of the following operations: adjusting the size of the augmented representation, adjusting the direction in which the augmented representation is displayed, or adjusting the transparency of the augmented representation.

[0017] In another embodiment of the present invention, a method for streaming an enhanced representation of a three-dimensional (3-D) model of an organ in an image of an augmented reality (AR) environment is provided. The method includes: receiving a series of data packets over a period of time through an AR head-mounted device, the data packets containing coordinate data identifying the coordinates of one or more parts of the 3-D model of the organ. The one or more parts collectively represent a portion of the 3-D model of the organ, rather than the entire 3-D model. The method also includes determining, through the AR head-mounted device and for each corresponding part of the 3-D model, where the enhanced representation of the corresponding part of the 3-D model is placed in the image of the AR environment. The method also includes superimposing, through the AR head-mounted device and for each corresponding part of the 3-D model, an enhanced representation of the corresponding part at a position that has been determined in the image of the AR environment. The method also includes presenting an enhanced representation of the corresponding part to the user for use in a medical procedure without presenting an enhanced representation of the entire 3-D model.

[0018] In another embodiment of the present invention, the method further includes receiving a series of data packets, the data packets also including coordinate data identifying the coordinates of one or more remaining parts of the 3-D model of the organ. The method further includes determining, for each corresponding remaining part of the 3-D model, a position at which an enhanced representation of the corresponding remaining part of the 3-D model is placed in the image of the AR environment. The method further includes, for each corresponding remaining part of the 3-D model, superimposing an enhanced representation of the corresponding remaining part at a position that has been determined in the image of the AR environment. The method also includes presenting the enhanced representation of the corresponding remaining part to the user for use in a medical procedure.

[0019] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the method further includes receiving image data identifying a 3-D model of the catheter. The method further includes determining an initial position in the image of the AR environment to place an augmented representation of the 3-D model of the catheter. The initial position corresponds to the position where the physical catheter is inserted into the organ. The method further includes superimposing the augmented representation of the 3-D model of the catheter at the initial position in the image of the AR environment. The initial position is updated based on the user moving the physical catheter to a different part of the organ.

[0020] In another embodiment of the invention, the method further comprises receiving electrical data indicative of electrical signals measured by the catheter in the organ. The electrical data comprises values ​​corresponding to coordinate data values ​​of coordinate data identifying coordinates of a specific portion of the 3-D model. The method further comprises superimposing an enhanced representation of the electrical data onto an enhanced representation of the specific portion of the 3-D model.

[0021] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the remote server streams one or more portions of the 3-D model to the AR head mounted device and the second AR head mounted device. In this embodiment, the method further includes receiving image data identifying the 3-D model of the catheter. The method further includes determining an initial position for placing an enhanced representation of the 3-D model of the catheter in the image of the AR environment. The initial position corresponds to the position where the physical catheter is inserted into the organ. The method further includes superimposing the enhanced representation of the 3-D model of the catheter on the initial position in the image of the AR environment. The method further includes receiving positioning data identifying an updated position of the enhanced representation of the 3-D model of the catheter. The updated position is based on the user moving the enhanced representation of the 3-D model of the catheter in the image of the AR environment displayed on the second AR device. The method also includes updating the position of the enhanced representation of the 3-D model of the catheter from the initial position to the updated position in the image of the AR environment displayed by the AR head mounted device.

[0022] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the remote server streams one or more portions of the 3-D model to the AR head mounted device and the second AR head mounted device. In this embodiment, the method further includes determining an initial position at which the augmented representation of the 3-D model of the catheter is placed in the image of the AR environment. The initial position corresponds to the position at which the physical catheter is inserted into the organ. The method further includes superimposing the augmented representation of the 3-D model of the catheter on the initial position in the image of the AR environment. The method further includes determining an updated position at which the augmented representation of the 3-D model of the catheter is placed in the image of the AR environment. The updated position is based on the user moving the physical catheter to a different portion of the organ. The method further includes updating the position of the augmented representation of the 3-D model of the catheter from the initial position to the updated position in the image of the AR environment displayed by the AR head mounted device. The method further includes providing positioning data identifying the updated position to the remote server so that the remote server provides the positioning data to the second AR head mounted device so that the second AR head mounted device updates the augmented representation of the 3-D model of the catheter in the image of the AR environment displayed by the second AR head mounted device.

[0023] In another embodiment of the present invention, the method further comprises receiving instructions to manipulate the augmented representation of one or more portions of the superimposed 3-D model. In this embodiment, the method further comprises manipulating the augmented representation by performing one or more of the following operations: adjusting the size of the augmented representation, adjusting the direction in which the augmented representation is displayed, or adjusting the transparency of the augmented representation.

[0024] In another embodiment of the present invention, a mapping system is provided. The mapping system includes an augmented reality (AR) head-mounted device, which is configured to: receive a series of data packets over a period of time, the data packets containing coordinate data identifying the coordinates of one or more parts of a three-dimensional (3-D) model of an organ. The one or more parts collectively represent a part of the 3-D model of the organ, rather than the entire 3-D model. The AR head-mounted device is also configured to determine, for each corresponding part of the 3-D model, a position to place an enhanced representation of the corresponding part of the 3-D model in an image of an AR environment. The AR head-mounted device is also configured to overlay the enhanced representation of the part of the 3-D model at a determined position in the image of the AR environment for each corresponding part of the 3-D model. The AR head-mounted device is also configured to present the enhanced representation of the corresponding part to the user for use in medical surgery without presenting an enhanced representation of the entire 3-D model. The mapping system also includes a mapping device, which is configured to: generate the 3-D model of the organ and provide the series of data packets to the AR head-mounted device.

[0025] In another embodiment of the present invention, the AR head-mounted device is further configured to receive a series of data packets, the data packets also including coordinate data identifying the coordinates of one or more remaining parts of the 3-D model of the organ. The AR head-mounted device is also configured to determine, for each corresponding remaining part of the 3-D model, a position to place an enhanced representation of the corresponding remaining part of the 3-D model in the image of the AR environment. The AR head-mounted device is also configured to, for each corresponding remaining part of the 3-D model, superimpose an enhanced representation of the corresponding remaining part at a position that has been determined in the image of the AR environment. The AR head-mounted device is also configured to present the enhanced representation of the corresponding remaining part to the user for use in medical surgery.

[0026] In another embodiment of the present invention, the AR head mounted device is located at a first geographic location. In this embodiment, the mapping system also includes a second AR head mounted device at a second geographic location different from the first geographic location. In this embodiment, the mapping system also includes a remote server, which is configured to: receive the series of data packets from the mapping device and provide the series of data to the AR head mounted device and the second AR head mounted device. For each corresponding part of the 3-D model received via the data stream, the second AR head mounted device superimposes an enhanced representation of the corresponding part in the image of the AR environment displayed by the second hand mounted device.

[0027] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the remote server streams one or more parts of the 3-D model to the AR head-mounted device and the second AR head-mounted device. In this embodiment, the AR head-mounted device is also configured to receive image data identifying the 3-D model of the catheter. The AR head-mounted device is also configured to determine an initial position for placing an enhanced representation of the 3-D model of the catheter in the image of the AR environment. The initial position corresponds to the position where the physical catheter is inserted into the organ. The AR head-mounted device is also configured to superimpose the enhanced representation of the 3-D model of the catheter at the initial position in the image of the AR environment. The AR head-mounted device is also configured to receive positioning data identifying an updated position of the enhanced representation of the 3-D model of the catheter. The updated position is based on the user moving the enhanced representation of the 3-D model of the catheter in the image of the AR environment displayed on the second AR device. The AR head-mounted device is also configured to update the position of the enhanced representation of the 3-D model of the catheter from the initial position to the updated position in the image of the AR environment displayed by the AR head-mounted device.

[0028] In another embodiment of the present invention, the medical procedure includes inserting a physical catheter into the organ. In this embodiment, the remote server streams one or more parts of the 3-D model to an AR head-mounted device and a second AR head-mounted device. In this embodiment, the AR head-mounted device is also configured to determine an initial position for placing an enhanced representation of the 3-D model of the catheter in the image of the AR environment. The initial position corresponds to the position where the physical catheter is inserted into the organ. The AR head-mounted device is also configured to superimpose the enhanced representation of the 3-D model of the catheter at the initial position in the image of the AR environment. The AR head-mounted device is also configured to determine an updated position for placing the enhanced representation of the 3-D model of the catheter in the image of the AR environment. The updated position is based on the user moving the physical catheter to a different part of the organ. The AR head-mounted device is also configured to update the position of the enhanced representation of the 3-D model of the catheter from the initial position to the updated position in the image of the AR environment displayed by the AR head-mounted device. The AR head mounted device is also configured to provide positioning data identifying the updated location to the remote server, so that the remote server provides the positioning data to the second AR head mounted device, so that the second AR head mounted device updates the augmented representation of the 3-D model of the catheter in the image of the AR environment displayed by the second AR head mounted device. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figures 1A-1H is an overview diagram of one or more example embodiments described herein.

[0030] Figure 2 is a schematic diagram of an example environment in which systems and / or methods described herein may be implemented.

[0031] Figure 3 yes Figure 2 A schematic diagram of example components of one or more devices in FIG.

[0032] Figure 4 An example process for streaming an augmented representation of a three-dimensional (3-D) model of an organ in an image of an augmented reality (AR) environment is illustrated. DETAILED DESCRIPTION OF THE INVENTION

[0033] The following detailed description of example embodiments refers to the accompanying drawings. The same reference numerals in different drawings represent the same or similar elements.

[0034] Figures 1A-1H is an overview diagram of an example method 100 described herein. Figures 1A-1HAs shown, example method 100 shows a user (such as a surgeon) performing a medical procedure using an augmented reality (AR) head mounted device 102. In some embodiments, the medical procedure may be an electrophysiological procedure, such as a cardiac ablation procedure. Although one or more embodiments described herein relate to a cardiac ablation procedure, these embodiments may also be applied to other types of electrophysiological procedures and / or other types of surgical procedures, such as procedures using electrocardiograms, procedures using pacemakers and / or implantable cardioverter defibrillators (ICDs), cardiovascular procedures, interventional radiology procedures, neurosurgery, orthopedic surgery, general surgery, other medical monitoring procedures, etc. Example method 100 shows a surgeon using an AR head mounted device 102 to superimpose an augmented representation of a patient's heart onto an image projected by a camera of the AR head mounted device 102. It should be understood that this is provided as an example, and in practice, an augmented representation (sometimes referred to as an AR object) may represent another type of 3-D object, such as another organ or body part of a patient, or a completely different object in an application outside of a medical use environment.

[0035] like Figure 1A As shown, the AR head-mounted device 102 and the mapping device 104 can be located within a building 106, such as a hospital with an operating room. An AR application can be installed on the AR head-mounted device 102. In some embodiments, the AR application can communicate with the mapping device 104. For example, although not shown, the AR application can communicate with the AR mapping device 104 through a secure cloud background service. That is, the communication between the AR head-mounted device 102 and the mapping device 104, although shown as direct communication, should be understood to be provided through a secure cloud background service (e.g., using the AR platform 164 as an intermediary). For example, instead of communicating directly with the mapping device 104, the AR head-mounted device 102 can send data to the AR platform 164, and then the AR platform 164 can relay the data to the mapping device 104. Similarly, the mapping device 104 can generate a 3-D model and, instead of communicating directly with the AR headset 102, can send data to the AR platform 164, which can then stream the data to the AR headset 102. In other embodiments, the AR headset 102 and the mapping device 104 can communicate directly.

[0036] One or more of the steps described below are performed by the mapping device 104. Although these steps are described as being performed by a single device (e.g., the mapping device 104), it should be understood that the mapping device 104 includes a series of different components required to perform each corresponding step. For example, the mapping device 104 may include a signal acquisition board, a needle box, a switch, a 3-D mapping console, etc. A first set of wires can connect the needle box and one or more catheters. A second set of wires can connect the needle box and a set of electrodes attached to a patient receiving treatment. A signal acquisition board, such as a patient interface unit (PIU), can be connected to the needle box. Switches can be used to allow communication between components of the mapping device 104 (e.g., between the PIU, the 3-D mapping console, etc.). For additional description of these components, see Figure 2 and Figure 3 .

[0037] The mapping device 104 can store or access patient data, including medical data, such as data collected or generated by a signal acquisition board (e.g., PIU) of the mapping device 104, coordinate data identifying the size of a 3-D model of a patient's organ, electronic data identifying one or more electrical signals corresponding to the coordinates of the 3-D model of the organ, image data identifying a 3-D model of one or more catheters, etc. In some embodiments, data collected from a magnetic resonance imaging (MRI) device, a computed tomography (CT) scanner, an electrocardiograph (EKG), etc. can be imported into the signal acquisition board of the mapping device 104.

[0038] In some embodiments, the mapping device 104 can generate coordinate data of a 3-D model of the patient's heart. For example, the mapping device 104 can analyze medical data to generate coordinate data using anatomical mapping technology, image segmentation technology, volume reconstruction technology, surface extraction technology, texture mapping technology, technology driven by machine learning, etc. The coordinate data can include 3-D coordinate values, such as x-coordinate values, y-coordinate values, and z-coordinate values.

[0039] In some embodiments, the mapping device 104 can store or access image data identifying a 3-D model of one or more catheters. For example, various types of catheter 3-D models can be created and image data for these models can be stored or made accessible by the mapping device 104. The 3-D model of the catheter can be defined using 3-D data points, where each data point includes an x-coordinate value, a y-coordinate value, and a z-coordinate value.

[0040] As shown at 108, the surgeon may interact with the AR application to request modeling data for performing a cardiac ablation procedure. The requested modeling data may be a series of data points of a 3-D model of the patient's heart and / or a 3-D model of one or more catheters to be used during the cardiac ablation procedure.

[0041] To submit a request for modeling data, the surgeon may wear and activate the AR head mounted device 102. Activating the AR application may cause the AR head mounted device 102 to superimpose an augmented representation of a menu in the ambient display. The menu may, for example, allow the surgeon to submit a request for modeling data. The menu may also allow the surgeon to enter or select patient information, the type of surgery being performed, the type of organ being operated on, specific medical equipment to be used during the surgery, one or more surgery-specific preferences and / or user preferences, and the like. To select an option in the AR application, the surgeon may use voice commands, gestures, and the like. In this way, the surgeon is able to use the AR application without re-sterilizing his hands.

[0042] The modeling data request can be transmitted to the mapping device 104 using a network such as the Internet, using an application programming interface (API), and / or using another type of communication interface. In some embodiments, the AR head-mounted device 102 can further request any other relevant information, such as patient data and / or surgical data. To provide an example, the AR head-mounted device 102 can further request surgical data, such as data identifying the type of surgery being performed, data identifying the location of the surgery. In some embodiments, the requested data can be encrypted to protect the patient's personally identifiable information (PII) data during data transmission.

[0043] As indicated by reference numeral 110, the mapping device 104 may identify the requested data. For example, the mapping device 104 may access a data structure that associates one or more patient identifiers with corresponding patient data, modeling data, and / or surgical data. In this case, the mapping device 104 may reference the data structure using the one or more patient identifiers included in the request to identify appropriate patient data, modeling data, and / or surgical data. For example, the mapping device 104 may reference the data structure using the patient identifier included in the request to identify coordinate data identifying a 3-D model of the patient's heart and / or image data of a 3-D model of one or more catheters to be used during a cardiac ablation procedure. The boundaries of the 3-D model (and / or the boundaries of components within the model) may be defined using a set of 3-D data points, each of which includes an x-coordinate, a y-coordinate, and a z-coordinate.

[0044] like Figure 1B As shown, referring to reference numeral 112, the mapping device 104 can provide a first set of one or more data packets to the AR head mounted device 102, which contain coordinate data identifying coordinates of a first portion of the heart model.

[0045] A 3-D model of a heart (hereinafter referred to as a heart model). In some embodiments, each data packet may include a set of data points, wherein each data point includes an x-coordinate value, a y-coordinate value, and a z-coordinate value. In some embodiments, each data packet may include a single data point. In some embodiments, each data packet may include a combination of coordinate values ​​of the same type. In some embodiments, each data packet may include a single coordinate value.

[0046] In the example shown, the first group of one or more data packets includes data points corresponding to approximately one-third of the heart model. This can be seen in the image depicting one-third of the patient's heart. An example data point is provided, including coordinates X1, Y1, and Z1. It should be understood that this example is provided for ease of illustration. In practice, the data packets transmitted to the AR head-mounted device 102 may include any number of 3-D data points, including one data point, three data points, five data points, ten data points, hundreds of data points, thousands of data points, etc. For example, the heart model may include 100,000 data points, and these 100,000 data points can be transmitted to the AR head-mounted device 102 in real time as the surgeon acquires the data points, with five data packets, ten data packets, one hundred data packets, etc. transmitted each time.

[0047] Furthermore, the heart model can be generated in any number of different ways. For example, Figure 1B , Figure 1D and Figure 1E Three adjacent regions of the heart model are depicted, which are generated one region at a time (ie, in different time periods). In another example, the outer boundary of the heart can be generated in a first time period, and components within the heart model can be generated in one or more subsequent time periods.

[0048] In some embodiments, the order in which data packets are sent and received can be optimized for the procedure being performed. For example, if a surgeon must place a catheter into a first location in the heart and then move the catheter further into the heart, a first set of one or more data packets may include coordinate data identifying coordinates corresponding to the first location in the heart. Subsequent data packet transmissions may include coordinate data identifying coordinates of one or more other locations in the heart.

[0049] In this way, the AR head mounted device 102 receives coordinate data for the heart model in a piecemeal manner (e.g., as part of a data stream). This provides the surgeon with the option of performing corrective actions before receiving a complete rendering of the heart model. For example, when rendering the heart model, the surgeon may notice that an insufficient number of data points are generated. In this case, the surgeon can request additional data points or perform another corrective action. Because the heart model is streamed in a piecemeal manner, corrective actions can be taken before a complete rendering of the heart model. This allows the surgeon to obtain a complete and accurate rendering of the heart model in a shorter time than the time required if the surgeon had to wait for a complete rendering of the heart model in order to perform these corrective actions.

[0050] As shown in reference numeral 114, the AR head mounted device 102 can generate an enhanced representation of the first portion of the heart model. For example, the AR head mounted device 102 can generate an enhanced representation of the first portion of the heart model by analyzing the coordinate data using one or more modeling and / or rendering techniques. The one or more modeling or rendering techniques may include: computer vision techniques, polygonal mesh generation techniques, point cloud reconstruction techniques, volume reconstruction techniques, surface reconstruction techniques (e.g., ball pivot techniques, Poisson reconstruction techniques, etc.), real-time rendering techniques, mesh simplification techniques, combinations thereof, and / or similar types of techniques.

[0051] Next, the AR head-mounted device 102 can determine the location where the enhanced representation of the first part of the heart model is to be placed in the augmented reality (AR) environment. In some embodiments, the AR head-mounted device 102 can be configured to place the enhanced representation of the first part of the heart model at a default position. For example, the AR head-mounted device 102 can set an origin (e.g., an x-coordinate value of 0, a y-coordinate value of 0, and a z-coordinate value of 0), which is used as a relative origin. That is, other AR objects in the surgeon's environmental image can be identified relative to this position. In some embodiments, the origin can be configured to be located at a certain distance in front of the AR head-mounted device 102. For example, a surgeon may want to view the heart model on or near the patient's chest so that the heart model is visible when the surgeon gazes at the patient. In this case, the origin can be configured as a center point located a short distance in front of the AR head-mounted device 102.

[0052] In some embodiments, the origin can be configured as a user-selectable location. For example, a reference point marker can be superimposed on the image of the AR environment. The surgeon can change the position of the reference point marker. For example, if the reference point marker is used to identify the location of the operating room floor, the surgeon can calibrate or move the reference point marker until the location of the marker matches the location of the floor. In this way, the AR head mounted device 102 is able to calibrate the image of the AR environment to accurately reflect the physical environment around the surgeon.

[0053] As shown in reference numeral 116, the AR head mounted device 102 may overlay the augmented representation of the first part of the heart model into the AR environment. For example, the AR head mounted device 102 may overlay the augmented representation of the first part of the heart model at a default position or a position selected by the user, so that the augmented representation of the first part of the heart model is displayed in the AR environment (e.g., as part of the operating room image).

[0054] In some embodiments, the AR head-mounted device 102 can present an augmented representation of the first portion for the medical procedure to the surgeon. For example, the AR head-mounted device 102 can present an augmented representation of the first portion so that the surgeon can begin the medical procedure without requiring the AR head-mounted device 102 to present an augmented representation of the entire 3D model. As used herein, the term "presentation" refers to an AR head-mounted device (e.g., AR head-mounted device 102, etc.) that displays an augmented representation of a specific portion of a 3D model in an AR environment, such that the augmented representation is visible to a user (e.g., a medical professional, a surgeon, etc.) wearing the AR head-mounted device.

[0055] like Figure 1C As shown and indicated by reference numeral 118, the AR headset 102 can overlay an augmented representation of the catheter model into the AR environment. For example, as described above, the AR headset 102 can receive image data identifying a complete 3-D model of a catheter and can overlay an augmented representation of the catheter model into the AR environment. This allows the surgeon to view and / or manipulate the augmented representation of the catheter model in the AR environment.

[0056] like Figure 1C As shown and indicated by reference numerals 120, 122, and 124, electrical data corresponding to the first portion of the heart model is generated and provided to the AR headset 102. For example, a set of electrode pads may be attached to the patient, such as a magnetic reference pad and a set of impedance pads. A magnetic field generator placed below the patient may emit a magnetic field upward toward the patient, causing the magnetic reference pad to act as a reference point.

[0057] Next, the surgeon can begin inserting one or more physical catheters into the first portion of the patient's heart. The one or more catheters can include electrodes or sensors and can be inserted into a blood vessel and guided to the patient's heart. The one or more catheters can include an electrophysiology (EP) catheter (e.g., a circular mapping catheter, etc.), an ablation catheter (e.g., a force sensing ablation catheter), etc. Once the catheter is positioned in the first portion of the heart, an impedance measurement can be performed. For example, the electrode can generate a small current, and the catheter can measure the electrical signal generated by the current.

[0058] As shown in the reference numeral 120, the one or more catheters can provide electrical signals related to the first portion of the heart to a signal acquisition board of the mapping device 104. In some embodiments, the mapping device 104 can convert the electrical signals into electrical data. In some embodiments, the mapping device 104 can further process the electrical data to determine impedance data, electrophysiological data identifying electrophysiological measurements, voltage map data (e.g., bipolar or unipolar), anatomical map data, activation timing map data, propagation map data, isochronal map data, grid map data, etc.

[0059] As indicated by reference numeral 122, the mapping device 104 can associate the electrical data with coordinate data identifying the coordinates of the heart model. For example, the mapping device 104 can analyze the electrical data to determine the association between each electrical data value and the corresponding coordinate value of the heart model. In some embodiments, the mapping device 104 can determine the association by generating a first portion of a voltage mapping (e.g., corresponding to a first portion of the heart model). The voltage map can be a graphical representation of the electrical activity and / or voltage distribution within the heart captured during the cardiac mapping process. The voltage map can be generated by plotting the measured electrical data values ​​or impedance values ​​onto a 3-D heart model.

[0060] As indicated by reference numeral 124, the mapping device 104 can provide the associated electrical data values ​​to the AR head mounted device 102. For example, when the heart model portion coordinate data has been transmitted, the mapping device 104 can transmit the electrical data in a manner that allows the AR head mounted device 102 to recognize the appropriate association between the electrical data values ​​and the corresponding coordinate values. In some embodiments, the mapping device 104 can transmit the associated coordinate data and the electrical data as part of the same data transmission.

[0061] In some embodiments, electrical data can be transmitted to the AR head mounted device 102 in a piecemeal manner. For example, electrical data can be transmitted periodically based on a configurable trigger condition being met. To provide some examples, electrical data can be transmitted after a threshold amount of time has passed based on a request from a user and / or any other type of configurable trigger condition being met. In some embodiments, electrical data for the entire heart can be captured, mapped, and sent to the AR head mounted device 102 (e.g., as part of a single data transmission).

[0062] In some embodiments, the AR headset 102 can also receive electrical data from a surface electrocardiogram (ECG) patch that provides data representing heart rhythm (e.g., ECG data, electromyogram (EGM) data, etc.). For example, the patch can be placed on the patient's skin at various points on the patient's body. As this activity propagates through the body to the skin, the corresponding patch can detect the electrical activity of the heart. These signals can represent the electrical activity of the entire heart in order to provide a global view of the patient's heart rhythm. This data can be provided directly to the AR headset 102, or it can be relayed to the AR headset 102 via the mapping device 104 and / or the AR platform 156. The data can be superimposed on an image of the AR environment in the same or similar manner as described in conjunction with the electrical signals measured by the catheter.

[0063] As shown in reference numeral 126, the AR headset 102 can overlay an enhanced representation of the electrical data onto the enhanced representation of the first portion of the heart model. For example, the AR headset 102 can overlay an enhanced representation of the first portion of the voltage map onto the enhanced representation of the first portion of the heart model. This allows the surgeon to view the electrical activity measured by the catheter at a specific location on or within the enhanced representation of the first portion of the heart model, thereby helping the surgeon determine the location of the arrhythmia.

[0064] To assist the surgeon in determining the location of the arrhythmia, the voltage map may include visual indicators for distinguishing between different types of electrical signals. For example, electrical data values ​​may be represented using different colors, where each color is assigned to a range of electrical signal values. In addition, or as an alternative, other visual indicators may be implemented, such as visual numerical values, different types of shading of the same color, text indicators of different sizes, and the like. In this way, the AR head mounted device 102 displays data that can be used to assist the surgeon in determining the location of the arrhythmia.

[0065] like Figure 1DAs shown, by reference numeral 128, the mapping device 104 can provide a second set of one or more data packets to the AR head mounted device 102, the data packets including coordinate data for identifying the coordinates of a second portion of the heart model. The second set of one or more data packets can be provided at a time (T2) different from the time (T1) at which the first set of one or more data packets are sent. In the example shown, the second set of one or more data packets includes data points corresponding to approximately one-third of the heart model. The image shown depicts two-thirds of the heart model (e.g., one-third of each of the one or more data packets from the first and second sets, respectively).

[0066] As shown at reference numeral 130, the AR headset 102 can generate an enhanced representation of the second portion of the heart model. For example, the AR headset 102 can generate the enhanced representation of the second portion of the heart model by analyzing the second set of one or more data packets using one or more modeling or rendering techniques.

[0067] In some embodiments, the AR headset 102 can determine a location to place the second portion of the heart model in the AR environment. In the example shown, the AR headset 102 can determine a location so that the augmented representation of the second portion of the heart model is adjacent to the augmented representation of the first portion of the heart model. This placement can be determined using reference point markers as described elsewhere herein.

[0068] In some embodiments, the 3-D data points of the second portion of the heart model may be generated in another order. For example, the AR head mounted device 102 may first generate data points defining the boundary of the second portion of the heart model, and may then generate data points defining the inner wall (and related details) of the second portion of the heart model.

[0069] As shown in reference numeral 132, the AR head mounted device 102 may overlay an augmented representation of the second portion of the heart model in the AR environment. For example, the AR head mounted device 102 may overlay an augmented representation of the second portion of the heart model at the determined position so that the augmented representations of the first portion and the second portion of the heart model are visibly connected in the AR environment.

[0070] In some embodiments, the AR head mounted device 102 can present an augmented representation of the first and second parts for the medical procedure to the surgeon. For example, the AR head mounted device 102 can present an augmented representation of the first part and the second part so that the surgeon can continue the medical procedure without requiring the AR head mounted device 102 to present an augmented representation of the entire 3D model. To provide a specific example, the surgeon can continue the medical procedure by performing an action on the second part of the 3D model without requiring the AR head mounted device 102 to present an augmented representation of the first part of the 3D model.

[0071] like Figure 1E As shown, and by reference numeral 134, the mapping device 104 (e.g., a signal acquisition board) can receive electrical signals associated with the second portion of the heart from one or more catheters. For example, a surgeon can position the catheter into the second portion of the heart, and the electrical pulses generated by the one or more catheters can be acquired by or provided to the signal acquisition board of the mapping device 104. In some embodiments, the mapping device 104 can convert the electrical signals into electrical data. In some embodiments, the mapping device 104 (e.g., a signal acquisition board) can further process the electrical data to determine impedance values ​​and / or related values.

[0072] As indicated by reference numeral 136, the mapping device 104 can associate these electrical data values ​​with coordinate values ​​of a portion of the second portion of the heart model. For example, the mapping device 104 can analyze the electrical data to determine an association between corresponding electrical data values ​​and corresponding coordinate values ​​of the second portion of the heart model. In some embodiments, the mapping device 104 can determine the association by generating a second portion of the voltage map (e.g., which corresponds to the second portion of the heart model).

[0073] As indicated at 138, the mapping device 104 may provide the associated electrical data values ​​to the AR headset 102. As described elsewhere herein, in particular embodiments, the electrical data, coordinate data, and / or associated electrical and coordinate data may be transmitted separately or simultaneously.

[0074] As shown in reference numeral 140, the AR headset 102 can overlay an enhanced representation of the electrical data onto an enhanced representation of the second portion of the heart model. For example, the AR headset 102 can overlay an enhanced representation of the second portion of the voltage map onto an enhanced representation of the second portion of the heart model. This allows the surgeon to view the electrical activity measured by the catheter at a specific location on or within the enhanced representation of the second portion of the heart model, thereby helping the surgeon identify the location of the arrhythmia.

[0075] like Figure 1FAs shown by reference numeral 142, the mapping device 104 can provide a third set of one or more data packets to the AR head mounted device 102, the data packets including coordinate data for identifying coordinates of a third portion of the heart model. The third set of one or more data packets can be provided at a time (T3) different from the time (T1, T2) at which the first and second sets of one or more data packets are sent.

[0076] As shown at reference numeral 144, the AR head mounted device 102 may generate an enhanced representation of a third portion of the heart model. For example, the AR head mounted device 102 may generate an enhanced representation of the third portion of the heart model by analyzing the third group of one or more data packets using one or more modeling or rendering techniques.

[0077] In some embodiments, the AR head mounted device 102 can determine a location to place the third portion of the heart model in the AR environment. In the example shown, the AR head mounted device 102 can determine a location such that the augmented representation of the third portion of the heart model is adjacent to the augmented representation of the first portion of the heart model. This placement can be determined using reference point markers as described elsewhere herein.

[0078] In some embodiments, the 3-D data points of the third portion of the heart model may be generated in another order. For example, the AR head mounted device 102 may first generate data points defining the boundary of the third portion of the heart model, and then may generate data points defining the inner wall (and related details) of the third portion of the heart model.

[0079] As shown in reference numeral 146, the AR head mounted device 102 may overlay an augmented representation of the third portion of the heart model in the AR environment. For example, the AR head mounted device 102 may overlay an augmented representation of the third portion of the heart model at the determined position so that the first, second, and third portions of the heart model are visibly connected in the AR environment.

[0080] In some embodiments, the AR headset 102 can present augmented representations of the first, second, and third portions for the medical procedure to the surgeon. For example, the AR headset 102 can present augmented representations of the first, second, and third portions so that the surgeon can continue the medical procedure, such as by performing aspects of the medical procedure involving the third portion of the 3D model.

[0081] like Figure 1G As shown, and by reference numeral 148, the mapping device 104 can receive electrical signals related to the third portion of the heart from the one or more catheters. For example, a surgeon can position a catheter into the third portion of the heart, and one or more electrical pulses generated by the one or more catheters can be acquired or received by a signal acquisition board of the mapping device 104.

[0082] The mapping device 104 may associate the electrical data values ​​with coordinate values ​​that are part of the third portion of the heart model, as shown at 150. For example, the mapping device 104 may convert the electrical signals into electrical data and / or may process the electrical data to determine impedance values ​​and / or related values.

[0083] As indicated at 152, the mapping device 104 may provide associated electrical data values ​​to the AR head mounted device 102. As described elsewhere herein, in some embodiments, electrical data, coordinate data, and / or associated electrical and coordinate data may be transmitted separately or simultaneously.

[0084] As shown in reference numeral 154, the AR headset 102 can overlay the enhanced representation of the electrical data on the enhanced representation of the third portion of the heart model. For example, the AR headset 102 can overlay the enhanced representation of the third portion of the voltage map on the enhanced representation of the third portion of the heart model. This allows the surgeon to view the electrical activity measured by the catheter at a specific location on or within the enhanced representation of the third portion of the heart model, thereby helping the surgeon identify the location of the arrhythmia.

[0085] In some embodiments, the surgeon can manipulate the enhanced representation of the heart model. For example, the surgeon can manipulate the enhanced representation of the heart model by expanding, contracting, enlarging, reducing, changing the direction of the enhanced representation display, adjusting the transparency, etc.

[0086] In some embodiments, the surgeon can interact with the AR headset 102 using audio communication. For example, the AR application can be configured to receive voice commands from the surgeon. In this way, the AR headset 102 allows the surgeon to view and manipulate an augmented representation of the heart model without having to re-sterilize his or her hands or use a marker or operator at the mapping interface. In some embodiments, the AR headset 102 can interpret voice commands using machine learning and / or natural language processing. This allows the AR headset 102 to correctly interpret voice commands even if the surgeon does not provide pre-configured commands verbatim.

[0087] In some embodiments, a remotely guided catheter can be used to perform a cardiac ablation procedure. For example, a surgeon can adjust the position of an augmented representation of a catheter model in an AR environment. This causes the AR headset 102 to generate instructions and provide instructions to a controller of the physical catheter. The controller can process the instructions and can cause the position of the physical catheter to be updated based on the instructions to reflect the adjusted position of the augmented representation of the catheter model. In some embodiments, the remotely guided catheter movement can be driven by machine learning.

[0088] like Figure 1H As shown, the AR platform 156 allows a second AR head mounted device (represented as AR head mounted device 158) to access the AR session in real time (or near real time). The AR head mounted device 158 can access the AR session even if the AR head mounted device 102 and the AR head mounted device 158 are located in different geographical locations (e.g., building 106 and building 160). In order to access the AR environment, the AR head mounted device 158 can launch an AR application and can submit an access request to the AR platform 156. The request can include a session identifier, a session password, a user identifier of the user for accessing the AR head mounted device 158, etc.

[0089] As indicated by reference numeral 162, the AR headset 158 ​​may provide a request to the AR platform 156 to access the AR session. As indicated by reference numeral 164, the AR platform 156 may identify the AR session data. For example, the AR platform 156 may access a data structure that associates an AR session identifier with AR session data (e.g., AR environment data, image data identifying one or more 3-D models of a catheter, a data packet including coordinate data for building a heart model, etc.). In this case, the AR platform 156 may use the session identifier to reference the data structure to identify the associated AR session data.

[0090] In some embodiments (not shown), the AR platform 156 can determine access rights to the AR session. For example, a first group of one or more users can have read access rights to the AR session, while a second group of users can have write access rights to the AR session. In addition, the AR platform 156 can access a data structure that associates user identifiers of various users with access identifiers corresponding to specific access types. In some embodiments, a user can be assigned a default read-only access right, and if the authentication process is successfully completed, the user can be provided with write access rights. In some embodiments, the host or initiator of the AR session can assign specific access rights to the AR session.

[0091] In some embodiments, one or more types of data described herein may be stored on the AR platform 156, the mapping device 104, or both the AR platform 156 and the mapping device 104. In this case, each local device may have a temporary copy of the data. However, each local copy can be updated by the AR platform 156. Thus, the AR platform 156, the mapping device 104, and any connected AR head-mounted devices (e.g., AR head-mounted devices 102, 158, etc.) can remain synchronized with each other during an AR session.

[0092] As indicated at 166, the AR platform 156 may provide the AR session data to the AR headset 158. This provides the AR headset 158 ​​with real-time (or near real-time) access to the AR session.

[0093] In some embodiments, the AR platform 156 can provide AR environment data of the AR environment to the AR head mounted device 158. The AR environment data can include image data identifying an operating room, and / or data that can be used to create an augmented representation of an operating room (or similar medical environment). In addition, or as an alternative, the AR platform 156 can provide image data of one or more catheters to the AR head mounted device 158.

[0094] In addition, or as an alternative, the AR platform 164 can provide coordinate data of the heart model, and / or associated electrical data to the AR head-mounted device 158. In some embodiments, the coordinate data and / or associated electrical data can be provided in a fragmented manner as described elsewhere herein. For example, assume that the AR head-mounted device 158 is connected to the AR session shortly after the AR head-mounted device 102. In this example, the AR platform 156 can send one or more data packets of the first group, the second group, and the third group to the AR head-mounted device 158, respectively. Similarly, the AR platform 164 can send appropriate corresponding electrical data separately in a similar manner. In some embodiments, the coordinate data and / or associated electrical data can be provided to the AR head-mounted device 158 as part of the same data transmission.

[0095] As indicated by reference numeral 168, the AR headset 158 ​​can generate and / or display AR session data. In some embodiments, the AR headset 158 ​​can load and display an AR environment so that the user can see an image of the environment. In addition, or as an alternative, the AR headset 158 ​​can load and display one or more augmented representations of the catheter. In addition, or as an alternative, the AR headset 158 ​​can load and display one or more parts of the heart model. In some embodiments, electrical data can be superimposed on one or more displayed parts of the heart model.

[0096] To provide a specific example, assume that the AR head-mounted device 158 receives a first group, a second group, and a third group of data packets in three separate time periods. Further assume that the AR head-mounted device 158 receives corresponding electrical data. In this example, the AR head-mounted device 158 can generate and display an enhanced representation of a first portion of the heart model in a first time period (e.g., shortly after receiving the coordinate data), can generate and display an enhanced representation of a second portion of the heart model in a second time period, and can generate and display an enhanced representation of a third portion of the heart model in a third time period. In this way, each AR head-mounted device connected to the AR session is able to generate a heart model when a data packet with coordinate data is received in real-time (or near real-time), without always receiving a complete heart model and loading or rendering a complete heart model at the same time.

[0097] In some embodiments, the AR platform 156 can provide data identifying updated positioning of one or more catheters to the AR headset 158. For example, as the surgeon moves the catheter further into the patient's heart, data identifying the updated position of the catheter can be provided to the AR platform 156 (e.g., from the mapping device 104). In this case, the AR platform 156 can provide this data to the AR headset 158. This allows the AR headset 158 ​​to adjust the position of the augmented representation of the catheter to reflect the physical movement of the catheter controlled by the surgeon.

[0098] In some embodiments, the AR platform 156 can support audio transmission between one or more AR head-mounted devices (e.g., AR head-mounted device 102, AR head-mounted device 158, etc.). Audio data can be multiplexed and transmitted on a different channel from video data / image data associated with an AR object.

[0099] In some embodiments, a user accessing the AR head mounted device 158 can manipulate a local copy of the augmented representation of the heart model. For example, the user may have read-only access and may access view-related functions (e.g., expand the heart model, shrink the heart model, zoom in, zoom out, etc.). However, the user's manipulation of the heart model may not affect the heart model displayed by the AR head mounted device 102.

[0100] In some embodiments, a user accessing the AR head mounted device 158 can manipulate each respective copy of the heart model and / or the catheter model. For example, assume that the user has write access rights. In this case, the user can manipulate the local copy of the augmented representation of the heart model and / or the augmented representation position of the catheter. This may cause the same operation to occur on the AR head mounted device 102 (and / or any other AR head mounted device connected to the AR session). For example, the AR head mounted device 158 may provide data indicating the manipulation performed by the user (i.e., manipulation data) to the AR platform 156. This may cause the AR platform 156 to verify the user's access rights and provide the manipulation data to the AR head mounted device 102.

[0101] In this way, the AR platform 156 allows multiple AR headsets (e.g., the AR headset 102, the AR headset 158, etc.) to gain real-time access to an AR session, thereby allowing surgeons to obtain second opinions from specialists even in different geographic locations.

[0102] As mentioned above, Figures 1A-1H This is provided as an example only. Other examples may exist and may be similar to Figures 1A-1H For example, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or Figures 1A-1H The devices and / or networks shown in FIG. 1 may be arranged differently from the devices and / or networks shown in FIG. 1 . In addition, Figures 1A-1H Two or more of the devices shown may be implemented in a single device, or Figures 1A-1H The single device shown may be implemented as multiple and / or distributed devices. In addition, or as an alternative, a set of devices (e.g., one or more devices) of the example method 100 may perform one or more functions described as being performed by another set of devices in the example method 100. In addition, although the embodiments described herein describe an AR environment, in other embodiments, a virtual reality (VR) environment utilizing VR technology may be implemented. Furthermore, in some embodiments, an environment utilizing both VR technology and AR technology, mixed reality, or some combination thereof may be utilized.

[0103] Figure 2 2 is a schematic diagram of an example environment 200 in which the systems and / or methods described herein may be implemented. Figure 2As shown, environment 200 may include a group of augmented reality (AR) devices 202 (collectively referred to as AR head mounted devices 202, or individually represented as AR head mounted devices 202, 202-1, 202-2, ..., 202-N, etc.), a mapping device 204, a building 206, an AR platform 208 hosted in a cloud computing environment 210 (which may include one or more computing resources 212), and a network 214. The devices in environment 200 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections. Figure 2 One or more devices in may correspond to the devices in Figure 1. For example, the AR head mounted device 102 may correspond to the AR head mounted device 202, the mapping device 102 may correspond to the mapping device 204, and so on.

[0104] The AR head-mounted device 202 includes one or more devices capable of receiving, generating, storing, processing, transmitting and / or displaying information related to the AR session. For example, the AR head-mounted device 202 may include a communication and / or computing device. In a preferred embodiment, the AR head-mounted device may be a wearable communication device (e.g., a smart watch, smart glasses, smart clothing, etc.), a gaming device, or a similar type of device.

[0105] In other embodiments, the AR head-mounted device 202 may include a mobile phone (e.g., a smartphone, a wireless phone, etc.), a portable computer, a tablet computer, a handheld computer, or a similar type of device.

[0106] The AR head mounted device 202 includes an image capture component, such as a camera, a camcorder, and / or other types of image capture components. The AR head mounted device 202 may support an AR application that allows the AR head mounted device 202 to host an AR session. When an AR session is active, the AR head mounted device 202 may use the image capture component to display an AR environment (e.g., a display of the environment including AR objects).

[0107] In some embodiments, the AR head-mounted device 202 may have multiple cameras. The AR head-mounted device 202 may have an eye tracking camera, a depth camera, an environmental camera, an infrared (IR) camera, a simultaneous localization and mapping (SLAM) camera, a gesture tracking camera, a thermal imager, a wide-angle camera, a high-resolution camera, a color and IR hybrid camera, a lidar sensor, etc. For example, the AR head-mounted device 202 may include an eye tracking camera for tracking eye movement, pupil dilation, gaze direction, etc., one or more cameras focused on the external environment, and / or one or more cameras capturing different spectral ranges, such as a visible light (RGB) camera, an IR camera, an ultraviolet (UV) camera, a multi-spectral camera, etc.

[0108] In some embodiments, the AR head mounted device 202 includes a display device (e.g., primary optical components) that displays the 3D model and any associated 3D data. In this case, the display device can perform or enable the optical functions of the device, and the AR device 202 can further include one or more cameras for performing or enabling other functions, such as acquiring local spatial data, acquiring data indicating a particular spectrum, etc.

[0109] In some embodiments, a first AR head-mounted device, such as AR head-mounted device 202-1, can be used in building 206 to assist a user in performing a medical procedure. In some embodiments, the AR head-mounted device 202-1 can establish a communication session with the mapping device 204. In this case, the AR head-mounted device 202-1 can use the communication session to request image data of one or more catheter models and / or coordinate data identifying a three-dimensional (3-D) model of the heart (i.e., a heart model). In some embodiments, the AR head-mounted device 202-1 can receive a series of data packets from the mapping device 204 over time. For example, the mapping device 204 can provide the AR head-mounted device 202-1 with a series of data packets containing coordinate data identifying the coordinates of parts of the heart model. This allows the AR head-mounted device 202-1 to generate and overlay an enhanced representation of each corresponding model onto a display of the user environment (e.g., sometimes referred to as an AR environment).

[0110] In some embodiments, the AR head mounted device 202, such as the AR head mounted device 202-1, can communicate with the AR platform 208. The AR platform 208 is configured to assist the AR head mounted device 202-1 in hosting an AR session, which can be accessed in real time by one or more other users (e.g., users using one or more other AR head mounted devices, such as the AR head mounted device 202-2, ..., the AR head mounted device 202-N, etc.).

[0111] The mapping device 204 includes one or more devices capable of acquiring, receiving, generating, storing, processing and / or transmitting information related to the AR object and / or related to the patient's medical data. The mapping device 204 may include a cardiac mapping system for performing one or more steps or functions described herein. The cardiac mapping system can be used for electrophysiological surgery such as cardiac ablation surgery. The cardiac mapping system can combine real-time electrical signal mapping in the heart with imaging technology to create a 3-D model of the heart and identify areas of abnormal electrical activity. The cardiac mapping system may include a signal acquisition board, such as a patient interface unit (PIU), a needle box, a switch, a 3-D mapping console, one or more processors, a flushing pump, an electronic tracking processing unit, a positioning processing unit, a cardiac radio frequency generator, etc. In some embodiments, a first set of wires can connect the needle box to one or more catheters, and a second set of wires can connect the needle box to a set of electrode pads. The signal acquisition board can be connected to the needle box. The switch can connect or allow communication between these components (e.g., a signal acquisition board, a 3-D mapping console, one or more processors and / or other components of the cardiac mapping system, etc.).

[0112] Building 206 includes one or more physical structures capable of housing AR head mounted devices 202 and / or mapping devices 204. In some embodiments, building 206 is a hospital. The hospital may include one or more operating rooms, and surgeons may perform surgery in the operating room using AR head mounted devices 202. In some embodiments, AR head mounted device 202-1 and / or mapping device 204 may be located in the same room in building 206. In some embodiments, one or more other AR head mounted devices 202 (e.g., AR head mounted devices 202-2, 202-3, ..., 202-N) may be located in another building 206.

[0113] The AR platform 208 includes one or more devices (e.g., computing resources 212) capable of receiving, generating, storing, processing, and / or transmitting information related to an AR session. For example, the AR platform 208 may include a server device or a group of servers. In some embodiments, the AR platform 208 may support an AR session hosted by the AR head mounted device 202-1. In this case, the AR platform 208 may provide real-time (or near real-time) access to the AR session to one or more other AR head mounted devices (e.g., AR head mounted devices 202-2, ..., 202-N).

[0114] In some embodiments, the AR platform 208 can communicate directly with the AR head-mounted device 202. In some embodiments, the AR platform 208 can communicate with the mapping device 204, and the mapping device 204 can communicate directly with the AR head-mounted device 202.

[0115] The cloud computing environment 210 includes an environment that provides computing as a service, whereby shared resources, services, etc. can be provided to the AR head mounted device 202. The cloud computing environment 210 can provide computing, software, data access, storage, and / or other services that require the end user to know the physical location and configuration of the system and / or device providing the service. As shown above, the cloud computing environment 210 can include a set of computing resources 212 (hereinafter individually referred to as computing resources 212).

[0116] It is worth noting that although the embodiments described in this specification describe the AR platform 208 as being hosted in the cloud computing environment 210, in some embodiments, the AR platform 208 is not cloud computing-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud computing-based.

[0117] The computing resources 212 include one or more personal computers, workstation computers, servers, or other types of computing and / or communication devices. In some embodiments, the computing resources 212 can host the AR platform 208. Cloud resources can include computing instances executed in the computing resources 212, storage devices provided within the computing resources 212, data transmission devices provided by the computing resources 212, etc. In some embodiments, the computing resources 212 can communicate with other computing resources 212 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0118] like Figure 2 As further shown in FIG. 1 , the computing resources 212 may include a set of cloud resources, such as one or more applications (“APPs”) 212 - 1, one or more virtual machines (“VMs”) 212 - 2, virtualized storage (“VSs”) 212 - 3, one or more hypervisors (“HYPs”) 212 - 4, and / or similar resources.

[0119] The applications 212-1 include one or more software applications that can be provided to or accessed by the AR head mounted device 202. The applications 212-1 can eliminate the need to install and execute software applications on the AR head mounted device 202. For example, the applications 212-1 can include software associated with the AR platform 208, and / or any other software that can be provided through the cloud computing environment 210. In some embodiments, one application 212-1 can send / receive information to one or more other applications 212-1 through the virtual machine 212-2.

[0120] The virtual machine 212-2 comprises a software implementation of a machine (e.g., a computer) that executes programs just like a physical machine. The virtual machine 212-2 can be a system virtual machine or a process virtual machine, depending on the use case and the degree to which the virtual machine 212-2 corresponds to any actual machine. A system virtual machine can provide a complete system platform that supports execution of a complete operating system ("OS"). A process virtual machine can execute a single program and can support a single process. In some embodiments, the virtual machine 212-2 can execute on behalf of a user (e.g., an AR headset 202) and can manage the infrastructure of the cloud computing environment 210, such as data management, synchronization, or long-term data transfer.

[0121] Virtualized storage 212-3 includes one or more storage systems and / or one or more devices using virtualization technology in the storage systems or devices of computing resources 212. In some embodiments, in the environment of the storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization refers to separating logical storage from physical storage so that the storage system can be accessed without considering physical storage or heterogeneous structures. This separation can give administrators of the storage system flexibility in how the administrator manages storage for end users. File virtualization can eliminate the dependency between data accessed at the file level and the actual storage location of the file. This makes it possible to optimize the performance of storage usage, server consolidation and / or non-disruptive file migration.

[0122] Hypervisor 212-4 can provide hardware virtualization technology to enable multiple operating systems (e.g., "guest operating systems") to execute simultaneously on a host computer (e.g., computing resource 212). Hypervisor 212-4 can present a virtual operating platform to the guest operating systems and can manage the execution of the guest operating systems. Multiple instances of various operating systems can share virtualized hardware resources.

[0123] The network 214 includes one or more wired and / or wireless networks. For example, the network 214 may include a cellular network (e.g., a long-term evolution (LTE) network, a code division multiple access (CDMA) network, a 3G network, a 4G network, a 5G network, another next-generation network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0124] Figure 2The number and arrangement of devices and networks shown in the figure are provided as an example only. In practice, there may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or different Figure 2 Those devices and / or networks that are arranged differently than those shown. In addition, Figure 2 Two or more of the devices shown in may be implemented in a single device, or Figure 2 A single device shown in the environment 200 may be implemented as multiple distributed devices. In addition, or as an alternative, a set of devices (eg, one or more devices) of the environment 200 may perform one or more functions described as being performed by another set of devices of the environment 200.

[0125] Figure 3 300 is a schematic diagram of example components of device 300. Device 300 may correspond to one or more AR head-mounted devices 202 (e.g., AR head-mounted device 202-1, AR head-mounted device 202-2, ..., AR head-mounted device 202-N, etc.), mapping device 204, and / or AR platform 208. In some embodiments, one or more AR head-mounted devices (e.g., AR head-mounted device 202-1, AR head-mounted device 202-2, ..., AR head-mounted device 202-N, etc.), mapping device 204, and / or AR platform 208 may include one or more devices 300 and / or one or more components of device 300. Figure 3 As shown, device 300 may include a bus 310 , a processor 320 , a memory 330 , a storage component 340 , an input component 350 , an output component 360 , and a communication interface 370 .

[0126] Bus 310 includes a component that allows communication between components of device 300. Processor 320 can be implemented in hardware, firmware, or a combination of hardware and software. Processor 320 may include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or other types of processing components. In some embodiments, processor 320 may include one or more processors that can be programmed to perform functions. Memory 330 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by processor 320.

[0127] Storage component 340 stores information and / or software related to the operation and use of device 300. For example, storage component 340 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium, and a corresponding drive.

[0128] Input components 350 include components that allow device 300 to receive information, such as through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). In addition, or as an alternative, input components 350 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 360 include components that provide output information from device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs).

[0129] The communication interface 370 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 300 to communicate with other devices, such as through a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 370 can allow the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 370 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0130] Device 300 may perform one or more processes described herein. Device 300 may perform these processes in response to processor 320 executing software instructions stored by non-transitory computer-readable media (e.g., memory 330 and / or storage component 340). Computer-readable media is defined herein as non-transitory storage devices. Storage devices include storage space of a single physical storage device or storage space distributed on multiple physical storage devices.

[0131] The software instructions may be read into the memory 330 and / or storage component 340 from another computer-readable medium or from another device through the communication interface 370. When executed, the software instructions stored in the memory 330 and / or storage component 340 may cause the processor 320 to perform one or more processes described herein. In addition, or as an alternative, hardwired circuitry may be used instead of or in combination with software instructions to perform one or more processes described herein. Therefore, the various embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0132] Figure 3The number and arrangement of components shown in FIG. 3 are provided only as examples. In practice, the device 300 may include more than Figure 3 More, fewer, different or differently arranged components than those shown in . In addition, or as an alternative, one set of components (e.g., one or more components) of device 300 can perform one or more functions performed by another set of components of device 300.

[0133] Figure 4 4 is a flow chart of an example process 400 for streaming an augmented representation of a three-dimensional (3-D) model of an organ in an image of an augmented reality (AR) environment. As shown in box 410, the example process 400 includes receiving a series of data packets over a period of time, the data packets containing coordinate data identifying coordinates of one or more parts of the 3-D model of the organ. For example, the AR head mounted device 102 (e.g., using a processor) can receive a series of data packets over a period of time, the data packets containing coordinate data identifying coordinates of one or more parts of the 3-D model of the organ. The one or more parts collectively represent a portion of the 3-D model of the organ, rather than the entire 3-D model.

[0134] As shown in box 420, the example process 400 also includes determining, for each respective portion of the 3-D model, a location to place an augmented representation of the respective portion of the 3-D model in the image of the AR environment. For example, the AR head mounted device 102 (e.g., using a processor) may determine, for each respective portion of the 3-D model, a location to place the augmented representation of the respective portion of the 3-D model in the image of the AR environment.

[0135] As shown in box 430, the example process 400 also includes, for each corresponding part of the 3-D model, superimposing an augmented representation of the corresponding part at the determined location in the image of the AR environment. For example, for each corresponding part of the 3-D model, the AR head mounted device 102 (e.g., using a processor) may superimpose an augmented representation of the corresponding part at the determined location in the image of the AR environment.

[0136] As shown in box 440, the example process 400 also includes presenting an augmented representation of the corresponding portion to the user for use in the medical procedure without presenting an augmented representation of the entire 3-D model. For example, the AR head mounted device 102 (e.g., using a processor, one or more cameras, etc.) can present an augmented representation of the corresponding portion for use in the medical procedure to the user (e.g., a medical professional, a surgeon, etc.) without presenting an augmented representation of the entire 3D model. This saves time by allowing the medical professional to begin the medical procedure without having to wait for the 3-D model to be fully rendered.

[0137] The above disclosed contents are only for illustration and description, and are not intended to be exhaustive description or to limit the embodiments to the exact form disclosed. According to the above disclosed contents, modifications and changes may be made, or possible modifications and changes may be obtained by implementing these embodiments.

[0138] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, and / or a combination of hardware and software.

[0139] It is apparent that the systems and / or methods described herein may be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the embodiments. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0140] Even if specific combinations of features are disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. In fact, many of these features can be combined in ways not specifically listed and / or disclosed in the specification.

[0141] Unless explicitly stated, any element, behavior or instruction used in this article should not be interpreted as critical or essential. In addition, as used herein, the article "a / an" is intended to include one or more items and can be used interchangeably with "one / a or more / a". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.), and can be used interchangeably with "one or more". When only one item is intended to be used, the term "one" or similar terms are used. In addition, as used herein, the term "has / have / having" or similar terms are intended to be open terms. In addition, the phrase "based on" is intended to mean "based at least in part on", unless otherwise explicitly stated.

Claims

1. For use in augmented reality (AR) head-mounted devices, including: One or more cameras; Memory; and A processor is communicatively coupled to the memory to: receiving a series of data packets over a period of time, the data packets comprising coordinate data identifying coordinates of one or more portions of a three-dimensional (3-D) model of an organ, wherein the one or more portions collectively represent a portion of the 3-D model of the organ, but not the entire 3-D model; For each respective portion of the 3-D model, determining a location to place an augmented representation of the respective portion of the 3-D model in an image of the AR environment captured by at least one of the one or more cameras; For each corresponding portion of the 3-D model, superimposing an augmented representation of the corresponding portion at the determined position in the image of the AR environment; and An enhanced representation of the corresponding portion is presented to the user for use in the medical procedure without presenting an enhanced representation of the entire 3-D model.

2. The AR head mounted device according to claim 1, wherein the processor is further configured to: continuing to receive a series of data packets, the data packets further comprising coordinate data identifying coordinates of one or more remaining portions of the 3-D model of the organ; For each respective remaining portion of the 3-D model, determining a location in the image of the AR environment to place an augmented representation of the respective remaining portion of the 3-D model; For each respective remaining portion of the 3-D model, superimposing an augmented representation of the respective remaining portion at the determined location in the image of the AR environment; and An enhanced representation of the corresponding remaining portion is presented to a user for use in a medical procedure.

3. The AR head mounted device of claim 1, wherein the medical procedure comprises inserting a physical catheter into the organ; and wherein the processor is further configured to: receiving image data identifying a 3-D model of a catheter; determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which the physical catheter is inserted into the organ; and An augmented representation of the 3-D model of the catheter is superimposed at an initial position in the image of the AR environment, wherein the initial position is updated based on the user moving the physical catheter to a different portion of the organ.

4. The AR head mounted device according to claim 3, wherein the processor is further configured to: receiving electrical data indicative of electrical signals measured by the catheter in the organ, wherein the electrical data includes values ​​corresponding to coordinate data values ​​of coordinate data identifying coordinates of a particular portion of the 3-D model; and The enhanced representation of the electrical data is superimposed onto the enhanced representation of the particular portion of the 3-D model.

5. The AR head mounted device according to claim 1, wherein the AR head mounted device is part of a mapping system, and the mapping system further comprises: A mapping device and a remote server, wherein the mapping device is configured to generate a 3-D model of the organ, and wherein the mapping device provides one or more parts of the 3-D model to the remote server so that the remote server streams the one or more parts to the AR head-mounted device within the time period.

6. The mapping system according to claim 5, wherein the mapping system further comprises: One or more additional AR head-mounted devices including at least a second AR head-mounted device located at a geographical location different from the geographical location of the AR head-mounted device, wherein when the portions of the 3-D model are streamed to the AR device during the time period, the remote server will also stream the one or more portions to the second AR head-mounted device during the time period, and wherein the second AR head-mounted device superimposes the augmented representation of each corresponding portion of the 3-D model in an image of the AR environment displayed by the second AR head-mounted device.

7. The AR head mounted device of claim 6, wherein the medical procedure comprises inserting a physical catheter into the organ; and wherein the processor is further configured to: receiving image data identifying a 3-D model of a catheter; determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which the physical catheter is inserted into the organ; superimposing an augmented representation of the 3-D model of the catheter at an initial position in the image of the AR environment; receiving positioning data identifying an updated position of the augmented representation of the 3-D model of the catheter, wherein the updated position is based on a user moving the augmented representation of the 3-D model of the catheter in the image of the AR environment displayed on the second AR head mounted device; and In an image of the AR environment displayed by the AR head mounted device, a position of the augmented representation of the 3-D model of the catheter is updated from the initial position to an updated position.

8. The AR head mounted device according to claim 1, wherein the processor is further configured to: receiving instructions to manipulate the augmented representation of one or more portions of the superimposed 3-D model; and The augmented representation is manipulated by performing one or more of the following operations: resizing the augmented representation, adjusting the orientation of the display of the enhanced representation, or Adjusts the transparency of the enhanced representation.

9. A method for streaming an augmented representation of a three-dimensional (3-D) model of an organ in an image of an augmented reality (AR) environment, the method comprising: receiving, via the AR head mounted device and over a period of time, a series of data packets containing coordinate data identifying coordinates of one or more portions of the 3-D model of the organ, wherein the one or more portions collectively represent a portion of the 3-D model of the organ, but not the entire 3-D model; determining, by the AR head mounted device and for each respective portion of the 3-D model, a location in the image of the AR environment at which to place an augmented representation of the respective portion of the 3-D model; superimposing, by the AR head mounted device and for each corresponding portion of the 3-D model, an augmented representation of the corresponding portion at the determined location in the image of the AR environment; and An enhanced representation of the corresponding portion is presented to the user for use in the medical procedure without presenting an enhanced representation of the entire 3-D model.

10. The method according to claim 9, wherein the method further comprises: continuing to receive a series of data packets, the data packets further comprising coordinate data identifying coordinates of one or more remaining portions of the 3-D model of the organ; For each respective remaining portion of the 3-D model, determining a location in the image of the AR environment to place an augmented representation of the respective remaining portion of the 3-D model; For each respective remaining portion of the 3-D model, superimposing an augmented representation of the respective remaining portion at the determined location in the image of the AR environment; and An enhanced representation of the corresponding remaining portion is presented to a user for use in a medical procedure.

11. The method of claim 9, wherein the medical procedure comprises inserting a physical catheter into the organ; and wherein the method further comprises: receiving image data identifying a 3-D model of a catheter; determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which a physical catheter is inserted into the organ; and An augmented representation of the 3-D model of the catheter is superimposed at an initial position in the image of the AR environment, wherein the initial position is updated based on the user moving the physical catheter to a different portion of the organ.

12. The method according to claim 11, further comprising: receiving electrical data indicative of electrical signals measured by the catheter in the organ, wherein the electrical data includes values ​​corresponding to coordinate data values ​​of coordinate data identifying coordinates of a particular portion of the 3-D model; and The enhanced representation of the electrical data is superimposed onto the enhanced representation of the particular portion of the 3-D model.

13. The method of claim 9, wherein the medical procedure comprises inserting a physical catheter into the organ; wherein the remote server streams the one or more parts of the 3-D model to the AR head mounted device and the second AR head mounted device; and wherein the method further comprises: receiving image data identifying a 3-D model of a catheter; determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which a physical catheter is inserted into the organ; superimposing an augmented representation of the 3-D model of the catheter at the initial position in the image of the AR environment; receiving positioning data identifying an updated position of the augmented representation of the 3-D model of the catheter, wherein the updated position is based on a user moving the augmented representation of the 3-D model of the catheter in an image of the AR environment displayed on the second AR device; and In an image of the AR environment displayed by the AR head mounted device, a position of the augmented representation of the 3-D model of the catheter is updated from an initial position to an updated position.

14. The method of claim 9, wherein the medical procedure comprises inserting a physical catheter into the organ; wherein the remote server streams one or more portions of the 3-D model to the AR head mounted device and a second AR head mounted device; and wherein the method further comprises: determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which a physical catheter is inserted into the organ; superimposing an augmented representation of the 3-D model of the catheter at the initial position in the image of the AR environment; determining an updated location to place the augmented representation of the 3-D model of the catheter in the image of the AR environment, wherein the updated location is based on the user moving the physical catheter to a different portion of the organ; updating, in an image of the AR environment displayed by the AR head mounted device, a position of the augmented representation of the 3-D model of the catheter from an initial position to an updated position; and Providing positioning data identifying the updated location to the remote server, so that the remote server provides the positioning data to the second AR head mounted device, so that the second AR head mounted device updates the augmented representation of the 3-D model of the catheter in an image of the AR environment displayed by the second AR head mounted device.

15. The method according to claim 9, further comprising: receiving instructions to manipulate the superimposed augmented representation of one or more portions of the 3-D model; and The augmented representation is manipulated by performing one or more of the following operations: resizing the augmented representation, adjusting the orientation of the display of the enhanced representation, or Adjusts the transparency of the enhanced representation.

16. A mapping system, comprising: An augmented reality (AR) head mounted device, configured to: receiving a series of data packets over a period of time, the data packets comprising coordinate data identifying coordinates of one or more portions of a three-dimensional (3-D) model of an organ, For each corresponding portion of the 3-D model, determining a location in the image of the AR environment at which to place an augmented representation of the corresponding portion of the 3-D model, for each respective portion of the 3-D model, superimposing the augmented representation of the portion of the 3-D model at the determined location in the image of the AR environment, presenting an enhanced representation of the corresponding portion to a user for use in a medical procedure without presenting an enhanced representation of the entire 3-D model; and A mapping device is configured to generate the 3-D model of the organ and provide the series of data packets to the AR head-mounted device.

17. The mapping system according to claim 16, wherein the AR head mounted device is further used for: continuing to receive a series of data packets, the data packets further comprising coordinate data identifying coordinates of one or more remaining portions of the 3-D model of the organ; For each respective remaining portion of the 3-D model, determining a location in the image of the AR environment to place an augmented representation of the respective remaining portion of the 3-D model; For each respective remaining portion of the 3-D model, superimposing an augmented representation of the respective remaining portion at the determined location in the image of the AR environment; and An enhanced representation of the corresponding remaining portion is presented to a user for use in a medical procedure.

18. The mapping system of claim 16, wherein the AR head mounted device is located at a first geographic location, and wherein the mapping system further comprises: a second AR headset at a second geographic location different from the first geographic location; and A remote server is configured to: receive the series of data packets from the mapping device and provide the series of data to the AR head-mounted device and the second AR head-mounted device, wherein for each corresponding part of the 3-D model received via the data stream, the second AR head-mounted device superimposes an enhanced representation of the corresponding part in the image of the AR environment displayed by the second hand-mounted device.

19. The mapping system of claim 18, wherein the medical procedure comprises inserting a physical catheter into the organ; and wherein the AR headset is further configured to: receiving image data identifying a 3-D model of a catheter; determining an initial position at which to place the augmented representation of the 3-D model of the catheter in the image of the AR environment, the initial position corresponding to the position at which the physical catheter is inserted into the organ; superimposing an augmented representation of the 3-D model of the catheter at an initial position in the image of the AR environment; receiving positioning data identifying an updated position of the augmented representation of the 3-D model of the catheter, wherein the updated position is based on a user moving the augmented representation of the 3-D model of the catheter in an image of the AR environment displayed on the second AR device; and In an image of the AR environment displayed by the AR head mounted device, a position of the augmented representation of the 3-D model of the catheter is updated from the initial position to an updated position.

20. The mapping system of claim 16, wherein the medical procedure comprises inserting a physical catheter into the organ; and wherein the AR headset is further configured to: determining an initial position in the image of the AR environment at which to place the augmented representation of the 3-D model of the catheter, the initial position corresponding to a position at which the physical catheter is inserted into the organ; superimposing an augmented representation of the 3-D model of the catheter at an initial position in the image of the AR environment; determining an updated location to place the augmented representation of the 3-D model of the catheter in the image of the AR environment, wherein the updated location is based on the user moving the physical catheter to a different portion of the organ; updating, in an image of the AR environment displayed by the AR head mounted device, a position of the augmented representation of the 3-D model of the catheter from an initial position to an updated position; and Providing positioning data identifying the updated location to the remote server, so that the remote server provides the positioning data to the second AR head mounted device, so that the second AR head mounted device updates the augmented representation of the 3-D model of the catheter in the image of the AR environment displayed by the second AR head mounted device.

Images