Planning and performing three-dimensional holographic interventional procedures using holographic guides

Abstract

Embodiments of the present disclosure may include a method (200) for planning and performing an interventional procedure on a patient, the method (200) comprising providing an augmented reality system (102), a tracked instrument (104), a first image acquisition system (108), a second image acquisition system (110), and a computer system (106) having a processor (118) and a memory (120). The tracked instrument (104) comprises a plurality of sensors (123 a), the first image acquisition system (108), and the computer system (106) for providing a tracked instrument dataset (132), the computer system (106) being in communication with the augmented reality system (102), the tracked instrument (104), the first image acquisition system (108), and the second image acquisition system (110). Embodiments may also include acquiring a first holographic image dataset (114) from the patient by the first image acquisition system (108). The embodiment may also include acquiring a second holographic image data set (116) from the patient by a second image acquisition system (110).

Description

[Technical Field]

[0001] [CROSS-REFERENCE TO RELATED APPLICATIONS] This application claims the benefit of U.S. Provisional Application No. 63 / 479,139, filed January 9, 2023, the entire disclosure of which is incorporated herein by reference.

[0002] The present technology relates to holographic augmented reality applications, and more particularly to medical applications using holographic augmented reality. [Background technology]

[0003] This section provides background information related to the present disclosure that is not necessarily prior art.

[0004] Image-guided surgery has become standard practice in many different surgeries. Image-guided surgery visually correlates intraoperative and preoperative data. The use of image-guided surgery has been shown to improve the safety and success rate of these procedures. However, there are many known challenges that can arise during image-guided surgery. For example, how intraoperative and preoperative data are presented to the surgeon can directly correlate to the surgeon's surgical performance. Typically, this information is displayed on a two-dimensional (2D) display positioned around the patient. However, this undesirably shifts the surgeon's focus from the patient to the 2D display. Additionally, the surgeon must constantly look at the 2D display during the procedure, which can place additional strain on the surgeon's neck.

[0005] Determining the optimal angle for instrument insertion during surgery can be challenging due to the way intraoperative and preoperative data are displayed. As previously mentioned, this data is typically displayed in 2D, requiring the surgeon to mentally interpret instrument position and trajectory based on the data displayed on the 2D display. This can lead to confusion and errors, as instrument position and trajectory do not translate properly between 2D and the three-dimensional (3D) representation of the patient's body, potentially resulting in undesirable outcomes.

[0006] Therefore, there is a continuing need for visualization, guidance, and navigation methods and systems for surgery involving holographic augmented reality that allow the practitioner to view the surgical data and the patient in the same field of view. Summary of the Invention

[0007] In accordance with the present disclosure, a method has surprisingly been discovered that provides visualization, guidance, and navigation for procedures with holographic augmented reality, allowing the practitioner to view the surgical data and the patient in the same field of view.

[0008] Embodiments of the present disclosure may include a method for planning and performing an interventional procedure on a patient. The method may include providing an augmented reality system, a tracked instrument, a first image acquisition system, a second image acquisition system, and a computer system with a processor and memory. The tracked instrument may have a plurality of sensors for providing a tracked instrument dataset. The computer system may be in communication with the augmented reality system, the tracked instrument, the first image acquisition system, and the second image acquisition system.

[0009] Embodiments may also include acquiring a first holographic image dataset from the patient with a first image acquisition system. Embodiments may also include acquiring a second holographic image dataset from the patient with a second image acquisition system. Embodiments may also include tracking, with a computer system, the tracked instrument using the plurality of sensors to provide the tracked instrument dataset.

[0010] Embodiments may also include registering the first holographic image data set, the second holographic image data set, and the tracked instrument data set with the patient by a computer system. Embodiments may also include rendering the first hologram, the second hologram, and the treatment zone hologram by an augmented reality system. Embodiments may include multiple holograms corresponding to multiple tracked instruments and treatment zones. Embodiments may also include calibrating the treatment zone hologram using the computer system.

[0011] Embodiments may also include rendering the guide hologram with an augmented reality system. Embodiments may also include a practitioner performing a procedure on the patient while viewing the patient, the first hologram, and the guide hologram using the augmented reality system. In some embodiments, the practitioner may use the guide hologram and the augmented reality system to position tracking instruments during the procedure. The planned set of holographic needle guides and positioning of the tracking instruments may be saved to a file for post-procedure review and data mining.

[0012] Augmented reality systems can assist multiple operators (each wearing a headset) in dividing up tasks and providing guidance during a procedure, such as one operator performing the procedure while another adjusts ablation parameters such as time and duration for each applicator.

[0013] Further areas of applicability will become apparent from the description provided herein. The description and specific examples in this summary are for purposes of illustration only and are not intended to limit the scope of the present disclosure. [Brief explanation of the drawings]

[0014] The drawings described herein are for purposes of illustrating selected embodiments only, not all possible embodiments, and are not intended to limit the scope of the present disclosure. [Figure 1] FIG. 1 is a block diagram illustrating a system according to an embodiment of the present disclosure. [Figure 2A] 1 is a flowchart illustrating a method for a practitioner to plan and perform an interventional procedure on a patient, according to an embodiment of the present disclosure. [Figure 2B] 1 is a flowchart illustrating a method for a practitioner to plan and perform an interventional procedure on a patient, according to an embodiment of the present disclosure. [Figure 2C] 1 is a flowchart illustrating a method for a practitioner to plan and perform an interventional procedure on a patient, according to an embodiment of the present disclosure. [Figure 2D] 1 is a flowchart illustrating a method for a practitioner to plan and perform an interventional procedure on a patient, according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a system schematic diagram showing an augmented reality system, a computer, an image acquisition system, a hologram, and a patient. [Figure 4] 10 is an image illustrating a further use of the system, including the use of a guide hologram, according to an embodiment of the present disclosure. [Figure 5] 10 is an image illustrating a further use of a system including multiple guide holograms, according to an embodiment of the present disclosure. [Figure 6] 10 is an image showing the ablation parameter settings utilized by the practitioner. DETAILED DESCRIPTION OF THE INVENTION

[0015] The following description of the technology is merely illustrative of the subject matter, manufacture, and use of one or more inventions and is not intended to limit the scope, application, or uses of any particular invention(s) claimed in this application or any other application that may be filed claiming priority to this application, or any patent based thereon. With respect to the disclosed methods, the order of steps presented is exemplary in nature, unless otherwise specified, and thus the order of steps may be varied in various embodiments, including instances where certain steps are performed simultaneously.

[0016] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0017] As used herein, the singular form of a term indicates the presence of "at least one" of an item. Where possible, a plurality of such items may be present. Unless expressly stated otherwise, in describing the broadest scope of the present technology, all numerical values ​​herein are understood to be modified by the word "about" and all geometric and spatial descriptions are understood to be modified by the word "substantially." "About" applied to a numerical value indicates that some imprecision in the value is permitted by calculation or measurement (e.g., approaching the precision of the value to some extent, approximating the value, or reasonably close to the value, approximately). If for any reason the imprecision provided by "about" and / or "substantially" is not understood in this ordinary sense in the art, then "about" and / or "substantially," as used herein, will at least account for the variation that can result from ordinary methods of measuring or using such parameters.

[0018] All documents cited in this detailed description, including patents, patent applications, and scientific literature, are incorporated herein by reference unless expressly stated otherwise. In the event of a conflict or ambiguity between a document incorporated by reference and this detailed description, this detailed description will control.

[0019] The open-ended term "comprising" is used herein to describe and claim embodiments of the present technology as synonymous with non-limiting terms such as including, containing, and having; however, embodiments can also be described using more restrictive terms such as "consisting of" or "consisting essentially of." Thus, for any embodiment reciting a material, component, or process step, the present technology also specifically includes embodiments consisting of, or consisting essentially of, such material, component, or process step, excluding additional materials, components, or processes (if consisting of), or excluding additional materials, components, or processes (if consisting essentially of), that affect a critical characteristic of the embodiment. However, such additional materials, components, or processes are not expressly excluded herein. For example, a recitation of a process reciting elements A, B, and C specifically contemplates embodiments consisting of, or consisting essentially of, A, B, and C, excluding element D, which may be recited in the art, even if element D is not explicitly described as excluded herein.

[0020] The disclosure of ranges referred to herein includes the endpoints, unless otherwise specified, and includes all different values ​​and further subdivided ranges throughout the range. Thus, for example, a range "from A to B" or "from about A to about B" includes A and B. The disclosure of a value and range of values ​​for a particular parameter (amount, weight percent, etc.) does not exclude other values ​​and ranges of values ​​useful herein. It is contemplated that two or more specific exemplary values ​​for a parameter may define the endpoints of a range of values ​​that may be claimed for that parameter. For example, if parameter X is exemplified herein as having a value A and also as having a value Z, it is contemplated that parameter X may have a range of values ​​from approximately A to about Z. Similarly, the disclosure of two or more ranges of values ​​for a parameter (whether such ranges are nested, overlapping, or separate) is contemplated to encompass all possible combinations of value ranges that may be claimed using the endpoints of the disclosed ranges. For example, if a parameter X is exemplified herein as having a value in the range of 1 to 10, or 2 to 9, or 3 to 8, it is also contemplated that the parameter X may have other ranges of values, including 1 to 9, 1 to 8, 1 to 3, 1 to 2, 2 to 10, 2 to 8, 2 to 3, 3 to 10, 3 to 9, etc.

[0021] When an element or layer is described as being "on," "engaged," "connected," or "coupled" to another element or layer, the element or layer may be directly resting on, engaging, connected, or coupled to the other element or layer, or intervening elements or layers may be present. Conversely, when an element is described as being "directly on," "directly engaged," "directly connected," or "directly coupled" to another element or layer, there may be no intervening elements or layers. Other terms used to describe relationships between elements should be interpreted similarly (e.g., "between" and "directly between," "adjacent" and "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0022] Terms such as "first," "second," and "third" may be used herein to describe various elements, components, regions, layers, and / or sections; however, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another region, layer, or section. Terms such as "first," "second," and other numerical terms used herein do not imply any order or sequence unless clearly indicated by context. Thus, a first element, component, region, layer, or section described below may be referred to as a second element, component, region, layer, or section without departing from the teachings of the exemplary embodiments.

[0023] Spatially relative terms such as "inside," "outside," "below," "beneath," "lower," "below," "upper," and "top" may be used herein for ease of description to describe the relationship of one element or feature to another, as shown in the figures. Spatially relative terms may be intended to encompass different orientations of the device during use or operation in addition to the orientation shown in the figures. For example, if a device in the figures is turned upside down, elements described as "beneath" or "beneath" another element or feature would be oriented "above" that other element or feature. Thus, the exemplary term "below" may encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at another orientation), and the spatially relative descriptions used herein should be interpreted accordingly.

[0024] As used herein, the term "transdermal" refers to something made, performed, or carried out through the skin.

[0025] As used herein, the term "percutaneous medical procedure" refers to accessing internal organs or tissues by needle puncture through the skin, rather than through an open approach (usually with a scalpel) in which the internal organs or tissues are exposed.

[0026] As used herein, the term "non-vascular" when used in conjunction with "percutaneous medical procedure" refers to a medical procedure performed on any part of a subject's body other than the vascular system that is accessed percutaneously. Examples of percutaneous medical procedures include biopsies, tissue ablation, cryotherapy, brachytherapy, endovascular procedures, drainage procedures, orthopedic procedures, pain management procedures, vertebroplasty, pedicle / screw placement procedures, guidewire placement procedures, sacroiliac joint fusion procedures, training procedures, etc.

[0027] As used herein, the term "interventional device" or "tracked instrument" refers to a medical instrument used during a non-vascular percutaneous medical procedure.

[0028] As used herein, the term "tracking system" refers to a system that observes one or more objects in motion and provides a timely, ordered sequence of tracking data (e.g., position data, orientation data, etc.) in a tracking coordinate system for further processing. For example, a tracking system may be an electromagnetic tracking system that can observe an interventional device equipped with sensor coils as it moves through a patient's body.

[0029] As used herein, the term "tracking data" refers to information recorded by a tracking system relating to the observation of one or more objects in motion.

[0030] As used herein, the term "tracking coordinate system" refers to a three-dimensional Cartesian coordinate system that uses one or more numerical values ​​to determine the location of points or other geometric elements specific to a particular tracking system. For example, the tracking coordinate system may be rotated, scaled, etc. from a standard 3D Cartesian coordinate system. By way of non-limiting example, additional coordinate systems may be utilized, such as spherical coordinate systems, cylindrical coordinate systems, ellipsoidal coordinate systems, prolate spheroidal coordinate systems, oblate spheroidal coordinate systems, and quaternion coordinate systems.

[0031] As used herein, the terms "head-mounted device" or "headset" or "HMD" refer to a display device configured to be worn on the head and having one or more display optics (including lenses) in front of one or more eyes. These terms are sometimes more generally referred to as "augmented reality systems," although it can be understood that the term "augmented reality systems" is not limited to display devices configured to be worn on the head. In some cases, the head-mounted device may also include non-transitory memory and a processing unit. An example of a suitable head-mounted device is the Microsoft HoloLens®.

[0032] As used herein, terms such as "imaging system," "image acquisition device," and "image acquisition system" refer to technology that creates a visual representation of the inside of a patient's body. For example, imaging systems include computed tomography (CT) systems, fluoroscopy systems, positron emission tomography (PET), magnetic resonance imaging (MRI) systems, ultrasound (US) systems including contrast agents and color flow Doppler, and the like.

[0033] As used herein, the terms "coordinate system" or "augmented reality system coordinate system" refer to a 3D Cartesian coordinate system that uses one or more numerical values ​​to determine the location of points or other geometric elements specific to a particular augmented reality or image capture system. For example, a 3D point in the headset coordinate system may be translated, rotated, scaled, etc. from a standard 3D Cartesian coordinate system.

[0034] As used herein, the terms "image data" or "image dataset" or "imaging data" refer to information recorded in 3D by an imaging system related to observing the interior of a patient's body. For example, "image data" or "image dataset" may include processed two-dimensional or three-dimensional images or models, such as cross-sectional images, represented by data formatted according to, for example, the Digital Imaging and Communications in Medicine (DICOM) standard or other relevant imaging standards.

[0035] As used herein, the terms "imaging coordinate system" or "image acquisition system coordinate system" refer to a 3D Cartesian coordinate system that uses one or more numerical values ​​to determine the location of points or other geometric elements specific to a particular imaging system. For example, 3D points and vectors in the imaging coordinate system can be translated, rotated, scaled, etc., into the 3D Cartesian coordinate system of an augmented reality system (head-mounted display).

[0036] As used herein, the terms "hologram," "holographic," "holographic projection," or "holographic representation" refer to a computer-generated image that is stereoscopically projected through the lenses of a headset. Generally, holograms may be synthetically generated (as in augmented reality (AR)) and are not physical entities.

[0037] As used herein, the term "physical" refers to something real. Something physical is not holographic (or computer-generated).

[0038] As used herein, the terms "two-dimensional" or "2D" refer to something that is expressed in two physical dimensions.

[0039] As used herein, the terms "three-dimensional" or "3D" refer to something expressed in three physical dimensions. Elements of "4D" (e.g., 3D plus the dimensions of time and / or motion) are included in the definition of three-dimensional or 3D.

[0040] As used herein, the term "integrated" may mean that two things are linked or coordinated. For example, a coil sensor may be integrated with an interventional device.

[0041] As used herein, the term "real-time" refers to the actual time that a process or event occurs. In other words, a real-time event is executed live (within milliseconds so that results are available as immediate feedback). For example, a real-time event may be represented within 100 milliseconds of the event occurring.

[0042] As used herein, the terms "subject" and "patient" are used interchangeably and refer to any vertebrate animal.

[0043] As used herein, the term "spatial registration" refers to the step of transforming virtual representations of the holographic guide, applicator, and tracking device, including the ultrasound image stream, and additional body image data for mutual registration and correspondence of said virtual devices and image data in the coordinate system of the head-mounted display, resulting in a volumetric holographic projection display of images and information relative to the physical patient's body during treatment, as further described, for example, in U.S. Patent Application Publication No. 2018 / 0303563 to West et al., and commonly owned U.S. Patent Application No. 17 / 110,991 to Black et al. and U.S. Patent Application No. 17 / 117,841 to Martin III et al., the entire disclosures of which are incorporated herein by reference.

[0044] With reference to FIGS. 1-6 , methods for a practitioner to plan and perform an interventional procedure on a patient can employ various system configurations and various combinations of method steps. It can be appreciated that the holographic augmented reality visualization guidance system 100 of the present disclosure can be utilized to plan and perform an interventional procedure on a patient. The holographic augmented reality visualization guidance system 100 can be utilized in interventional procedures to identify predetermined or planned treatment regions. By way of non-limiting example, the holographic augmented reality visualization guidance system 100 can be utilized in the treatment of solid tumors and localized tumors. Treatments can include various types of treatments affecting various types of treatment regions. Examples of treatments include the delivery of various types of energy, including thermal energy, radiofrequency energy, and altered electromagnetic energy. Specific examples include energy (thermal and non-thermal) ablation, such as heating or freezing (e.g., cryoablation), irreversible electroporation, and pulsed wave ablation. Other treatments include the delivery of therapeutic substances or devices, such as active ingredients, chemotherapy, radioactive seeds, and barrier materials.

[0045] As shown in the system diagrams of the present disclosure in FIGS. 1 and 3-5, a holographic augmented reality visualization and guidance system 100 for performing an interventional procedure on a patient can include an augmented reality system 102, a tracked instrument 104, a computer system 106, and a first image acquisition system 108. In certain examples, the holographic augmented reality visualization and guidance system 100 can further include a second image acquisition system 110, also shown in FIG. 1. Each of the augmented reality system 102, the tracked instrument 104, the first image acquisition system 108, and the second image acquisition system 110 can be in selective or persistent communication with the computer system 106, for example, via a computer network 112. Other suitable instruments, tools, equipment, subsystems, etc. for use in the present disclosure, including the holographic augmented reality visualization and guidance system 100, as well as other networking means, including wired and wireless communication means between components of the holographic augmented reality visualization and guidance system 100, can be employed by a skilled technician as needed.

[0046] The tracked instrument 104 may be a sensorized intervention device such that both the position and orientation of the tracked instrument 104 may be determined by the computer system 106. The system 100 may have multiple sensors 123, each of which may be in communication with or detectable by the computer system 106. In particular, the tracked instrument 104 may have instrument sensor 123a, each of which may be in communication with or detectable by the computer system 106. In particular embodiments, the augmented reality system 102 may include multiple sensors 123, such as augmented reality sensor 123b. In particular examples, the sensor 123 may be part of and / or part of an electromagnetic (EM) tracking system used by the computer system 106 to detect the position and orientation of the physically tracked instrument 104. For example, the sensor may include one or more sensor coils. The computer system 106 may detect the one or more sensor coils and provide tracking data (e.g., six degrees of freedom) in response to the detection. For example, the tracking data may include real-time 3D position data and real-time 3D orientation data. The tracking system of the computer system 106 may also be capable of detecting coil sensors not located on the physical interventional device (e.g., located on fiducial markers or other imaging targets).

[0047] Other suitable tracking systems are specifically contemplated, such as optical tracking systems used in combination with the augmented reality system 102 and the computer system 106. Embodiments in which the tracked device 104 can communicate with the augmented reality system 102 and the computer system 106 via wireless communication or a wired connection are further contemplated. Those skilled in the art will also understand that different types of position sensors 123 can be used, as desired. Furthermore, those skilled in the art will understand that different types of tracking systems can be utilized, as desired, within the scope of the present disclosure.

[0048] 1 , the first image acquisition system 108 can be configured to acquire a first image dataset 114 from a patient. In particular, the first image acquisition system 108 can be configured to acquire the first holographic image dataset 114 from the patient preoperatively. In certain embodiments, the first image acquisition system 108 is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device. Other types of equipment suitable for the first image acquisition system 108 can also be used, if desired. In another embodiment, the first image acquisition system 108 can be configured to acquire the first holographic image dataset 114 from the patient in a perioperative feedback loop as a means of intraoperative real-time improvement.

[0049] The first holographic image dataset 114 may include information about the patient acquired prior to the medical procedure using, for example, the first image acquisition system 108, as well as data acquired, processed, and / or annotated from various sources. Embodiments of the first holographic image dataset include various images, composite images, annotated images, or portions of the patient's anatomy. Specific, non-limiting examples of the first holographic image dataset include still images or recordings from a transesophageal echocardiogram, a transabdominal echocardiogram, a transthoracic echocardiogram, a computed tomography (CT) scan, a magnetic resonance imaging (MRI) scan, or an X-ray. It should be noted that the pre-operative data may include information from other diagnostic medical procedures, imaging modalities, and modeling systems, as appropriate.

[0050] Similarly, the second image acquisition system 110 is configured to acquire a second image data set 116 from the patient. In particular, the second image acquisition system 110 can be configured to acquire the second holographic image data set 116 from the patient intraoperatively, in particular in real time while a procedure is being performed. In certain embodiments, the second image acquisition system 110 can be an ultrasound imaging device. Other types of equipment and modalities suitable for the second image acquisition system 110 can also be used, if desired.

[0051] Although the use of both the first image acquisition system 108 and the second image acquisition system 110 is shown and described herein, embodiments in which only one of the first image acquisition system 108 or the second image acquisition system 110 is used are also considered to be within the scope of the present disclosure.

[0052] 1 , the computer system 106 of the present disclosure may have at least one processor 118. The one or more processors 118 may perform functions related to the operation of the holographic augmented reality visualization and guidance system 100. The one or more processors 118 may be any type of general-purpose or special-purpose processor. In some cases, multiple processors 118 may be utilized according to other embodiments. Indeed, the one or more processors 118 may include, by way of non-limiting example, one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture.

[0053] The computer system 106 may have at least one memory 120 on which tangible, non-transitory machine-readable instructions 122 are stored. The memory 120 may be one or more memories and may be any type of memory suitable for the local application environment and may be implemented using any suitable volatile or non-volatile data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. For example, the memory 120 may be comprised of any combination of random access memory (RAM), read-only memory (ROM), static storage devices such as magnetic or optical disks, hard disk drives (HDDs), or any other type of non-transitory machine-readable or computer-readable medium. The instructions stored in the memory 120 may include program instructions or computer program code that, when executed by the one or more processors 118, enable the holographic augmented reality visualization and guidance system 100 to perform the tasks described herein.

[0054] The machine-readable instructions 122 stored in memory 120 may include modules. These modules may be implemented as one or more of functional logic, hardware logic, electronic circuitry, software modules, etc. These modules may include one or more of an augmented reality system module, an image acquisition module, an instrument tracking module, an image dataset registration module, a hologram rendering module, an image registration module, a trajectory hologram rendering module, and / or other suitable modules, as desired.

[0055] The computer system 106 may be in communication with the augmented reality system 102, the tracked instrument 104, the first image acquisition system 108, and the second image acquisition system 110, for example, via a network 112, and may be configured with machine-readable instructions 122 to operate according to the method 200 described herein. The computer system 106 may be separate and remote from the augmented reality system 102, or may be provided as an integral unit with the augmented reality system 102, if desired.

[0056] The network 112 of the holographic augmented reality visualization guidance system 100 may include, but is not limited to, a wireless access network such as LTE or 5G, a local area network (LAN), a wide area network (WAN) such as the Internet, or a wireless LAN (WLAN). However, this is not intended to be limiting, and it can be understood that the scope of the present disclosure includes implementations in which one or more computing platforms of the holographic augmented reality visualization guidance system 100 are operably linked via other communication couplings. The one or more computing platforms may be configured to communicate with a network environment via wireless or wired connections. Furthermore, in one embodiment, the one or more computing platforms may be configured to communicate directly with each other via wireless or wired connections. Examples of the one or more computing platforms include, but are not limited to, smartphones, wearable devices, tablets, laptop computers, desktop computers, Internet of Things (IoT) devices, or other mobile or fixed devices such as standalone servers, network servers, server arrays, etc.

[0057] The augmented reality system 102 may be configured to render multiple holograms according to the method 200 of the present disclosure. In particular, the augmented reality system 102 may be a mixed reality (MR) display, such as MR smart glasses or a MR head-mounted display. Non-limiting examples of the augmented reality system 102 include Magic Leap One® and Microsoft HoloLens®. It may be understood that other types of MR displays may be used for the augmented reality system 102, as long as they are capable of overlaying computer-generated images onto real-world objects. Furthermore, while the augmented reality system 102 is primarily described herein as a head-mounted display, it may be understood that other types of displays that are not head-mounted but are capable of generating and overlaying holograms onto a real-world view may also be used, if desired.

[0058] If the augmented reality system 102 does not include a computer system 106, the augmented reality system 102 may further include additional non-transitory memory and processing units (which may include one or more hardware processors) that assist in rendering or generating holograms. The augmented reality system 102 may also include a camera that records one or more images, one or more image generation components for generating / displaying a visualization of the hologram, and / or other visualization and / or recording elements. Recording may include tracking one or more steps or actions of a medical procedure, the movement of one or more surgical instruments, and the patient's anatomy (pre- and post-intervention) in three-dimensional space in real time.

[0059] In yet another example, the augmented reality system 102 may also include multiple position sensors 123b. The multiple position sensors 123b of the augmented reality system 102 may be configured to determine various position information for the augmented reality system 102, such as an approximate position in three-dimensional (3D) space, an orientation of the augmented reality system 102, angular velocity, acceleration, etc. In particular, it may be appreciated that the position sensors and the alignment methods described above may enable the holographic image to be displayed in precise alignment with the imaged anatomical structure within the practitioner's field of view during operation.

[0060] Non-limiting examples of the plurality of position sensors 123b include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. Furthermore, one skilled in the art will appreciate that different types and numbers of the plurality of position sensors 123b of the augmented reality system 102 can be employed depending, for example, on the procedure or situation in which the augmented reality system 102 is used.

[0061] The augmented reality system 102 may be configured to generate multiple holograms for the practitioner to view throughout the planning and execution of an interventional procedure. For example, as shown in FIG. 1 , the holograms generated by the augmented reality system 102 may include a first hologram 124, a second hologram 126, an ablation treatment zone hologram 128, and an applicator guide hologram 130. The first hologram 124 generated by the augmented reality system 102 may be based on a first holographic image dataset 114 from the patient. The second hologram 126 generated by the augmented reality system 102 may be generated based on the second holographic image dataset 116. The treatment zone hologram 128 may be visualized in relation to a tracked instrument dataset 132. The tracked instrument dataset 132 may be selected manually or automatically and stored in the memory 120 of the computer system 106, as further described herein. The applicator guide hologram 130 may be generated within the treatment area at an adjustable depth from the applicator tip, as further described herein.

[0062] It is understood that there may be multiple treatment zones, and therefore multiple treatment zone holograms 128. The multiple treatment zone holograms 128 may intersect. As a non-limiting example, the multiple treatment zones may be a Mickey Mouse-shaped treatment zone, where a spherical first treatment zone hologram, a spherical second treatment zone hologram, and a spherical third treatment zone hologram intersect and overlap to form a uniquely shaped polygon. Referring to the example above, the resulting polygon may include three different lobes obtained by simultaneously tracking three different types of instruments, or three different lobes obtained by tracking a single instrument multiple times with different settings and different trajectories. All of these lobes and trajectories are displayed in unique ways (e.g., color, texture, overlaid data values, heat map gradients of actual and predicted differences, etc.).

[0063] The ablation treatment zone hologram 128 can be generated using the augmented reality system 102 and the computer system 106. The treatment zone hologram 128 can allow the practitioner to visualize where the treatment will be performed by the tracked instrument 104. By simultaneously visualizing the first hologram 124 and the treatment zone hologram 128, the practitioner can visualize the predicted effect of the treatment or procedure on the patient's anatomy. The practitioner can adjust the predetermined treatment zone using the computer system 106. The augmented reality system 102 can then adjust the treatment zone hologram 128 accordingly, allowing the practitioner to visualize the effect of the treatment or procedure in real time. The augmented reality system 102 and the computer system 106 can then plan and generate a guide hologram 130 based on the treatment zone selected by the practitioner. The guide hologram 130 can assist the practitioner in 3D positioning of the tracked instrument 104 so that the predicted treatment is applied according to the treatment zone hologram 128. The guide hologram 130 may include at least one of a guide hub 134, a guide hoop 136, and a guide path 138, as shown in FIG. 5 . The guide hub 134 may provide an initial guide for the practitioner to initially align the tracked instrument 104 during a procedure. The guide hub 134, imprinted at a specific location, may also provide the practitioner with an indication of the depth to which the tracked instrument 104 should be inserted into the patient during a procedure. This allows the tracked instrument 104 to be positioned at the desired depth when aligned with the guide hub 134 during operation. The guide hoop 136 may provide the practitioner with the desired angle and further path guidance for inserting the tracked instrument 104 into the patient during a procedure. The guide path 138 may provide a more detailed and thorough path for the practitioner to follow with the tracked instrument 104 during a procedure, allowing the practitioner to assess the angle, position, and depth the tracked instrument 104 should follow during a procedure. Together, the guide hub 134, the guide hoop 136, and the guide path 138 can provide a trajectory for the practitioner to follow during the procedure.

[0064] Note that the practitioner can select parameters for performing ablation during the procedure, as shown in FIG. 6. The practitioner can vary the ablation time and ablation power for each generated guide hologram 130. Furthermore, the ablation can be represented on the hologram by varying the color, pattern, size of the ablation zone, or other visual cues or indications of time, power, etc. In certain embodiments, the ablation zone may exhibit transient elements as it grows. Those skilled in the art can select appropriate ablation times and ablation powers within the scope of this disclosure.

[0065] In certain embodiments, the holographic augmented reality visualization guidance system 100 may include multiple tracking instruments 104. Using multiple tracking instruments allows a practitioner or multiple practitioners to treat irregularly shaped and / or intersecting treatment areas sequentially or simultaneously. Using multiple tracking instruments 104 allows a practitioner to fully realize a predetermined 3D margin area, especially when there are variations in tumor characteristics, healthy tissue, structures, voids, or fluids from hydrodissection. When using multiple tracking instruments, complex interactions of position, power, time, and the like can occur when the treatment areas of each tracking instrument 104 overlap. Advantageously, using multiple tracking instruments 104 allows for the avoidance of adjacent structures by utilizing multiple simultaneous trajectories. A practitioner can select the appropriate number of tracking instruments 104 appropriate for a given procedure as needed.

[0066] In addition to rendering or generating various holograms, the augmented reality system 102 may be further configured to show the practitioner multiple surgical information or details. For example, the augmented reality system 102 may project multiple surgical information aligned with real-world objects, such as a patient. The surgical information may include, for example, real-time navigation instructions or guidance regarding the trajectory to be used. It should be appreciated that the augmented reality system 102 may project multiple surgical information onto various real-world objects, such as the tracked instrument 104, as well as onto the various rendered holograms, if desired.

[0067] Desirably, this generation of surgical information or details allows the practitioner to view the patient and multiple surgical information simultaneously in the same field of view. Additionally, the generation of surgical information or details along with various holograms allows the physician to plan, size, or pre-orient the tracked instrument 104 for the procedure and then align the tracked instrument 104 to a particular planned trajectory.

[0068] The computer system 106 can be in communication with the augmented reality system 102 and the tracked instrument 104. The computer system 106 can be configured to store and generate the plurality of surgical information through manual intervention by a physician or other medical professional, or automatically based on machine-readable instructions 122 encoded in the memory 120. For example, the plurality of surgical information can be generated within the augmented reality system 102 in response to the position or orientation determined by sensors of the tracked instrument 104, by algorithms, artificial intelligence (AI) protocols, or other data or thresholds input by the physician, etc.

[0069] Additionally, the computer system 106 may be configured to allow the practitioner to selectively adjust the plurality of pieces of operational information in real time. Additionally, the practitioner may determine which of the plurality of pieces of operational data is actively displayed to the practitioner. It can be understood that other settings and attributes of the plurality of pieces of operational information may be adjusted by the practitioner in real time within the scope of this disclosure.

[0070] In particular, it may be appreciated that the augmented reality system 102 of the present disclosure advantageously enables a practitioner to perform a method 200 for performing an interventional procedure on a patient while viewing the patient and the holograms described herein using the augmented reality system 102. Similarly, the practitioner may advantageously use the augmented reality system 102 for at least one of visualizing, guiding, and navigating a tracked instrument 104 during the interventional procedure, as further described herein with respect to the method 200 of the present disclosure.

[0071] Referring to Figure 3, an overview of a holographic augmented reality visualization and guidance system 100 is shown. During operation, a practitioner may view a patient and a first hologram 124 via the augmented reality system 102. As described herein, the augmented reality system 102 and the tracked instrument 104 may utilize multiple sensors 123. Additionally, the practitioner may utilize a second image acquisition system 110 during surgery to collect a second holographic image dataset 116 intraoperatively.

[0072] 2A-2D illustrate an exemplary flow diagram of a method 200 according to one embodiment of the present disclosure. The method 200 may include step 202 of providing a holographic augmented reality visualization and guidance system 100 as described herein. In step 204, the method 200 may include acquiring a first holographic image dataset 114 from a patient via a first image acquisition system 108, and optionally, in step 206, acquiring a second holographic image dataset 116 from the patient via a second image acquisition system 110. In a particular example, the first holographic image dataset 114 may include an anatomical region of the patient to be treated. The method 200 may include step 208 of tracking, by a computer system 106, a tracked instrument 104 using multiple sensors 132a to provide a tracked instrument dataset 132. The method 200 may include step 210 of co-registering, by the computer system 106, the first holographic image dataset 114, the second image dataset 116, and the tracked instrument dataset 132 for projection onto the physical patient.

[0073] In step 212, the method 200 can include rendering, by the augmented reality system 102, a first hologram 124 based on the first holographic image dataset 114 from the patient, a second hologram 126 based on the second holographic image dataset 116, and a treatment zone hologram 128 based on the tracked instrument dataset 132 for viewing by the practitioner.

[0074] The method 200 may include step 214 of collecting, by the computer system 106, operational information or details for provision to the practitioner. In step 216, the augmented reality system 102 may project the operational information onto a real-world object, such as a patient. The operational information may include, for example, real-time navigation instructions or guidance regarding the trajectory to be used. Note that the augmented reality system 102 may project the surgical information onto various real-world objects, such as the tracked instrument 104, and onto various rendered holograms as needed.

[0075] The method may include simultaneously displaying 218 the first hologram 124 and the treatment zone hologram 128. In this way, the practitioner may visualize the predicted effect of the treatment or procedure on the patient's anatomy and, in step 220, assess the treatment area against reality.

[0076] The method may include step 222 of adjusting the treatment zone hologram 128 using a computer system. The practitioner may adjust the trajectory of the treatment zone hologram 128 relative to the first hologram 124. Once the practitioner is satisfied with the placement of the treatment zone hologram 128, the practitioner may stamp or lock the planned trajectory, depth, and predicted treatment area using voice commands, hand gestures, or other operator input provided to the augmented reality system. The method may include step 224 of rendering an applicator guide hologram 130 based on the selected treatment area, which may include parameters related to the predicted area (such as time and duration of treatment, if applicable). Depending on the shape and volume of the tumor to be treated, the practitioner may repeat steps 222 and 224 until the entire treatment area is covered by the treatment zone hologram 128.

[0077] The method 200 may include step 226 of simultaneously displaying the guide hologram 130, the first hologram 124, and the treatment zone hologram 128. As described above, the practitioner can visualize the predicted effect of the treatment or procedure on the patient's anatomy and determine whether adjustments to the guide hologram 130 are necessary. The method 200 may include step 228 of imprinting a holographic guide, displayed within the augmented reality system, with the tracked instrument. The operator may imprint multiple holographic applicator guides based on the trajectories of one or more tracked instruments. Each holographic guide provides a planned trajectory, including, but not limited to, a guide hub 134, a guide path 138, a guide hoop 136, etc., aligned with the tracked instrument when imprinting the holographic guide, as shown in FIG. 4 . Planning a holographic applicator guide aligned with the tracked live ultrasound and tomographic image datasets may facilitate avoidance of critical structures, such as blood vessels and nearby organs, during treatment. Placement of the holographic applicator guide can be facilitated by image-based holographic images aligned with the tracked instrument, e.g., image-based holographic images derived from pre-operative tomographic images of segmented structures or tumors, anatomical planes, or multi-planar images co-planar with the tracked ultrasound image. Three anatomical planes derived from the tomographic images can be updated with adjustments of the tracked instrument 104 to aid in placement of the planned ablation zone by aligning their intersection with the center point of the planned ablation zone, the tip of the tracked instrument, or other point of interest in head-mounted display coordinates.

[0078] The method 200 may then include step 230 of the practitioner performing an interventional procedure on the patient while viewing the patient, the first hologram 124, and the guide hologram 130 using the augmented reality system 102. In step 230, the practitioner uses the augmented reality system 102 to position the tracked instrument 104 during the procedure.

[0079] The method may include using a computer system to record 232 an element selected from the group consisting of a first holographic image data set, a second holographic image data set, a tracked instrument data set, a guide hologram, and combinations thereof. Recording portions of a procedure with any combination of data sets may allow a practitioner to learn from the procedure and implement changes in real time or for future procedures.

[0080] The method may include generating feedback based on the first holographic image dataset, the second holographic image dataset, and the tracked instrument dataset by the augmented reality system in step 232 and step 234. Real-time feedback can assist the practitioner in implementing procedural changes during the procedure, thereby providing a more efficient and effective procedure. Visual, auditory, or numerical holographic feedback may be provided to assist in aligning the tracked instruments with their respective holographic applicator guides. The correspondence between the tracked instruments and the holographic guides may be indicated using the color, texture, tag, etc., of the holographic ablation zone. The holographic ablation zone of the tracked instrument may be referenced to the tip of the tracked instrument. The tracked instrument may be advanced to the planned depth so that the holographic ablation zones of the tracked guide and the imprinted guide coincide. The planned depth of the holographic applicator guide may be adjusted to intersect with the tracking ultrasound image at an off-plan ultrasound transducer position, and then adjusted proximally or distally using voice commands from the augmented reality system or other operator input methods associated with the augmented reality system for in-plane or out-of-plane ultrasound procedures. After planning a set of corresponding holographic applicator guides, multiple separate tracking instruments can be inserted and positioned as a set, so that the union of the planned ablation zones matches the tumor or target tissue, which is important when the tumor does not match the target tissue.

[0081] After treatment, additional imaging is often performed using contrast agents to monitor the effectiveness of treatment. The updated images, used to monitor the actual treatment, are loaded into a computer system and registered with the reference and tracked ultrasound images for visualization. An additional holographic applicator guide followed by a tracked instrument can then be used to treat the tumor with a sufficient 3D margin (typically around 5 mm) and achieve the 3D visualization endpoint. Methods that enable visualization of overlapping holograms based on the fused image data and predicted ablation area include using cross-contours of tomographic image segmentation results on live ultrasound images, using wireframe shading models to visualize the 3D predicted ablation area, and toggling the holograms on and off using voice commands or other AR / UX methods such as hand gestures.

[0082] In a further embodiment, the plurality of surgical information includes fused preoperative and intraoperative data. The preoperative and intraoperative data are fused to synergistically combine the advantages of each imaging modality. In some cases, after the fusion is performed by the computer system 106, the fusion can be fine-tuned manually, using at least one of a plurality of algorithms described in the machine-readable instructions 122, or via artificial intelligence (AI), for example.

[0083] While several exemplary embodiments and details have been shown to illustrate the present invention, it will be apparent to those skilled in the art that various modifications may be made without departing from the scope of the present disclosure, which is further set forth in the following appended claims.

[0084] The exemplary embodiments are provided so that this disclosure will be thorough and will fully convey its scope to those skilled in the art. Numerous specific details are described, such as examples of specific components, devices, and methods, to provide a thorough understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that the embodiments may be embodied in various forms, and that none of these should be construed as limiting the scope of the present disclosure. In some embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail. Equivalent changes, modifications, and variations of some embodiments, materials, compositions, and methods may be made within the scope of the present technology to achieve substantially similar results.

Claims

1. 1. A method for a practitioner to plan and perform an interventional procedure on a patient, comprising: providing a tracked instrument having a plurality of sensors configured to provide a tracked instrument dataset, an augmented reality system, a first image acquisition system, a second image acquisition system, and a computer system having a processor and memory, wherein the computer system is in communication with the augmented reality system, the tracked instrument, the first image acquisition system, and the second image acquisition system; acquiring a first holographic image data set from the patient with the first image acquisition system; acquiring a second holographic image data set from the patient with the second image acquisition system; tracking, with the computer system, the tracked instrument using the plurality of sensors to provide the tracked instrument data set; registering, by the computer system, the first holographic image data set, the second holographic image data set, and the tracked instrument data set to the patient; rendering, by the augmented reality system, a first hologram from the first hologram image dataset, a second hologram from the second hologram image dataset, and a treatment zone hologram from the tracked instrument dataset; adjusting the treatment zone hologram using the computer system; rendering a guide hologram with the augmented reality system; performing an interventional procedure on the patient while the practitioner views the patient, the first hologram, and the guide hologram using the augmented reality system; Including, 10. A method for planning and performing an interventional procedure, wherein the practitioner uses the guide hologram and the augmented reality system to position the tracked instrument during the interventional procedure.

2. In claim 1, 10. A method for planning and performing an interventional procedure, wherein adjusting the treatment zone hologram using the computer system and rendering the guide hologram by the augmented reality system is performed until the entire treatment area is covered with the treatment zone hologram.

3. In claim 1, 10. A method for planning and performing an interventional procedure, wherein the first image acquisition system is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device.

4. In claim 2, 10. A method for planning and performing an interventional procedure, wherein the first holographic image data set from the patient is one of a pre-operative image and an intra-operative feedback loop.

5. In claim 1, 10. A method for planning and performing an interventional procedure, wherein the first holographic image data set from the patient is pre-operative.

6. In claim 1, 10. A method for planning and performing an interventional procedure, wherein the second image acquisition system is an ultrasound device.

7. In claim 1, 1. A method for planning and performing an interventional procedure, the method comprising the use of multiple tracked instruments having multiple treatment zone holograms.

8. In claim 1, 10. A method for planning and performing an interventional procedure, further comprising the step of evaluating the treatment zone hologram against a real-world condition.

9. In claim 1, 10. The method for planning and performing an interventional procedure, further comprising simultaneously displaying the first hologram and the treatment zone hologram using the augmented reality system.

10. In claim 9, 10. A method for planning and performing an interventional procedure, wherein the guide hologram is displayed simultaneously with the first hologram and the treatment zone hologram.

11. In claim 1, 10. The method for planning and performing an interventional procedure, further comprising the step of collecting operational information by the computer system.

12. In claim 11, 10. A method for planning and performing an interventional procedure, wherein the operational information includes real-time navigation instructions.

13. In claim 1, 10. The method for planning and performing an interventional procedure, further comprising projecting, by the augmented reality system, operational information onto the tracked instrument.

14. In claim 1, 10. The method for planning and performing an interventional procedure, further comprising the step of the practitioner using the tracked instrument to imprint the guide hologram displayed in the augmented reality system.

15. In claim 14, 10. A method for planning and performing an interventional procedure, wherein the guide hologram includes at least one of a hub, a guide path, and a hoop.

16. In claim 1, The method further includes generating, by the augmented reality system, feedback based on the first holographic image dataset, the second holographic image dataset, and the tracked instrument dataset.

17. In claim 16, 10. A method for planning and performing an interventional procedure, wherein the feedback comprises at least one of visual feedback, auditory feedback, and numerical holographic feedback.

18. In claim 1, 10. The method for planning and performing an interventional procedure, further comprising using the computer system to record elements selected from the group consisting of the first holographic image dataset, the second holographic image dataset, the tracked instrument dataset, the guide hologram, and combinations thereof.

19. In claim 1, The guide hologram is positioned within the treatment region a predetermined distance from the tracked instrument.

20. 1. A system for a practitioner to plan and perform an interventional procedure on a patient, comprising: an augmented reality system; a tracked instrument having a plurality of sensors; a first image acquisition system configured to acquire a first holographic image data set from the patient; a second image acquisition system configured to acquire a second holographic image data set from the patient; and a computer system having a processor and memory in communication with the augmented reality system, the tracked instrument, the first image acquisition system, and the second image acquisition system, and configured with machine readable instructions to: track the tracked instrument using the plurality of sensors; provide the tracked instrument dataset; and register the patient with the first holographic image dataset, the second holographic image dataset, and the tracked instrument dataset; and the augmented reality system is configured to render the first hologram, the second hologram, the treatment zone hologram, and the guide hologram for viewing by the practitioner; The system for planning and performing an interventional procedure, wherein the first hologram and the guide hologram are displayed to the practitioner during the interventional procedure on the patient using the augmented reality system, and the guide hologram and the augmented reality system are configured to be used by the practitioner to position the tracked instrument during the interventional procedure.

Images