Planning and performing three-dimensional holographic interventional procedures with holographic guidance

Abstract

Embodiments of the present disclosure may include a method (200) for planning and performing an interventional procedure on a patient, the method (200) comprising the steps of providing an augmented reality system (102), a tracking instrument (104), a first image acquisition system (108), a second image acquisition system (110), and a computer system (106) having a processor (118) and a memory (120), the tracking instrument (104) has a plurality of sensors (123a) to provide a tracking instrument data set (132), the first image acquisition system (108) and the computer system (106) are in communication with the augmented reality system (102), the tracking instrument (104), the first image acquisition system (108), and the second image acquisition system (110). Embodiments may also include acquiring, by a first image acquisition system (108), a first holographic image data set (114) from the patient. Embodiments may also include acquiring, by a second image acquisition system (110), a second holographic image data set (116) from the patient.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 479,139, filed January 9, 2023. The entire disclosure of the above application is incorporated herein by reference. Technical Field

[0003] The present technology relates to holographic augmented reality applications, and in particular to medical applications using holographic augmented reality. Background Art

[0004] This section provides background information related to the present disclosure which is not necessarily prior art.

[0005] Image-guided surgery has become standard practice for many different surgeries. Image-guided surgery intuitively links intraoperative data with preoperative data. The use of image-guided surgery has been shown to increase the safety and success rate of these surgeries. However, there are many known difficulties that may arise in image-guided surgery. For example, how the intraoperative and preoperative data are displayed to the physician may be directly related to the physician's surgical performance. Typically, this information is displayed on a two-dimensional (2D) display positioned around the patient. Undesirably, this diverts the physician's attention from the patient to the 2D display. In addition, since the physician must constantly look at the 2D display while performing the surgery, this may place additional strain on the physician's neck.

[0006] Determining the optimal angle for instrument insertion during surgery can also be difficult due to the way intraoperative and preoperative data is displayed. As previously mentioned, this data is typically displayed in 2D, meaning the physician must mentally translate the instrument's position and trajectory relative to the data on the 2D display. This can undesirably lead to confusion and errors because the instrument's position and trajectory do not translate well between 2D and the three-dimensional (3D) patient's body.

[0007] Therefore, for surgeries involving holographic augmented reality, there is a continuing need for visualization, guidance, and navigation methods and systems that allow physicians to observe both the operating data and the patient in the same field of view. Summary of the Invention

[0008] In accordance with the present disclosure, methods have surprisingly been discovered to provide visualization, guidance, and navigation for surgery involving holographic augmented reality, allowing the physician to view both the procedure data and the patient in the same field of view.

[0009] Embodiments of the present disclosure may include a method for planning and performing an interventional procedure on a patient. The method may include providing an augmented reality system, a tracking device, a first image acquisition system, a second image acquisition system, and a computer system having a processor and a memory. The tracking device may include multiple sensors to provide a tracking device dataset. The computer system may communicate with the augmented reality system, the tracking device, the first image acquisition system, and the second image acquisition system.

[0010] The embodiment may further include acquiring, by the first image acquisition system, a first holographic image dataset from the patient. The embodiment may further include acquiring, by the second image acquisition system, a second holographic image dataset from the patient. The embodiment may further include tracking, by the computer system, a tracking instrument using a plurality of sensors to provide a tracking instrument dataset.

[0011] Embodiments may also include registering the first holographic image dataset, the second holographic image dataset, and the tracking device dataset with the patient by a computer system. Embodiments may also include rendering the first hologram, the second hologram, and the treatment zone hologram by an augmented reality system. Embodiments may include multiple holograms corresponding to multiple tracking devices and treatment zones. Embodiments may also include adjusting the treatment zone hologram by a computer system.

[0012] Embodiments may also include rendering a guidance hologram using an augmented reality system. Embodiments may also include a physician performing surgery on a patient while viewing the patient, the first hologram, and the guidance hologram using an augmented reality system. In some embodiments, the physician may utilize the guidance hologram and the augmented reality system to place tracking instruments during surgery. A set of planned holographic needle guidance and tracking instrument placements may be stored in a file for subsequent review and data mining after surgery.

[0013] The augmented reality system can support more than one operator (each equipped with a headset) to divide tasks or provide guidance while performing a procedure, for example, one operator is performing the procedure while another operator is adjusting ablation parameters such as the timing and duration of each applicator.

[0014] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0016] Figure 1 is a block diagram illustrating a system according to an embodiment of the present disclosure;

[0017] Figure 2A-2D is a flowchart illustrating a method for a physician to plan and perform an interventional procedure on a patient according to an embodiment of the present disclosure;

[0018] Figure 3 is a system overview depicting the augmented reality system, computer, image acquisition system, hologram, and patient;

[0019] Figure 4 is an image depicting further use of the system including use of a guided hologram according to an embodiment of the present disclosure;

[0020] Figure 5 is an image depicting further use of a system including a plurality of guided holograms according to an embodiment of the present disclosure; and

[0021] Figure 6 is an image depicting the ablation parameter settings used by the physician. DETAILED DESCRIPTION

[0022] The following description of the techniques is merely an exemplary description of the subject matter, manufacture, and use of one or more inventions and is not intended to limit the scope, application, or use of any particular invention claimed in this application or in other applications claiming priority from this application or in patents issued therefrom. With respect to the disclosed methods, the order of steps presented is exemplary in nature and, unless otherwise disclosed, the order of steps may vary in various embodiments, including instances where certain steps may be performed simultaneously.

[0023] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0024] As used herein, the terms "a" and "an" mean that "at least one" of an item is present; where possible, a plurality of such items may be present. Unless expressly stated otherwise, all numerical values in this specification are to be understood as modified by the word "about," and all geometric and spatial descriptors are to be understood as modified by the word "substantially," when describing the broadest scope of the present technology. "About," when applied to a numerical value, indicates that the calculation or measurement allows for some slight imprecision in the value (the value has some approach to exactness; is approximately or reasonably close to the value; roughly). If, for some reason, the imprecision provided by "about" and / or "substantially" is not understood in this ordinary sense in the art, then "about" and / or "substantially" as used herein at least indicate the variation that would result from ordinary methods of measuring or using such parameters.

[0025] Unless otherwise expressly stated, all documents cited in this detailed description, including patents, patent applications, and scientific literature, are incorporated herein by reference. In the event of any conflict or ambiguity between the documents incorporated by reference and this detailed description, the detailed description shall prevail.

[0026] Although the open-ended term "comprising" is used herein as a synonym for non-limiting terms such as including, containing, or having to describe and claim embodiments of the present technology, more restrictive terms such as "consisting of" or "consisting essentially of" may also be used to describe embodiments. Thus, for any given embodiment that enumerates materials, components, or process steps, the present technology also specifically includes embodiments that consist of or consist essentially of such materials, components, or process steps, excluding additional materials, components, or processes (to constitute), and excluding additional materials, components, or processes (to constitute essentially) that affect important characteristics of the embodiment, even if such additional materials, components, or processes are not expressly enumerated in this application. For example, an enumeration of a process enumerating elements A, B, and C specifically contemplates embodiments consisting of and consisting essentially of A, B, and C, to the exclusion of element D, which may be enumerated in the art, even if element D is not expressly described as excluded herein.

[0027] As described herein, unless otherwise stated, disclosed ranges include endpoints and include all different values and further divided ranges within the entire range. Thus, for example, a range of "from A to B" or "from about A to about B" includes A and B. The disclosure of values and value ranges for specific parameters (such as amounts, weight percentages, etc.) does not exclude other values and value ranges useful herein. It is foreseeable that two or more specific example values for a given parameter can define the endpoints of the value range required for the parameter. For example, if parameter X is exemplified herein as having value A and also exemplified as having value Z, it is conceivable that parameter X can have a value range from about A to about Z. Similarly, it is conceivable that the disclosure of two or more value ranges for a parameter (regardless of whether such ranges are nested, overlapping, or different) includes all possible range combinations of values that may be claimed using the endpoints of the disclosed ranges. For example, if parameter X is illustrated herein as having values in the range of 1-10, or 2-9, or 3-8, it is also contemplated that parameter X may have other ranges of values, including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, and so on.

[0028] When an element or layer is referred to as being "on," "engaged to," "connected to," or "coupled to" another element or layer, it may be directly on, engaged to, connected to, or coupled to the other element or layer, or intervening elements or layers may be present. Conversely, when an element is referred to as being "directly on," "directly engaged to," "directly connected to," or "directly coupled to" another element or layer, there may be no intervening elements or layers. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0029] Although the terms first, second, third etc. can be used to describe various elements, components, regions, layers and / or parts in this article, these elements, components, regions, layers and / or parts should not be limited by these terms. These terms are only used to distinguish an element, component, region, layer or part from another region, layer or part. When such as "first", "second" and other numerical terms are used in this article, they do not mean order or sequence unless the context clearly indicates. Therefore, the first element, component, region, layer or part discussed below can be referred to as the second element, component, region, layer or part, without departing from the teaching of the exemplary embodiments.

[0030] For ease of description, spatially relative terms such as "inside," "outside," "below," "beneath," "lower," "above," "upper," etc. may be used herein to describe the relationship of one element or feature to another element or feature shown in the figures. In addition to the orientations shown in the figures, spatially relative terms may be intended to cover different orientations of the device in use or operation. For example, if the device in the figures is flipped, an element described as being "below" or "below" other elements or features will be oriented "above" the other elements or features. Thus, the example term "below" can include both above and below orientations. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatially relative descriptors used herein may be interpreted accordingly.

[0031] As used herein, the term "percutaneous" refers to something made, done, or achieved through the skin.

[0032] As used herein, the term "percutaneous medical procedure" refers to accessing internal organs or tissues through a needle puncture of the skin, rather than using an open approach (usually with a scalpel) that exposes the internal organs or tissues.

[0033] As used herein, the term "non-vascular," when used in conjunction with "percutaneous medical procedure," refers to a medical procedure performed on any part of a subject's body that is accessed percutaneously, as opposed to a blood vessel. Examples of percutaneous medical procedures may include biopsies, tissue ablations, cryotherapy procedures, brachytherapy procedures, endovascular procedures, drainage procedures, orthopedic procedures, pain management procedures, vertebroplasty procedures, pedicle / screw placement procedures, guidewire placement procedures, SI-arthrodesis procedures, training procedures, and the like.

[0034] As used herein, the term "interventional device" or "tracked instrument" refers to a medical device used in non-vascular percutaneous medical procedures.

[0035] As used herein, the term "tracking system" refers to something that is used to observe one or more moving objects and provide a timely and ordered sequence of tracking data (e.g., position data, orientation data, etc.) in a tracking coordinate system for further processing. For example, the tracking system can be an electromagnetic tracking system that can observe an interventional device equipped with sensor coils as the interventional device moves through a patient's body.

[0036] As used herein, the term "tracking data" refers to information recorded by a tracking system relating to observations of one or more moving objects.

[0037] As used herein, the term "tracking coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the location of a point or other geometric element that is unique to a particular tracking system. For example, the tracking coordinate system can be rotated, scaled, etc. from a standard 3D Cartesian coordinate system. As non-limiting examples, additional coordinate systems may be utilized, such as spherical, cylindrical, ellipsoidal, prolate spheroidal, oblate spheroidal, and quaternion coordinate systems.

[0038] As used herein, the terms "head mounted device" or "head mounted device" or "HMD" refer to a display device configured to be worn on the head, with one or more display optics (including lenses) in front of one or more eyes. These terms may even be referred to more generally as the term "augmented reality system", although it should be understood that the term "augmented reality system" is not limited to display devices configured to be worn on the head. In some cases, the head mounted device may also include non-transitory memory and a processing unit. An example of a suitable head mounted device is the Microsoft

[0039] As used herein, the terms "imaging system," "image acquisition device," "image acquisition system," and the like refer to technology that creates a visual representation of the interior of a patient's body. For example, the imaging system may be a computed tomography (CT) system, a fluoroscopy system, a positron emission tomography (PET) system, a magnetic resonance imaging (MRI) system, an ultrasound (US) system including contrast agents and color flow Doppler, and the like.

[0040] As used herein, the term "coordinate system" or "augmented reality system coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the position of points or other geometric elements that are unique to a particular augmented reality system or image acquisition system. For example, a 3D point in a headset's coordinate system can be translated, rotated, scaled, etc. relative to a standard 3D Cartesian coordinate system.

[0041] As used herein, the term "image data" or "image dataset" or "imaging data" refers to information related to observations of the interior of a patient's body recorded in 3D by an imaging system. For example, "image data" or "image dataset" may include processed two-dimensional or three-dimensional images or models, such as tomographic images, for example, represented by data formatted according to the Digital Imaging and Communications in Medicine (DICOM) standard or other related imaging standards.

[0042] As used herein, the term "imaging coordinate system" or "image acquisition system coordinate system" refers to a 3D Cartesian coordinate system that uses one or more numbers to determine the position of points or other geometric elements that are unique to a particular imaging system. For example, 3D points and vectors in the imaging coordinate system can be translated, rotated, scaled, etc. to the 3D Cartesian coordinate system of an augmented reality system (head-mounted display).

[0043] As used herein, the terms "hologram," "holographic," "holographic projection," or "holographic representation" refer to a computer-generated image that is stereoscopically projected through the lenses of a headset. Typically, a hologram can be synthetically generated (in augmented reality (AR)) and is not a physical entity.

[0044] As used herein, the term "physical" refers to something real. Physical things are not holographic (or not computer-generated).

[0045] As used herein, the term "two-dimensional" or "2D" refers to something that is represented in two physical dimensions.

[0046] As used herein, the term "three-dimensional" or "3D" refers to something represented in three physical dimensions. "4D" elements (e.g., 3D plus the dimensions of time and / or motion) would be encompassed by the definition of three-dimensional or 3D.

[0047] As used herein, the term "integrated" can refer to two things that are linked or coordinated. For example, a coil sensor can be integrated with an interventional device.

[0048] As used herein, the term "real-time" refers to the actual time at which a process or event occurs. In other words, a real-time event is completed in real time (within a few milliseconds so that the results can be provided immediately as feedback). For example, a real-time event can be represented as occurring within 100 milliseconds of the event.

[0049] As used herein, the terms "subject" and "patient" are used interchangeably to refer to any vertebrate organism.

[0050] As used herein, the term spatial “registration” refers to the steps of transforming a virtual representation of a tracking device (including a holographic guide, an applicator, and an ultrasound image stream) and attached body image data so that the virtual device and image data are aligned and correspond to each other in a head-mounted display coordinate system, thereby producing a stereoscopic holographic projection display of images and information relative to the physical patient body during surgery, as further described, for example, in U.S. patent application publication No. 2018 / 0303563 to West et al. and U.S. patent application serial No. 17 / 110,991 to Black et al. and U.S. patent application serial No. 17 / 117,841 to Martin III et al., both of which are co-owned by the applicants, the entire disclosures of which are incorporated herein by reference.

[0051] Now refer to Figures 1-6 , the way in which a physician plans and performs interventional procedures on a patient can employ various system configurations, and various combinations of method steps can be used. It should be understood that the holographic augmented reality visualization and guidance system 100 of the present disclosure can be used to plan and perform interventional procedures on a patient. The holographic augmented reality visualization and guidance system 100 can be used in interventional procedures, where a predetermined or planned treatment area can be identified. As a non-limiting example, the holographic augmented reality visualization and guidance system 100 can be used for the treatment of solid and localized tumors. Treatment can include various types of treatments that affect various types of treatment areas. Examples of treatments include delivering various types of energy, including variations in thermal energy, radiofrequency energy, and electromagnetic energy. Specific examples include energy (thermal and non-thermal) ablation; for example, heating or freezing (e.g., cryoablation), irreversible electroporation, and pulsed wave ablation. Other treatments include delivering therapeutic materials or devices, such as active pharmaceutical ingredients, chemotherapy, radiation particles, or barrier materials.

[0052] like Figure 1 and Figure 3-Figure 5As shown in the description of the system of the present disclosure, the holographic augmented reality visualization and guidance system 100 for performing an interventional procedure on a patient may include an augmented reality system 102, a tracking device 104, a computer system 106, and a first image acquisition system 108. In some examples, the holographic augmented reality visualization and guidance system 100 may also include a second image acquisition system 110, also in Figure 1 . Each of the augmented reality system 102, the tracking device 104, the first image acquisition system 108, and the second image acquisition system 110 can selectively or permanently communicate with the computer system 106, for example, via a computer network 112. As needed, those skilled in the art can also adopt other suitable devices, tools, equipment, subsystems, etc. for the holographic augmented reality visualization and guidance system 100, as well as other network devices including wired and wireless communication devices between the components of the holographic augmented reality visualization and guidance system 100.

[0053] The tracking device 104 can be an interventional device capable of sensing, such that the position and orientation of the tracking device 104 can be determined by the computer system 106. The system 100 can have a plurality of sensors 123, each of which can be in communication with or otherwise detectable by the computer system 106. Specifically, the tracking device 104 can have a device sensor 123a, each of which can be in communication with or otherwise detectable by the computer system 106. In certain embodiments, the augmented reality system 102 can include a plurality of sensors 123, such as augmented reality sensor 123b. In certain examples, the sensor 123 can be part of an electromagnetic (EM) tracking system that can be part of and / or used by the computer system 106 to detect the position and orientation of the physical tracking device 104. For example, the sensor can include one or more sensor coils. The computer system 106 can detect the one or more sensor coils and provide tracking data (e.g., with six degrees of freedom) in response to the detection. For example, the tracking data may include real-time 3D position data and real-time 3D orientation data.The tracking system of the computer system 106 may also detect coil sensors that are not located on the physical interventional device (eg, located on a fiducial marker or other imaging target).

[0054] Other suitable tracking systems, such as optical tracking systems for use in conjunction with the augmented reality system 102 and the computer system 106, are also specifically contemplated. Embodiments are also contemplated in which the tracking device 104 can communicate with the augmented reality system 102 and the computer system 106 via wireless transmission or via a wired connection. It should also be understood that different types of position sensors 123 may be employed as desired by one skilled in the art. Furthermore, different types of tracking systems may be utilized as desired by one skilled in the art within the scope of the present disclosure.

[0055] Reference again Figure 1 , the first image acquisition system 108 can be configured to acquire a first image data set 114 from the patient. Specifically, the first image acquisition system 108 can be configured to acquire a first holographic image data set 114 from the patient in a preoperative manner. In some embodiments, the first image acquisition system 108 is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device. Other suitable types of devices for the first image acquisition system 108 can also be used as needed. In another embodiment, the first image acquisition system 108 can be configured to acquire the first holographic image data set 114 from the patient through an intraoperative real-time feedback loop as a means of real-time intraoperative improvement.

[0056] The first holographic image dataset 114 may include information related to the patient obtained prior to the medical procedure, for example, using the first image acquisition system 108 and data obtained, processed, and / or annotated from various sources. Embodiments of the first holographic image dataset include various images, composite images, annotated images, or portions of the patient's anatomical region. Some non-limiting examples of the first holographic image dataset include recordings or static images from a transesophageal echocardiogram, a transabdominal echocardiogram, a transthoracic echocardiogram, a computed tomography (CT), a magnetic resonance imaging (MRI) scan, or an X-ray. It should be understood that the preoperative data may include information from other diagnostic medical procedures, imaging modalities, and modeling systems, as desired.

[0057] Similarly, the second image acquisition system 110 is configured to acquire a second image dataset 116 from the patient. Specifically, the second image acquisition system 110 can be configured to acquire the second holographic image dataset 116 from the patient intraoperatively, and most particularly, in real time during surgery. In some embodiments, the second image acquisition system 110 can be an ultrasound imaging device. Other suitable types of devices and modalities for the second image acquisition system 110 can also be employed as desired.

[0058] Although the use of the first image acquisition system 108 and the second image acquisition system 110 is shown and described herein, embodiments employing only one or the other of the first image acquisition system 108 and the second image acquisition system 110 are considered to be within the scope of the present disclosure.

[0059] Continue to refer Figure 1 , the computer system 106 of the present disclosure may have at least one processor 118. The one or more processors 118 may perform functions associated with the operation of the holographic augmented reality visualization and guidance system 100. The one or more processors 118 may be any type of general-purpose or special-purpose processor. In some cases, according to other embodiments, multiple processors 118 may be utilized. In fact, as non-limiting examples, the one or more processors 118 may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a processor based on a multi-core processor architecture.

[0060] The computer system 106 may have at least one memory 120 on which tangible, non-transitory machine-readable instructions 122 are stored. The memory 120 may be one or more memories of any type suitable for the local application environment and may be implemented using any suitable volatile or non-volatile data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. For example, the memory 120 may be composed of any combination of random access memory (RAM), read-only memory (ROM), static memory (such as a magnetic disk or optical disk), a hard disk drive (HDD), or any other type of non-transitory machine or computer-readable medium. The instructions stored in the memory 120 may include program instructions or computer program code that, when executed by the one or more processors 118, enables the holographic augmented reality visualization and guidance system 100 to perform the tasks described herein.

[0061] The machine-readable instructions 122 stored in the memory 120 may include modules. The modules may be implemented as one or more of functional logic, hardware logic, electronic circuits, software modules, etc. As needed, the modules may include one or more of an augmented reality system module, an image acquisition module, an instrument tracking module, an image dataset registration module, a hologram rendering module, an image registration module, a trajectory hologram rendering module, and / or other suitable modules.

[0062] The computer system 106 can communicate with the augmented reality system 102, the tracking device 104, the first image acquisition system 108, and the second image acquisition system 110, for example, via the network 112, and can be configured by machine-readable instructions 122 to operate according to the method 200 described herein. If desired, the computer system 106 can be provided separately and apart from the augmented reality system 102, or can be provided as an integrated unit with the augmented reality system 102.

[0063] It should be understood that, as non-limiting examples, the network 112 of the holographic augmented reality visualization and guidance system 100 may include a wireless access network (such as LTE or 5G), a local area network (LAN), a wide area network (WAN) (such as the Internet), or a wireless LAN (WLAN). It should be understood that this is not limiting, and the scope of the present disclosure includes embodiments in which one or more computing platforms of the holographic augmented reality visualization and guidance system 100 can be operably linked via some other communication coupling. One or more computing platforms can be configured to communicate with the networked environment via a wireless or wired connection. In addition, in an embodiment, one or more computing platforms can be configured to communicate directly with each other via a wireless or wired connection. Examples of one or more computing platforms may include, but are not limited to, smartphones, wearable devices, tablets, laptops, desktop computers, Internet of Things (IoT) devices, or other mobile or fixed devices, such as standalone servers, networked servers, or server arrays.

[0064] The augmented reality system 102 may be configured to render a plurality of holograms according to the method 200 of the present disclosure. Specifically, the augmented reality system 102 may be a mixed reality (MR) display, such as MR smart glasses or a MR head-mounted display. Non-limiting examples of the augmented reality system 102 include Magic Leap or Microsoft It should be understood that other types of MR displays can be used with the augmented reality system 102, as long as they are capable of overlaying computer-generated images on real-world objects. Furthermore, although the augmented reality system 102 is primarily described herein as a head-mounted display, it should be understood that other types of displays that are not head-mounted but are capable of generating and overlaying holograms on a real-world view may also be employed as desired.

[0065] It should be understood that in the case where the augmented reality system 102 does not include the computer system 106, the augmented reality system 102 may also include additional non-transitory memory and processing units (which may include one or more hardware processors) that may facilitate the rendering or generation of holograms. The augmented reality system 102 may also include a camera that records one or more images, one or more image generation components that generate / display visualizations of holograms, and / or other visualization and / or recording elements. Recording may include tracking one or more steps or actions of a medical procedure, the movement of one or more surgical instruments, and the patient's (pre- and post-operative) anatomy in real time in three-dimensional space.

[0066] In other examples, it should be understood that the augmented reality system 102 may also include a plurality of position sensors 123b. The plurality of position sensors 123b of the augmented reality system 102 may be configured to determine various position information of the augmented reality system 102, such as the approximate position, orientation, angular velocity, and acceleration of the augmented reality system 102 in three-dimensional (3D) space. In particular, it should be understood that this and the described registration method allow the holographic image to be accurately registered with the imaged anatomical structure in the physician's field of view during operation and displayed.

[0067] Non-limiting examples of the plurality of position sensors 123b include accelerometers, gyroscopes, electromagnetic sensors, and optical tracking sensors. It should also be understood that those skilled in the art may employ different types and quantities of the plurality of position sensors 123b of the augmented reality system 102, for example, as required by the surgery or condition in which the augmented reality system 102 is to be used.

[0068] The augmented reality system 102 can be configured to generate multiple holograms for the physician to view throughout the process of planning and performing an interventional procedure. Figure 1 As shown, for example, the holograms generated by the augmented reality system 102 may include a first hologram 124, a second hologram 126, an ablation treatment zone hologram 128, and an applicator guidance hologram 130. The first hologram 124 generated by the augmented reality system 102 may be based on a first holographic image dataset 114 from a patient. The second hologram 126 generated by the augmented reality system 102 may be based on the second holographic image dataset 116. As further described herein, the treatment zone hologram 128 may be visualized based on a tracking instrument dataset 132, which may be manually or automatically selected and stored on the memory 120 of the computer system 106. As further described herein, the applicator guidance hologram 130 may be generated within the treatment zone at an adjustable depth from the applicator tip.

[0069] It should be understood that there can be multiple treatment zones, and therefore, there can be multiple treatment zone holograms 128. Multiple treatment zone holograms 128 can intersect. As a non-limiting example, multiple treatment zones can form a Mickey Mouse-shaped treatment zone, such that a first treatment zone hologram in the shape of a sphere, a second treatment zone hologram in the shape of a sphere, and a third treatment zone hologram in the shape of a sphere intersect and overlap to form a uniquely shaped polygon. Referring to the above example, the resulting polygon can include three different leaves generated simultaneously by three different types of tracking devices, or three different leaves generated multiple times by a tracking device, the three different leaves including different settings and different trajectories, all displayed in a unique manner (color, texture, data value overlay, heat map gradient of actual and predicted differences, etc.).

[0070] An ablation treatment zone hologram 128 may be generated using the augmented reality system 102 and the computer system 106. The treatment zone hologram 128 may provide the physician with a visualization of where the tracking instrument 104 will apply treatment. The physician may simultaneously visualize the first hologram 124 and the treatment zone hologram 128 to visualize the predicted effects of the treatment or surgery on the patient's anatomy. Using the computer system 106, the physician may adjust the predetermined treatment zone. The augmented reality system 102 may then adjust the treatment zone hologram 128 accordingly, which may allow the physician to visualize the effects of the treatment or surgery in real time. The augmented reality system 102 and the computer system 106 may then plan and generate a guidance hologram 130 based on the treatment zone selected by the physician. The guidance hologram 130 may assist the physician in 3D placement of the tracking instrument 104 to apply the predicted treatment according to the treatment zone hologram 128. Figure 5 As shown, the guidance hologram 130 may include at least one of a guidance center 134, a guidance ring 136, and a guidance path 138. The guidance center 134 may provide initial guidance for the physician to initially align the tracking instrument 104 during surgery. The guidance center 134, marked at a specific location, may also provide the physician with an indication of the depth to which the tracking instrument 104 should be inserted into the patient during surgery, such that, during operation, once the tracking instrument 104 is aligned with the guidance center 134, the tracking instrument 104 is at the desired depth. The guidance ring 136 may provide the physician with the desired angle and further guidance for inserting the tracking instrument 104 into the patient during surgery. The guidance path 138 may provide the physician with a more detailed and thorough path to follow the tracking instrument 104 during surgery, allowing the physician to assess the angle, position, and depth that the tracking instrument 104 should follow during surgery. Together, the guidance center 134, the guidance ring 136, and the guidance path 138 may provide the physician with a trajectory to follow during surgery.

[0071] It should be noted that Figure 6As shown, the physician can select parameters to perform ablation during surgery. The physician can modify the ablation time and ablation power for each generated guidance hologram 130. In addition, the ablation portion can be represented in the hologram by changing the color, pattern, size, or other visual cues or markings representing time, power, etc. of the ablation zone. In some embodiments, the ablation zone can display transient elements as it grows. Those skilled in the art can select appropriate ablation time and ablation power within the scope of the present disclosure.

[0072] In certain embodiments, the holographic augmented reality visualization and guidance system 100 may include multiple tracking instruments 104. The use of multiple tracking instruments may allow a physician or multiple physicians to treat irregularly shaped and / or intersecting treatment areas sequentially or simultaneously. Multiple tracking instruments 104 may allow a physician to fully implement a predetermined 3D margin zone, particularly where there are variations in tumor characteristics, healthy tissue, structures, voids, or hydropic fluids. In the case of using multiple tracking instruments, there may be a complex interaction of position, power, timing, etc. when each tracking instrument 104 overlaps in the treatment area. Advantageously, the use of multiple tracking instruments 104 may allow for avoidance of adjacent structures by utilizing multiple synchronized trajectories. A physician may select an appropriate number of tracking instruments 104 as needed for a given procedure.

[0073] In addition to rendering or generating various holograms, the augmented reality system 102 can also be configured to display a plurality of operational information or details to the physician. For example, the augmented reality system 102 can project a plurality of operational information aligned with a real-world object (such as a patient). For example, the operational information can include real-time navigation instructions or guidance for a trajectory to be taken. It should be understood that the augmented reality system 102 can project a plurality of operational information onto various real-world objects (such as the tracking device 104) as well as the various rendered holograms as needed.

[0074] Ideally, the generation of operation information or details allows the physician to simultaneously observe the patient, with multiple operation information within the same field of view. In addition, the generation of operation information or details and various holograms allows the physician to plan, adjust, or pre-orient the tracking instrument 104 during the operation, and then align the tracking instrument 104 with a specific planned trajectory.

[0075] The computer system 106 can communicate with the augmented reality system 102 and the tracking device 104. The computer system 106 can be configured to store and generate a plurality of operational information, either through full manual intervention by a physician or other medical professional, or automatically based on machine-readable instructions 122 encoded on the memory 120. For example, a plurality of operational information can be generated in the augmented reality system 102 based on the position or orientation of the tracking device 104 determined by a sensor, such as through an algorithm, artificial intelligence (AI) protocol, or other physician-entered data or thresholds.

[0076] Furthermore, the computer system 106 can also be configured to allow the physician to selectively adjust the plurality of operational information in real time. Furthermore, the physician can determine which of the plurality of operational data is actively displayed to the physician. It should be understood that within the scope of this disclosure, other settings and attributes of the plurality of operational information can be adjusted by the physician in real time.

[0077] Specifically, it should be understood that the augmented reality system 102 of the present disclosure advantageously allows a physician to perform the method 200 of performing an interventional procedure on a patient while viewing the patient and the holograms described herein using the augmented reality system 102. Likewise, the physician advantageously allows the physician to utilize the augmented reality system 102 to at least one of visualize, guide, and navigate the tracked instrument 104 during the interventional procedure, as further described herein with respect to the method 200 of the present disclosure.

[0078] refer to Figure 3 , shows an overview of a holographic augmented reality visualization and guidance system 100. In operation, a physician can observe a patient and a first hologram 124 through the augmented reality system 102. As described herein, the augmented reality system 102 and the tracking device 104 can utilize a plurality of sensors 123. The physician can also utilize a second image acquisition system 110 during surgery to collect a second holographic image dataset 116 intraoperatively.

[0079] Figure 2A-2DAn example flow chart of a method 200 according to one embodiment of the disclosure is shown. The method 200 may include step 202 of providing a holographic augmented reality visualization and guidance system 100 as described herein. In step 204, the method 200 may then include acquiring, by the first image acquisition system 108, a first holographic image dataset 114 from a patient, and optionally, in step 206, acquiring, by the second image acquisition system 110, a second holographic image dataset 116 from the patient. In some examples, the first holographic image dataset 114 may include a region of the patient's anatomy to be treated. The method 200 may include step 208 of tracking, by the computer system 106, a tracking device 104 using a plurality of sensors 132a to provide a tracking device dataset 132. The method 200 may then include step 210 of registering, by the computer system 106, the first holographic image dataset 114, the second image dataset 116, and the tracking device dataset 132 for projection onto a physical patient.

[0080] In step 212 , the method 200 may then include rendering, by the augmented reality system 102 , the first hologram 124 based on the first holographic image dataset 114 from the patient, the second hologram 126 based on the second holographic image dataset 116 , and the treatment area hologram 128 based on the tracked instrument dataset 132 for viewing by the physician.

[0081] The method 200 may include a step 214 in which the computer system 106 collects operational information or details to provide to the physician. In step 216, the augmented reality system 102 may project a plurality of operational information aligned with a real-world object (such as a patient). For example, the operational information may include real-time navigation instructions or guidance for a trajectory to be taken. It should be understood that the augmented reality system 102 may project the plurality of operational information onto various real-world objects (such as the tracking device 104) and various rendered holograms as needed.

[0082] The method may include a step 218 of simultaneously displaying the first hologram 124 and the treatment zone hologram 128. In this manner, the physician can visualize the predicted effect of the treatment or surgery on the patient's anatomy and, in step 220, can evaluate the treatment zone versus reality.

[0083] The method may include step 222 of adjusting the treatment zone hologram 128 using a computer system. The physician may adjust the trajectory of the treatment zone hologram 128 relative to the first hologram 124. When the physician is satisfied with the placement of the treatment zone hologram 128, the physician may use voice commands or hand gestures, or other operator input provided to the augmented reality system to mark or otherwise lock the planned trajectory, depth, and predicted treatment zone. The method may include step 224 of rendering an applicator guidance hologram 130 based on the selected treatment area, which may include parameters of the predicted area, such as the time and duration of treatment, if applicable. Depending on the shape and volume of the tumor to be treated, the physician may repeat steps 222, 224 until the treatment zone hologram 128 covers the entire treatment zone.

[0084] The method 200 may then include a step 226 of simultaneously displaying the guide hologram 130, the first hologram 124, and the treatment area hologram 128. As described above, the physician is able to visualize the predicted effect of the treatment or surgery on the patient's anatomy and may determine whether the guide hologram 130 needs to be adjusted. The method 200 may include a step 228 of marking, by the tracking instrument, a holographic guide to be viewed in the augmented reality system. The operator may mark multiple holographic applicator guides based on the trajectory of one or more tracking instruments. Figure 4 As shown, each holographic guide provides a planned trajectory, including but not limited to a guide center 134, a guide path 138, a guide ring 136, etc., which is aligned with the tracking instrument when the holographic guide is marked. The registration of the planned holographic applicator guide with the real-time ultrasound and tomography image data sets for tracking helps to avoid critical structures such as blood vessels and nearby organs during treatment. The placement of the holographic applicator guide can be facilitated by image-based holographic registration, which is obtained from pre-operative tomography imaging, including but not limited to segmented structures and tumors, anatomical planes or multi-plane images that are coplanar with the tracked ultrasound image. The three anatomical planes obtained from the tomography imaging can be updated as the tracking instrument 104 is adjusted so that their common point is consistent with the planned ablation zone center point, the tip of the tracking instrument, or other points of interest in the head-mounted display coordinates to help place the planned ablation zone.

[0085] Method 200 may then include step 230 of the physician performing an interventional procedure on the patient while viewing the patient and first hologram 124 and guide hologram 130 using augmented reality system 102. In step 230, the physician uses augmented reality system 102 to place tracking instrument 104 during the procedure.

[0086] The method may include, step 232, recording, using a computer system, a member selected from the group consisting of: a first holographic image dataset, a second holographic image dataset, a tracking instrument dataset, a guidance hologram, and combinations thereof. Recording portions of the surgery and any combination of datasets may allow the physician to learn from the surgery and implement changes in real time or with respect to future surgeries.

[0087] The method may include step 234 of generating feedback by the augmented reality system based on the first holographic image dataset, the second holographic image dataset, and the tracked instrument dataset, and step 232. Real-time feedback can help the physician implement surgical modifications during the procedure itself, thereby providing a more efficient and effective procedure. Visual, auditory, or digital holographic feedback is provided to help align the tracked instrument with the corresponding holographic applicator guide. The correspondence between the tracked instrument and the holographic guide can be indicated by holographic ablation zone color, texture, labeling, etc. The holographic ablation zone of the tracked instrument can be represented relative to the tip of the tracked instrument. The tracked instrument can then be advanced to the planned depth so that the holographic ablation zones of the tracked and marked guides coincide. The planned depth of the holographic applicator guide can be adjusted to intersect with the tracked ultrasound image in an out-of-plane ultrasound transducer position and can then be adjusted proximally or distally in an in-plane or out-of-plane ultrasound procedure using voice commands of the augmented reality system or another operator input method associated with the augmented reality system. After planning a set of corresponding holographic applicator guides, multiple individual tracking instruments can be inserted and placed as a group, whereby the merging of multiple zones of planned ablation zones conforms to the tumor or target tissue, which is important for tumors that cannot conform to the tumor or target tissue.

[0088] After treatment, additional imaging is performed, typically using contrast agents to monitor the effectiveness of the treatment. Updated images used to monitor the actual treatment or lack thereof are loaded into the computer system and registered with the reference image and tracking ultrasound for visualization. An additional holographic applicator can then be used to guide and track the instrument to achieve the endpoint of the treatment and to visualize the tumor in 3D with sufficient 3D margin (typically 5mm, etc.). Methods for visualizing overlapping holograms based on fused imaging data and predicted ablation zones include using cross-contours of tomographic segmentation results on real-time ultrasound images, visualizing the 3D predicted ablation zone using a wireframe shading model, and turning the hologram on and off using voice commands or another AR UX method (such as hand gestures).

[0089] In further embodiments, the plurality of operation information includes fused preoperative and intraoperative data. The preoperative and intraoperative data are fused in a manner that synergistically combines the advantages of each imaging modality. In some cases, after the computer system 106 completes the fusion, the fusion can be manually refined, for example, using at least one of the plurality of algorithms set forth in the machine-readable instructions 122 or through artificial intelligence (AI).

[0090] While certain representative embodiments and details have been shown for the purpose of illustrating the invention, it will be apparent to those skilled in the art that various changes can be made therein without departing from the scope of the disclosure as further described in the appended claims.

[0091] Exemplary embodiments are provided to make this disclosure comprehensive and to fully convey the scope to those skilled in the art. Many specific details, such as examples of specific components, devices, and methods, are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be adopted, and that the exemplary embodiments can be implemented in many different forms, and none of them should be construed as limiting the scope of the present disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail. Within the scope of the present technology, equivalent changes, modifications, and variations can be made to some embodiments, materials, compositions, and methods with substantially similar results.

Claims

1. A method for planning and performing an interventional procedure on a patient by a physician, the method comprising: providing an augmented reality system, a tracking device having a plurality of sensors configured to provide a tracking device dataset, a first image acquisition system, a second image acquisition system, and a computer system having a processor and a memory, the tracking device having a plurality of sensors configured to provide a tracking device dataset, the computer system communicating with the augmented reality system, the tracking device, the first image acquisition system, and the second image acquisition system; acquiring a first holographic image dataset from a patient by the first image acquisition system; acquiring a second holographic image dataset from the patient by the second image acquisition system; Tracking the tracking device by the computer system using a plurality of sensors to provide the tracking device dataset; registering, by the computer system, the first holographic image dataset, the second holographic image dataset, and the tracking device dataset with a patient; rendering, by the augmented reality system, a first hologram from the first holographic image dataset, a second hologram from the second holographic image dataset, and a treatment zone hologram from the tracking device dataset; adjusting the treatment area hologram using the computer system; rendering a guidance hologram by the augmented reality system; as well as An interventional procedure is performed on the patient by a physician while observing the patient, the first hologram, and the guidance hologram with the augmented reality system, whereby the physician uses the guidance hologram and the augmented reality system to place the tracking instrument during the interventional procedure.

2. The method according to claim 1, wherein The computer system is used to adjust the treatment area hologram and the augmented reality system is used to render the guide hologram until the treatment area hologram covers the entire treatment area.

3. The method according to claim 1, wherein The first image acquisition system is one of a magnetic resonance imaging (MRI) device and a computed tomography (CT) device.

4. The method according to claim 2, wherein: The first holographic image data set from the patient is one of pre-operative images and intra-operative feedback loop.

5. The method according to claim 1, wherein The first holographic image data set from the patient is pre-operative.

6. The method according to claim 1, wherein The second image acquisition system is an ultrasound device.

7. The method according to claim 1, wherein The method includes using multiple tracking devices having multiple treatment zone holograms.

8. The method of claim 1, further comprising the step of evaluating the treatment area hologram with reality.

9. The method of claim 1, further comprising the step of simultaneously displaying the first hologram and the treatment zone hologram using the augmented reality system.

10. The method according to claim 9, wherein: The guide hologram is displayed simultaneously with the first hologram and the treatment area hologram.

11. The method of claim 1 further comprising the step of collecting operational information by the computer system.

12. The method according to claim 11, wherein The operational information includes real-time navigation instructions.

13. The method of claim 1, further comprising the step of projecting operational information onto the tracking device by the augmented reality system.

14. The method of claim 1, further comprising the step of using the tracking instrument by a physician to mark the guidance hologram to be viewed in the augmented reality system.

15. The method according to claim 14, wherein The guide hologram includes at least one of a center, a guide path, and a ring.

16. The method of claim 1, further comprising the step of generating, by the augmented reality system, feedback based on the first holographic image dataset, the second holographic image dataset, and the tracking device dataset.

17. The method according to claim 16, wherein The feedback includes at least one of visual feedback, auditory feedback and digital holographic feedback.

18. The method of claim 1, further comprising recording, using the computer system, one of: the first holographic image dataset, the second holographic image dataset, the tracking device dataset, the guidance hologram, and combinations thereof.

19. The method according to claim 1, wherein The guidance hologram is located in a treatment area at a distance from the tracking device.

20. A system for planning and performing an interventional procedure on a patient by a physician, comprising: augmented reality systems; Tracking devices with multiple sensors; a first image acquisition system configured to acquire a first holographic image dataset from a patient; a second image acquisition system configured to acquire a second holographic image dataset from the patient; a computer system having a processor and a memory, the computer system being in communication with the augmented reality system, the tracking device, the first image acquisition system, and the second image acquisition system, and being configured by machine-readable instructions to: tracking the tracking device using the plurality of sensors to provide a tracking device dataset; as well as registering the first holographic image dataset, the second holographic image dataset, and the tracking instrument dataset with a patient, The augmented reality system is configured to render the first hologram, the second hologram, the treatment area hologram, and the guide hologram for observation by the physician, and During an interventional procedure on a patient using the augmented reality system, the first hologram and the guide hologram are displayed to a physician, and the guide hologram and the augmented reality system are configured to be used by the physician to place the tracking instrument during the interventional procedure.

Images