Holographic augmented reality ultrasound needle guide for percutaneous surgical procedure insertion

Abstract

A holographic augmented reality ultrasound needle guide system (100) and method (200) includes an augmented reality display (108), such as a head-mounted device (122) worn by a user (104). The augmented reality display (108) is configured to depict a virtual ultrasound image (110). The augmented reality display (108) is further configured to allow the user (104) to select a desired reference point (114) on the virtual ultrasound image (110). The system (100) is configured to depict a holographic needle guide (112) based on the selection of the desired reference point (114). The system (100) is further configured to adjust a trajectory of the holographic needle guide (114) to avoid penetrating an undesired anatomical structure (130). The augmented reality display (108) is further configured to mark the holographic needle guide (112) into a selected locked trajectory (128) and position.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefits of U.S. Provisional Application Serial No. 63 / 025,584, filed May 15, 2020, and U.S. Provisional Application Serial No. 62 / 942,857, filed December 3, 2019. The entire disclosure of the above applications is incorporated herein by reference. Technical Field

[0003] This disclosure relates to applications of holographic augmented reality, and more specifically, to medical applications employing holographic augmented reality. Background Technology

[0004] This section provides background information in connection with this disclosure, which is not necessarily prior art.

[0005] Ultrasound guidance has become standard procedure in many needle-based medical procedures, such as needle biopsy and local anesthesia. The use of ultrasound guidance has been shown to improve the safety and success rate of these procedures. However, difficulties in locating and orienting the needle can sometimes lead to misidentification of the needle tip, where the needle may not want to pass through or reach certain anatomical features or locations.

[0006] Certain technologies can be used to help practitioners confidently align needles. These technologies range from simple mechanical devices to advanced automated needle detection software. One particular technology involves a mechatronic ultrasonic needle guide, which is a physical device attached to an ultrasonic probe for guiding the needle along a trajectory visible in the ultrasound image. Specifically, the physical ultrasonic needle guide is typically secured to the ultrasonic probe using a reusable holder positioned above the transducer of the probe. The needle guide can be pre-selected and used based on a fixed or designed angle depth. Positioning needle guides selectable between a limited number of angles (e.g., up to five (5) different predetermined angle depths to accommodate different insertion trajectories) are also available. Typically, these physical needle guides are detachably attached to a reusable holder, which itself is coupled to the ultrasonic probe.

[0007] Physical ultrasonic needle guides have certain limitations, including cost and limited reusability. In fact, most ultrasonic needle guides are designed for single use. Such physical ultrasonic needle guides may also require a dedicated ultrasonic transducer designed for use with a needle guide or associated support. Even when certain predetermined angle depths can be selected, using these physical needle guides may not provide practitioners with a complete and unrestricted angular guidance range.

[0008] Known needle guides are also vendor and probe specific and are often limited to "in-plane" or "perpendicular to plane" angles. They are often criticized by experienced clinicians such as interventional radiologists because, as noted above, the user is constrained to a single or few angles supported by the mechanical guide. Clinicians desire the flexibility to move the probe independent of the guide, often needing to direct the needle from the probe out of plane for optimal visibility within the procedure.

[0009] Holographic augmented reality technology is being used more widely in healthcare applications to improve medical procedures, clinical outcomes, and long-term patient care. These augmented reality technologies are also used to augment the real environment in patient care scenarios, for example, using content-specific information to improve the treatment outcomes of patients. For example, practitioners can view additional information in the same field of view while performing a medical procedure without the practitioner having to change their specific perspective, which can slow down or decrease the efficiency of the procedure.

[0010] Accordingly, there is a continuing need for an ultrasound needle guide system and method that is cost effective, minimizes medical waste, and provides practitioners with a complete and unrestricted range of angle guidance to optimize percutaneous surgical procedures. Ideally, the system and method involves holographic augmented reality and can be used with any type of ultrasound transducer. SUMMARY

[0011] According to the present disclosure, a holographic augmented reality ultrasound needle guide system and method that is cost effective, minimizes medical waste, and provides practitioners with a complete and unrestricted range of angle guidance to optimize percutaneous surgical procedures, and can be used with any type of ultrasound transducer, has been surprisingly discovered.

[0012] In one embodiment, a holographic augmented reality ultrasound needle guide system for guiding a user to percutaneously insert a needle into a patient includes an augmented reality display. The augmented reality display is configured to depict a virtual ultrasound image of a portion of the patient. The augmented reality display is further configured to depict a holographic needle guide on the patient based on a selection of a reference point in the virtual ultrasound image.

[0013] In another embodiment, a method of using a holographic augmented reality ultrasound needle guide system can include the step of providing an augmented reality display, where the augmented reality display is configured to depict a virtual ultrasound image of a portion of a patient. The augmented reality display is further configured to depict a holographic needle guide on the patient based on a selection of a reference point in the virtual ultrasound image. The method can include the step of selecting the reference point in the virtual ultrasound image of the portion of the patient. The method can then include the step of displaying the holographic needle guide on the patient based on the selection of the reference point in the virtual ultrasound image of the portion of the patient. The method can then include the step of percutaneously inserting a needle along a trajectory of the holographic needle guide.

[0014] In another embodiment, the systems and methods of the present disclosure produce a holographic light ray to simulate the intended trajectory of a physical needle guide by using spatial computation, augmented reality, and artificial intelligence (AI), allowing for a holographic display of the intended needle trajectory. Such systems and methods can be used with any augmented reality display and optionally use electromagnetic or optical tracking. This allows the holographic needle guide to be adapted in design to any ultrasound probe, adjusted to any desired angle, and changed in size to accommodate any desired needle or trocar size.

[0015] In certain embodiments, the systems and methods of the present disclosure can include a unique combination of ultrasound technology with holography. At least one reference point can be selected on a virtual ultrasound image, and this reference point allows the user to actively change the angle of the virtual / holographic needle guide generated by the system relative to the patient’s anatomy. The system can include an additional conventional ultrasound probe, which can have known coordinates, gyroscopes, and position sensors (e.g., using gyroscopes and accelerometers in the probe). The ultrasound image can include one or more pre-recorded ultrasound images or can include a virtual ultrasound image obtained in real time.

[0016] Various embodiments of the present disclosure can include the following aspects. In operation, a needle guide, which is typically a physical stand, can be a “ghost” or superimposed into the field of view of a practitioner wearing a holographic visualization system, such as Microsoft HoloLens® head-mounted device, as one non-limiting example. This allows the practitioner to perform needle insertion at any desired angle without the need for additional, disposable, physical needle guides. Ultrasound or EM tracking of the needle can also be used and relayed to the practitioner through the holographic visualization system. The system can also generate an error bar or a region of related acceptability that can be associated with the insertion of the needle in a particular procedure.

[0017] ​It should be understood that the use of the system and method of the present disclosure allows for improved needle visualization, reduced procedure time, more reliable clinical outcomes, and ideally elimination of the need for sterilization of any physical ultrasound needle guide, stand, or operating room. Key structure avoidance for minimizing non-target harm is also provided. Advantageously, the system and method of the present disclosure can be used for a wide variety of medical procedures, including but not limited to nerve blocks, local anesthesia, vascular access, biopsy, ablation, endoluminal, transvaginal, transrectal, out-of-plane, bi-plane, curved path, in-plane straight path needle guidance at any variable or fixed angle. The system and method are also particularly suitable for use in mammography and related procedures.

[0018] In other embodiments, the system and method of the present disclosure addresses the limitations of mechanical needle guides by providing a holographic needle guide that supports virtually any trajectory or angle that can be desirably implemented within the context of a procedure. The holographic needle guide can be visualized by a stereoscopic or stereoscopic head mounted display such as Microsoft HoloLens® or other augmented reality device. The holographic needle guide is interactive by the user, not tethered to the probe, but has the ability to similarly guide the proceduralist's needle to a user-defined destination or target on any user-defined trajectory.

[0019] It should be understood that the holographic needle guide provides a guide that is superior to mechanical guides and has the potential to replace mechanical guides in the market. Instead of attaching a physical guide to the probe, the user dons a mixed reality head mounted device running an application of the present disclosure.

[0020] In particular embodiments, the system and method is initiated by the user selecting a target destination for the needle guide on the ultrasound plane. Once set, the holographic target is translated to real space within the patient. The complete needle guide is instantiated at that real location, and the user then proceeds to position the holographic guide while moving the ultrasound probe to create any needed view.

[0021] Typically, it is desirable for the ultrasound plane to "sweep" up and down the guide, providing visibility in all directions to the anatomy surrounding the needle guide. This sweeping is not possible with mechanical guides and is one of the primary reasons that experienced proceduralists have shied away from mechanical guides.

[0022] The practitioner needs both of their hands while performing an ultrasound guided needle procedure. Therefore, allowing for the marking of the holographic target, needle guide, and ultrasound probe location, allows them to set down the tool (e.g., needle or probe) and then return and know where they want to insert the needle based on the known and determined probe location and anatomical target.

[0023] Other areas of applications will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0024] The drawings described herein are for illustrative purposes only of selected embodiments and are not intended to limit the scope of the present disclosure.

[0025] Figure 1 is a perspective view of a holographic augmented reality ultrasound needle guide system in operation according to one embodiment of the present disclosure, depicting an augmented virtual window showing a virtual ultrasound scan visible through a head-mounted display and an augmented virtual ultrasound projection;

[0026] Figure 2 is a perspective view of a system according to one embodiment of the present disclosure, depicting a user selecting a reference point on an augmented virtual window; Figure 1

[0027] Figure 3 is a perspective view of a system according to one embodiment of the present disclosure, depicting a holographic needle guide being displayed and a user adjusting a trajectory of the holographic needle guide; Figure 1 Figure 2 is a perspective view of a system according to one embodiment of the present disclosure, depicting a user inserting a needle along the holographic needle guide;

[0028] Figure 4 Figures 1-3 is a perspective view of a system according to one embodiment of the present disclosure, depicting a user inserting a needle along the holographic needle guide;

[0029] Figure 5 is a perspective view of a system according to one embodiment of the present disclosure, shown from the perspective of a user wearing a head-mounted display and a magnified perspective of a user inserting a needle along a holographic needle guide at FIG. 5 in Figure 4

[0030] Figure 6 is a schematic view of a system according to another embodiment of the present disclosure, shown in a perspective view; Figures 1-5

[0031] Figure 7 is a front view of a system according to yet another embodiment of the present disclosure, depicting the system arranged on a movable cart; Figures 1-6

[0032] Figure 8 is a schematic view of an ultrasound scan according to one embodiment of the present disclosure, depicting an ultrasound scan with a selected reference point and displaying a virtual needle insertion point;

[0033] Figure 9A ​​​​​​is a top plan view of a needle insertion guide for determining a desired needle insertion point according to one embodiment of the present disclosure, depicting a physical template having a hole arranged adjacent to an ultrasound probe used with the needle insertion guide;

[0034] Figure 9B is a side view of a needle insertion guide for determining a desired needle insertion point according to another embodiment of the present disclosure, depicting a physical template having a measurement mark disposed adjacent to an ultrasound probe used with the needle insertion guide;

[0035] Figure 10A and Figure 10B is a schematic diagram showing a method for generating and moving a needle insertion guide according to the present disclosure;

[0036] Figure 11A , Figure 11B and Figure 11C is a partial perspective view showing a step-by-step procedure for needle guide targeting using a heads-up display method;

[0037] Figure 12A , Figure 12B and Figure 12C is a partial perspective view showing a step-by-step procedure for needle guide targeting using a flashlight display method; and

[0038] Figure 13 is a flowchart showing a method of using the system in Figures 1-12C according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0039] The following description of the technology is merely exemplary in nature of the subject matter, manufacture and use of one or more applications, and is not intended to limit the scope, application, or uses of any specific application claimed in this application or such other applications as can be permitted to claim priority from this application or patent arising therefrom. Regarding the disclosed methods, the order of presentation of steps is not essential inasmuch as, in various embodiments, the order of steps can be different, including instances in which certain steps can be performed concurrently. As used herein, “a” and “an” means “at least one” of the items being described; thus, if there are two of the items, both are meant to be covered. Unless specifically stated otherwise, all amounts expressed in this specification are understood to be modified in all instances by the term “about,” and all geometric and spatial descriptors are understood to be modified in all instances by the term “substantially.” “About” when applied to numerical values means that the calculation or the measurement allows some slight imprecision (has some room for error, with the room for error varying with different types of numerical values), with the imprecision resulting from any one of many potentially cause, such as rounding off, measurement error, measurement of a parameter caused by a lack of perfection in the measurement system, human error, and the like; and, in the context of financial, stock market, and the like, “about” can mean roughly in the associated direction. If there are uses of “about” caused by more specific instrumentalities, the more specific instrumentalities control for that instance.

[0040] Although the open-ended term “comprise” as a synonym for the terms including, containing, or having (e.g., including, including or having) is used in this document to describe and claim embodiments of the technology, embodiments can alternatively be described using the more limiting terms (e.g., “consist of’ or “consist essentially of’). Thus, for any given embodiment of materials, components or process steps, the technology also specifically includes embodiments which consist of, or consist essentially of, the materials, components and process steps of those embodiments excluding additional materials, components or processes (for embodiments consisting of), and excluding additional materials, components or processes which do not impact the essential characteristics of the embodiments (for embodiments consisting essentially of), even if such additional materials, components or processes are explicitly described in this document. For example, a recitation of a composition or method comprising elements A, B, and C specifically contemplates embodiments consisting of, and consisting essentially of, A, B, and C, excluding element D, even if element D can be recited in the art as an element which can be included.

[0041] As used herein, unless otherwise indicated, the scope of the disclosure includes endpoints and all values and further divisions between the endpoints. Thus, for example, a range of “from A to B” or “from about A to about B” includes A and B. Disclosure of a value and of a range of values for a specific parameter (e.g., an amount, a weight percent, etc.) does not exclude other values and ranges of values that are useful herein. It is envisioned that two or more particular example values for a given parameter can define the endpoints of a range of values for which the parameter can be claimed. For example, if a parameter X is exemplified herein as having a value of A and is also exemplified as having a value of Z, it is envisioned that the parameter X can have a range of values from about A to about Z. Similarly, it is envisioned that disclosure of two or more ranges of values for a parameter, whether these ranges are nested, overlapping, or distinct, encompasses all possible combinations of ranges of values that might be claimed using the disclosed endpoints. For example, if a parameter X is exemplified herein as having a range of values of 1-10, or 2-9, or 3-8, it is also envisioned that the parameter X can have other ranges of values, including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, and so on.

[0042] When an element or layer is referred to as being “on”, “engaged to”, “connected to” or “coupled to” another element or layer, it can be directly on, engaged, connected or coupled to the other element or layer, or one or more intervening elements or layers can be present. In contrast, when an element is referred to as being “directly on”, “directly engaged to”, “directly connected to” or “directly coupled to” another element or layer, then there are no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between”, “adjacent” versus “directly adjacent”, etc.). As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0043] Although the terms first, second, third, etc. can be used herein to describe various elements, components, regions, layers and / or sections, these elements, components, regions, layers and / or sections should not be limited by these terms. These terms can be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first”, “second”, and other numerical terms when used herein do not connote an ordering or sequence unless the context clearly dictates. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

[0044] For ease of description, spatially relative terms, such as "inner," "outer," "beneath," "below," "bottom," "above," "upper," and the like, can be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientations depicted in the figures. For example, if a device is inverted, elements described as "below" or "beneath" other elements or features would then be oriented "above" the other elements or features. Thus, the example term "below" can encompass both an orientation of above and below. The device can be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0045] Figures 1-7 A system 100 for guiding a user 104 to percutaneously insert a needle 102 into a patient 106 for a medical procedure is shown. The system 100 includes an augmented reality display 108. The augmented reality display 108 is configured to depict a virtual ultrasound image 110 of a portion of the patient 106 in a mode referred to as a heads-up display or "HUD" mode. The augmented reality display 108 is also configured to depict a holographic needle guide 112 on the patient 106 based on a selection of a reference point 114 in the virtual ultrasound image 110 in a mode referred to as a "flashlight" mode. As non-limiting examples, the holographic needle guide 112 can be virtually depicted in the form of an elongated shaft, a tube, or a cylinder, each example illustrating a preferred angle of trajectory of a physical instrument. Advantageously, by eliminating the need for physical needle guides and stands from the medical procedure, the system 100 is cost effective, minimizes medical waste, and provides practitioners with a complete and unencumbered angular guidance range for percutaneous surgical procedures.

[0046] In one example, the system 100 can also include a computer 116 having a processor (not shown) and a memory (not shown). The memory (not shown) can have non-transitory processor-executable instructions that direct the augmented reality display 108 to generate and display or depict the holographic needle guide 112 on the patient 106 based on a selection of the reference point 114 in the virtual ultrasound image 110 of a portion of the patient 106. In particular, the processor-executable instructions can allow the computer 116 to generate the holographic needle guide 112 on the patient 106 according to the method 200 shown. Figure 13 The method 200 is operated.

[0047] As Figures 1-2As shown, the augmented reality display 108 can be configured to depict the virtual ultrasound image 110 of the portion of the patient 106 as part of the virtual window 118 and / or as part of a virtual ultrasound projection 120 on the patient 106. In a more particular example, the augmented reality display 108 can include a head-mounted display 122 wearable by the user 104 in communication with the computer 116. In one more particular example, the computer 116 can be integrated into the head-mounted display 122 wearable by the user 104. More particularly, the head-mounted display 122 can be a Microsoft Hololens® with a tracking system (e.g., an inertial measurement unit), an integrated CPU and holographic processing unit, a camera, and a holographic projection lens as described in U.S. Patent Application Publication No. 2018 / 0303563 to West et al. (including the entirety of its disclosure defined therein hereby incorporated by reference), for example. Other suitable displays can be selected by those skilled in the art within the scope of the present disclosure.

[0048] The virtual ultrasound projection 120 generated by the computer 116 and depicted on the patient 106 can be further defined as a virtual display of the virtual ultrasound image 110 arranged adjacent to the ultrasound probe 124. In operation, the virtual ultrasound projection 120 can be linked to the ultrasound probe 124 such that the position of the virtual ultrasound projection 120 follows the position of the ultrasound probe 124. For example, the ultrasound probe 124 can be provided with a tracking device 125 (as shown, for example, in Figure 6 Figures 11A-11C and Figures 12A-12C More particularly, where the virtual ultrasound projection 120 is tagged at the virtual locked position (not shown), the computer 116 does not recognize movement of the user 104 as an instruction to adjust the position of the virtual ultrasound projection 120.

[0049] In a particular non-limiting example, the virtual ultrasound projection 120 can be displayed in operation directly above the ultrasound probe 124, for example, as shown in Figure 2 Advantageously, the virtual ultrasound projection 120 on the patient 106 can allow the user 104 to continue to view the patient 106 while monitoring the virtual ultrasound image 110.

[0050] With continued reference to Figure 2 , the user 104 can select the virtual ultrasound image 110 to identify the reference point 114. In a more particular example, as shown, for example, in Figure 6 the remote user 126 can also select the virtual ultrasound image 110 to identify the reference point 114. In an even more particular example, the remote user 126 can be located at a different location (not shown) relative to the location at which the user 104 is guiding the needle 102 percutaneously into the patient 106.​

[0051] like Figure 3 As shown, the angle or trajectory 128 of the holographic needle guide 112 is adjusted by the user 104. In a particular example, the trajectory 128 of the holographic needle guide 112 can also be adjusted by a remote user 126. In even more specific examples, the remote user 126 may be located at a different location (not shown) than where the user 104 guides the needle for percutaneous insertion into the patient 106. Advantageously, the user 104 and / or the remote user 126 can adjust the trajectory 128 of the holographic needle guide 112 to provide a less invasive path for the needle 102. Ideally, adjusting the trajectory 128 of the holographic needle guide 112 can more accurately and effectively avoid other non-target anatomical structures 130.

[0052] In specific cases, computer 116 can be configured to automatically or manually define modes or settings 132, 134 for selecting the trajectory 128 of the holographic needle guide 112 within the scope of this disclosure. More specifically, settings 132, 134 can be selected from: in-plane mode 132 (in...) Figure 3 (Displayed as a substantially vertical direction), out-of-plane mode 134 (in) Figure 3 The directions shown are those that are separated from the direction that is substantially perpendicular to the angle of trajectory 128, the freehand mode (not shown), and combinations thereof.

[0053] Settings 132, 134 are based on the angle of the trajectory 128 of the holographic needle guide 112 compared to the plane 136 associated with the patient 106. In a non-limiting example, such as Figure 3 , Figure 10A and Figure 10B As shown, when the patient 106 lies on the operating table, the plane 136 of the patient 106 can be substantially horizontal, i.e., parallel to the surface of the operating table. The in-plane mode 132 can be described as orienting the trajectory 128 of the holographic needle guide 112 substantially perpendicular to the plane 136 of the patient 106. The out-of-plane mode 134 can be described as automatically orienting the trajectory 128 of the holographic needle guide 112 to a predetermined or desired angle, rather than the substantially perpendicular angle of the in-plane mode 132. In cases where the computer 116 does not automatically orient the trajectory 128 of the holographic needle guide 112 to the desired angle, a manual mode (not shown) can be used. Instead, the manual mode (not shown) relies on the user 104 freely selecting the desired orientation for the trajectory 128 of the holographic needle guide 112.

[0054] In a more specific context, the holographic needle guide 112 may be depicted as a cylindrical or rod-shaped structure. The holographic needle guide 112 may dangle from a selected reference point 114 and extend outward from the patient 106 or through an external point 115 (e.g., ...). Figure 10A and Figure 10BThe external points 115 can be points on the plane 136, for example, points on the edge of a circle or ellipse on the plane 136 generally centered on the location of the ultrasound probe 124. The external points 115 can be selected by any suitable method, including automatic selection based on an algorithm configured to generate the optimal angle of approach for the needle 102, or by manual selection by the user 104.

[0055] In operation, the user 104 can select the holographic needle guide 112 by grasping, pinching, tapping, and / or holding the holographic needle guide 112. While grasping, pinching, and / or holding the holographic needle guide 112, the user 104 can adjust the trajectory 128 of the holographic needle guide 112 by moving their hand with the holographic needle guide 112 to the desired location. Depending on the selected reference point 114, the movement of the holographic needle guide 112 can be displayed as an arc. As Figures 10A-10B The movement of the holographic needle guide 112 can be related to the location of the ultrasound probe 124, as shown. A freehand mode (not shown) can further adjust on a three-dimensional setting, allowing the arc to be formed in a spherical pattern around the selected reference point 114. Additionally, the holographic needle guide 112 can be marked at a virtual locked position (not shown). More specifically, with the holographic needle guide 112 marked at the virtual locked position (not shown), the computer 116 does not recognize the movement of the user 104 as an instruction to adjust the location of the holographic needle guide 112. In the most specific case, the computer 116 can have the in-plane mode 132 as a default setting. Advantageously, the user 104 can select the desired setting 132, 134 based on the type of procedure being performed and the anatomy 130 of the patient 106 to more effectively set the trajectory 128 of the holographic needle guide 112. Within the scope of the present disclosure, the skilled artisan can select other suitable modes to set the trajectory 128 of the holographic needle guide 112.

[0056] As Figures 1-4 and Figure 6 The system 100 can also include an ultrasound probe 124, as shown. In a specific example, the system 100 can be configured to obtain the virtual ultrasound image 110 of the portion of the patient 106 from the ultrasound probe 124. In a more specific example, the system 100 can be configured to obtain the virtual ultrasound image 110 of the portion of the patient 106 from the ultrasound probe 124 in real-time. In an alternative specific example, the virtual ultrasound image 110 of the portion of the patient 106 can be pre-recorded.

[0057] In a particular example, the system 100 can also include a robotic arm (not shown). The robotic arm (not shown) can be configured to support each of the ultrasound probe 124 and the needle 102. In a more particular example, the remote user 126 can be able to move the robotic arm (not shown) by using the computer 116. In an even more particular example, the remote user 126 can be located at a different location (not shown) than the user 104 moves the robotic arm (not shown) to perform the percutaneous insertion of the needle 102 into the patient 106. Within the scope of the present disclosure, other suitable methods of remotely performing the percutaneous insertion of the needle 102 into the patient 106 can be selected by those skilled in the art.

[0058] In a particular example, the system 100 can include a tracking device (shown in Figures 11A-11C as 135). The tracking device can be configured to provide an augmented visualization of the anatomy 130 of the patient 106 and the needle 102. The tracking device can be placed on the patient 106, or placed on the needle 102, or placed on both. The tracking device (not shown) can be an infrared marker (not shown), an electromagnetic tracker (not shown), an image or model tracker (not shown), and / or an RFID tracker (not shown). As a non-limiting example, the electromagnetic tracking device (not shown) can be provided by the Polaris® tracking system available from Northern Digital Inc. As another non-limiting example, the infrared marker tracking device (not shown) can be used with the Stylus® tracking system available from Holo-Light Ltd. As a further non-limiting example, the image or model tracking device (not shown) can include the VisionLib® tracking system available from Visometry GmbH. Moreover, the RFID tracking device (not shown) can be used with RFID tags that are autoclavable, such as the Tag-it® tags available from Xerafy Singapore Pte. Ltd. The tracking system provides an augmented visualization of the anatomy 130 of the patient 106 and the needle 102. The tracking system can be used to provide a real-time visualization of the needle 102 as it is inserted into the patient 106. In a particular example, the tracking system can be used to provide a real-time visualization of the needle 102 as it is inserted into the patient 106, and to provide a real-time visualization of the needle 102 as it is moved within the patient 106. In a more particular example, the tracking system can be used to provide a real-time visualization of the needle 102 as it is inserted into the patient 106, and to provide a real-time visualization of the needle 102 as it is moved within the patient 106, and to provide a real-time visualization of the needle 102 as it is removed from the patient 106. In an even more particular example, the tracking system can be used to provide a real-time visualization of the needle 102 as it is inserted into the patient 106, and to provide a real-time visualization of the needle 102 as it is moved within the patient 106, and to provide a real-time visualization of the needle 102 as it is removed from the patient 106, and to provide a real-time visualization of the needle 102 as it is disposed of. The tracking system from Northern Digital Inc. As another non-limiting example, the infrared marker tracking device (not shown) can be used with the Stylus® tracking system available from Holo-Light Ltd. As a further non-limiting example, the image or model tracking device (not shown) can include the VisionLib® tracking system available from Visometry GmbH. Moreover, the RFID tracking device (not shown) can be used with RFID tags that are autoclavable, such as the Tag-it® tags available from Xerafy Singapore Pte. Ltd. The tracking system from Holo-Light Ltd. Non-limiting examples of image or model tracking devices (not shown) can include the VisionLib® tracking system available from Visometry GmbH. Moreover, non-limiting examples of RFID tracking devices (not shown) can be used with RFID tags that are autoclavable, such as the Tag-it® tags available from Xerafy Singapore Pte. Ltd. TM The tracking system from Holo-Light Ltd. Non-limiting examples of image or model tracking devices (not shown) can include the VisionLib® tracking system available from Visometry GmbH. Moreover, non-limiting examples of RFID tracking devices (not shown) can be used with RFID tags that are autoclavable, such as the Tag-it® tags available from Xerafy Singapore Pte. Ltd. The tracking system from Holo-Light Ltd. Non-limiting examples of image or model tracking devices (not shown) can include the VisionLib® tracking system available from Visometry GmbH. Moreover, non-limiting examples of RFID tracking devices (not shown) can be used with RFID tags that are autoclavable, such as the Tag-it® tags available from Xerafy Singapore Pte. Ltd.

[0059] Referring to Figures 11A-11C ​The tracking device 135 of system 100 may include at least one optical tracking marker disposed on the patient. The optical tracking marker is configured to track the position of the patient's body. Furthermore, the optical tracking marker may also be configured to spatially anchor operational information for viewing by the user via head-mounted displays 108, 122. For example, in the case where the operational projection is an ultrasound plane, the ultrasound plane can be spatially anchored to the patient's body via the optical tracking marker. Ideally, this allows the practitioner to set aside untracked instruments or needles 102 while allowing the ultrasound plane to remain anchored to the patient's body.

[0060] Non-limiting examples of optical tracking markers include passive and active markers. Passive markers may include retroreflective materials that reflect incident infrared light. Active markers may include infrared light-emitting diodes. However, it should be understood that other types of optical tracking markers may be employed by those skilled in the art within the scope of this disclosure.

[0061] Now for reference Figure 5 When the system 100 is equipped with a tracking device, the augmented reality display 108 can also be configured to depict a holographic error bar 137. The holographic error bar 137 can be further configured to indicate to the user 104 a deviation of a predetermined threshold from the variance of the position of the needle 102 relative to the trajectory 128 of the holographic needle guide 112. Non-limiting examples of the indication may include visual color changes, auditory sounds, visual signals, and / or vibrations. In a particular example, the display of the holographic needle guide 112 on the patient 106 may include a minimum range (not shown) and a maximum range (not shown) depending on the physical characteristics of the needle 102. As a non-limiting example, the holographic needle guide 112 may be adjustable and configured to depict the physical length and diameter of the needle 102 inserted into the patient 106. Advantageously, by providing the holographic error bar 137 and the physical limitations of the needle 102 in the display of the holographic needle guide 112, the user 104 can perform surgery more accurately, quickly, and confidently. Within the scope of this disclosure, those skilled in the art may use other methods to identify and alert the user 104 to a predetermined threshold of the variance of the position of the needle 102 relative to the trajectory 128 of the holographic needle guide 112.

[0062] like Figure 8 , Figure 9A and Figure 9B As shown, system 100 may include a needle insertion guide 138 for use in conjunction with the holographic needle guide 112 described herein. The needle insertion guide 138 may be a physical device that provides visual cues to the patient 106. The needle insertion guide 138 may be configured to indicate desired or predetermined needle insertion points 140, 148. In a particular example, such as Figure 9A and Figure 9BAs shown, the needle insertion guide 138 can be a physical template 142 placed on the patient 106 in a location proximate to the holographic needle guide 112. In a more particular example, the physical template 142 can include a strip 144 having reference marks that define possible insertion points 140 proximate to the reference marks (as shown in Figure 9B As shown, or a strip 145 having a plurality of holes 146 arranged in a linear row (as shown in Figure 9A As shown, or a strip 145 having a plurality of holes 146 arranged in a linear row (as shown in

[0063] In an alternative example, as shown in Figure 8 As shown, the needle insertion guide 138 can be disposed on or attached to the ultrasound probe 124. In the case where the needle insertion guide 138 is disposed on the ultrasound probe 124, the needle insertion guide 138 can be configured to provide a visual cue on the patient 106 and corresponding to the intended needle insertion point of the holographic needle guide 112.

[0064] As shown, the needle insertion guide 138 can be a physical template 142 placed on the patient 106 in a location proximate to the holographic needle guide 112. In a more particular example, the physical template 142 can include a strip 144 having reference marks that define possible insertion points 140 proximate to the reference marks (as shown in Figure 6 As shown, the needle insertion guide 138 can be a physical template 142 placed on the patient 106 in a location proximate to the holographic needle guide 112. In a more particular example, the physical template 142 can include a strip 144 having reference marks that define possible insertion points 140 proximate to the reference marks (as shown in

[0065] Figure 7 As shown, the needle insertion guide 138 can be a physical template 142 placed on the patient 106 in a location proximate to the holographic needle guide 112. In a more particular example, the physical template 142 can include a strip 144 having reference marks that define possible insertion points 140 proximate to the reference marks (as shown in

[0066] As shown, the needle insertion guide 138 can be a physical template 142 placed on the patient 106 in a location proximate to the holographic needle guide 112. In a more particular example, the physical template 142 can include a strip 144 having reference marks that define possible insertion points 140 proximate to the reference marks (as shown in Figure 13 ​As shown, the present technology includes a method 200 of using the holographic augmented reality ultrasound needle guide system 100 described herein. The method 200 can include a step 202 of providing an augmented reality display 108. The augmented reality display 108 can be configured to depict a virtual ultrasound image 110 of a portion of the patient 106. The augmented reality display 108 can also be configured to depict a holographic needle guide 112 on the patient 106 based on a selection of a reference point 114 on the virtual ultrasound image 110. The method 200 can also include a step 204 of providing the virtual ultrasound image 110 of the portion of the patient 106 in real-time using an ultrasound probe 124. Ideally, the real-time imaging provided by the ultrasound probe 124 can enable more accurate visualization of the anatomy 130 of the patient 106. Alternatively, the method 200 can include a step 206 of providing a pre-recorded virtual ultrasound image 110. Advantageously, where the virtual ultrasound image 110 is pre-recorded, the user 104 can not need to hold the ultrasound probe 124 while also adjusting the trajectory 128 of the holographic needle guide 112.

[0067] The method 200 can include a step 208 of selecting the reference point 114 in the virtual ultrasound image 110 of the portion of the patient 106. Subsequently, the method 200 can include a step 210 of displaying the holographic needle guide 112 on the patient 106 based on the selection of the reference point 114 on the virtual ultrasound image 110 of the portion of the patient 106.

[0068] With continued reference to Figure 13 , the method 200 can further include a step 212 of adjusting an angle of the trajectory 128 associated with the holographic needle guide 112 after displaying the holographic needle guide 112 on the patient 106. In particular examples, the computer 116 can include settings 132, 134 for automatically selecting the trajectory 128 of the holographic needle guide 112. The settings 132, 134 can be selected from an in-plane mode 132, an out-of-plane mode 134, and / or a freehand mode (not shown).

[0069] As Figure 13As shown, the method 200 can include a step 214 of marking the holographic needle guide 112 at a desired location after adjusting the angle of the trajectory 128 associated with the holographic needle guide 112. The holographic needle guide 112 can be marked at a virtual locked position (not shown) in situations where the user 104 can need both of their hands while performing an ultrasound-guided needle procedure. More specifically, with the holographic needle guide 112 marked at the virtual locked position (not shown), the computer 116 does not recognize movements of the user 104 as instructions to adjust the trajectory 128 of the holographic needle guide 112. In an even more specific example, the computer 116 can allow each of the reference point 114, the holographic needle guide 112, the virtual ultrasound projection 120, and combinations thereof to be marked at the virtual locked position (not shown). Advantageously, the marking feature enables the user 104 to set the needle 102 or ultrasound probe 124 down, then return and know where they want to insert the needle 102 based on the holographic needle guide 112 and the reference point 114.

[0070] As Figure 13 Further shown, the method 200 can include a step 216 of displaying a needle insertion guide 138 on the patient 106 to indicate a needle insertion point 140, 148 after displaying the holographic needle guide 112. Then, the method 200 can include a step 218 of tracking a position of the needle 102 in comparison to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. As Figure 5 Further shown, the method 200 can include a step 216 of displaying a needle insertion guide 138 on the patient 106 to indicate a needle insertion point 140, 148 after displaying the holographic needle guide 112. Then, the method 200 can include a step 218 of tracking a position of the needle 102 in comparison to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. As

[0071] Example

[0072] The system 100 and method 200 of the present disclosure can further be described as a unique combination of ultrasound technology and holography, which can be further illustrated in accordance with the following non-limiting examples described generally with reference to Figures 10A-10B 、 Figures 11A-11C and Figures 12A-12C

[0073] Needle guide object:

[0074] In a specific example, the needle guide is defined as a line between two points. The visual target is at the distal end of the line. The proximal end of the line also has a ghosted geometric shape, similar to a fixture that would connect a physical needle to a physical guide.

[0075] Interaction: ​

[0076] In certain examples, the target can not be interactive. The fixture geometry can be interactive and can only perform translation (i.e., no rotation or scaling). The fixture can support the near-far interaction defined by the Microsoft Mixed Reality Toolkit (MRTK). For “far interaction,” the user can select the fixture by pointing a hand ray at the handle and then performing a pinch-and-hold. At the pinch, the fixture can transition through a gesture. For “near interaction,” the system can support direct manipulation of the fixture via existing modes in the MRTK. The interaction state of the fixture can follow the MRTK modes, e.g., as described in the MRTK Interactable Function Overview (available at https: / / microsoft.github.io / MixedRealityToolkit- Unity / Documentation / README_Interactable.html View, last accessed November 27, 2020). The interaction states include: i) default (normal, observe); ii) focus (targeted, hover), e.g., the fixture can light up; and iii) active (pressed) (capture vs. uncapture) - the fixture will change color to blue.

[0077] Visual specifications for the needle guide:

[0078] In particular examples, the line can be rendered as a cylinder without end caps. This can also include a native line if rendering the cylinder is problematic. The material of the cylinder can be a white X-Ray shader. There can only be front-facing rendering on the cylinder. The target can be a billboard shape. The fixture can be a translucent green X-Ray shader, similar to a real-world plastic object. The scale of each needle guide feature can be as follows: i) target, one and a half (1.5) cm in diameter; ii) cylinder, one quarter (0.25) cm in diameter; and iii) fixture, approximately three (3) cm in length.

[0079] Needle guide targeting:

[0080] In other examples, the near cursor on the user’s index finger can be the default MRTK cursor. Targeting on the heads-up display (HUD) as shown in FIG. 6A and the flashlight ultrasonic plane as shown in FIG. 6B can both be possible. Targeting can support eye gaze, hand ray (far), and direct manipulation (near). When the HUD or flashlight plane is targeted / hovered, the cursor can visually change to the targeted / hovered state. The flashlight mode can toggle on and off. The HUD can toggle on and off. Figures 11A-11C Figures 12A-12C Targeting on the heads-up display (HUD) as shown in FIG. 6A and the flashlight ultrasonic plane as shown in FIG. 6B can both be possible. Targeting can support eye gaze, hand ray (far), and direct manipulation (near). When the HUD or flashlight plane is targeted / hovered, the cursor can visually change to the targeted / hovered state. The flashlight mode can toggle on and off. The HUD can toggle on and off.

[0081] The user can then submit the target location by one of the following: i) far interaction, where the user extends their arm, aims a hand ray, and AirTaps or speaks a voice command of “set target”; or ii) near interaction, where the user directly presses the HUD or flashlight ultrasonic plane with their index finger. ​

[0082] Upon submission of the target location, the following can occur. First, a visual indication can appear on the ultrasound plane (HUD or flashlight or both simultaneously) clearly indicating that an action (e.g. shockwave) has been received. Second, the submitted target visual on the HUD or flashlight plane can disappear immediately upon submission. Third, the target location on the HUD or flashlight plane can be converted to world space. Fourth, a needle guide can appear in the scene. Fifth, the distal end of the needle guide can be initialized at the target location. Sixth, the same visual indication (e.g. shockwave) can appear when the target is initialized in 3D space. Seventh, the needle guide’s orientation can be determined by the ultrasound plane and can be “in-plane” upon initialization. Eighth, the proximal fixation on the needle guide cylindrical shaft can be defaulted to be located “above” the transducer probe at approximately 4 cm (i.e. outside the patient’s body) so that the user can easily grasp it.

[0083] The user can be allowed to “reposition the target” at any time on the HUD or flashlight plane. If a target is submitted on the HUD, it can be automatically populated in the flashlight mode and vice versa. The user can lock and mark the target and the holographic needle guide in physical space and reference the relationship to the optical tracking markers on the patient (as shown in Figures 11A-11C and Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A-12C Figures 11A-11C Figures 12A- ) to account for movement between the patient and the ultrasound probe. The user can lock and mark the ultrasound probe’s location in physical space and reference the optical tracking and the relationship of the target and the holographic needle guide by referencing the optical markers on the patient.

[0084] Advantageously, the ultrasound needle guide system 100 and method 200 are cost effective, minimize medical waste, and provide practitioners with a complete and unencumbered perspective-guided range for percutaneous surgical procedures. Importantly, the system 100 and related method 200 involve holographic augmented reality and can be used with any type of ultrasound transducer.

[0085] Example embodiments are provided so as to be thorough and to convey the scope to those skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to one skilled in the art that specific details need not be employed, that example embodiments can be embodied in many different forms and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail in order to avoid obscuring the present disclosure. Some embodiments, materials, compositions, and methods can be practiced with equivalently elements and draws modifications and variations in the scope of the present technology.

Claims

1. A system for guiding a user to percutaneously insert a needle into a patient, comprising: an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient, and configured to depict a holographic needle guide on the patient based on a selection of a reference point in the virtual ultrasound image; a physical ultrasound probe; a physical needle insertion guide disposed on the physical ultrasound probe and configured to indicate at least one predetermined insertion point on the patient to the user, wherein the physical needle insertion guide is a physical template configured to be placed on the patient; and a computer having a processor and a memory, the memory including non-transitory processor-executable instructions that direct the augmented reality display to display a holographic needle guide on the patient based on a selection of a reference point in a virtual ultrasound image of a portion of the patient, the holographic needle guide showing a direction of insertion of a needle relative to the reference point, wherein a trajectory of the holographic needle guide is adjusted in real-time during a procedure by at least one of the user and a remote user, and the trajectory of the holographic needle guide is independent of a location of the needle, and after adjustment, the holographic needle guide is configured to be tagged at a virtual locked position. the augmented reality display is configured to depict the virtual ultrasound image of a portion of the patient as a member selected from the group consisting of: a virtual window, a virtual ultrasound projection on the patient, and combinations thereof.

2. The system of claim 1, wherein, the augmented reality display comprises a head-mounted display worn by the user.

3. The system of claim 1, wherein, the computer is integrated into a head-mounted display worn by the user.

4. The system of claim 3, wherein, the virtual ultrasound image is selectable to be identified by at least one of the user and a remote user located at a different location from the user to guide the needle to percutaneously insert into the patient.

5. The system of claim 1, wherein, the system is configured to obtain the virtual ultrasound image of a portion of the patient in real-time from the ultrasound probe, or to obtain the virtual ultrasound image of a portion of the patient as a pre-recorded image.

6. The system of claim 1, wherein, 7. The system of claim 1, further comprising a tracking device configured to provide an augmented visualization of at least one of an anatomical structure of the patient and the needle, the tracking device selected from the group consisting of: an electromagnetic tracker, an optical tracker, and combinations thereof; wherein the optical tracker comprises an infrared marker. the augmented reality display is configured to depict a holographic error bar adjacent to the holographic needle guide, the holographic error bar configured to alert the user that a deviation of a location of the needle relative to the holographic needle guide exceeds a predetermined threshold.

8. The system of claim 1, wherein, ​

Images