Holographic augmented reality ultrasound needle guide for insertion for percutaneous surgical procedures

Abstract

To provide a holographic augmented reality ultrasound needle guide system which has high cost effectiveness, minimizes medical waste, and optimizes a percutaneous surgical treatment.SOLUTION: According to the current invention, a system 100 includes an augmented reality display 108 wearable by a user configured to depict a virtual ultrasound image 110. The augmented reality display 108 is further configured to enable a user 104 to select a desired reference point 114 on the virtual ultrasound image 110. The system is configured to depict a holographic needle guide 112 on the basis of the selection of the desired reference point. The system is also configured to adjust a trajectory 114 of the holographic needle guide to avoid intersecting an undesired anatomical structure 130. The augmented reality display is further configured to stamp the holographic needle guide into a selectively locked trajectory 128.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 025,584, filed May 15, 2020, and U.S. Provisional Application No. 62 / 942,857, filed December 3, 2019, the entire disclosures of which are hereby incorporated by reference herein.

[0002] The present disclosure relates to holographic augmented reality applications, and more particularly to medical applications employing holographic augmented reality. [Background technology]

[0003] This section provides background information related to the present disclosure, but is not necessarily prior art.

[0004] Ultrasound guidance has become standard practice for many needle-based medical procedures, such as needle biopsies and local anesthesia. The use of ultrasound guidance has been shown to increase the safety and success of these procedures. Nevertheless, difficulties in positioning and orienting a needle can lead to misidentification of the needle tip, which can cause the needle to undesirably penetrate or miss certain anatomical structures or locations.

[0005] Certain techniques are sometimes used to help practitioners align the needle with confidence. These techniques range from simple mechanical devices to advanced automated needle detection software. One specific technique involves a mechanical ultrasound needle guide, a physical device attached to the ultrasound probe to guide the needle along a trajectory visible on the ultrasound image. Specifically, physical ultrasound needle guides are typically affixed to the ultrasound probe with a reusable bracket positioned on the probe's transducer. Needle guides may be preselected and used based on a fixed or designed angle depth. To accommodate different insertion trajectories, positionable needle guides can also be used, allowing selection between a limited number of angles, e.g., up to five different predetermined angle depths. Typically, these physical needle guides are removably attached to a reusable bracket coupled to the ultrasound probe.

[0006] Physical ultrasonic needle guides present certain limitations, including cost and limited reusability. In fact, most ultrasonic needle guides are designed for disposable use. Such physical ultrasonic needle guides may also require specialized ultrasound transducers designed for use with the needle guide or associated bracket. Even though a specific predetermined angulation depth may be selected, the practitioner is not afforded a complete and unlimited range of angulation possibilities with these physical needle guides.

[0007] Known needle guides are also vendor- and probe-specific and are typically limited to "in-plane" or "perpendicular to the plane" angles. These are often criticized by experienced clinicians, such as interventional radiologists, because, as noted above, they limit users to a single or few angles supported by the mechanical guide. Clinicians often desire the flexibility to move the probe independently of the guide, which requires them to orient the needle intraprocedurally out of plane from the probe for best visualization.

[0008] Holographic augmented reality technologies are finding wider use in healthcare applications to improve medical procedures, clinical outcomes, and long-term patient care. These augmented reality technologies are also useful, for example, to augment the real-world environment in a patient's medical setting with content-specific information, which could improve patient outcomes. For example, a practitioner could view additional information in the same field of view while performing a medical procedure, without the need for a change of gaze that could slow or reduce the efficiency of the procedure. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] U.S. Patent Application Publication No. 2018 / 0303563 Summary of the Invention

[0010] Therefore, there is a continuing need for ultrasonic needle guide systems and methods that are cost-effective, minimize medical waste, and provide practitioners with a full and unlimited range of angular guidance to optimize percutaneous surgical procedures. Desirably, the systems and methods involve holographic augmented reality and can be used with any type of ultrasound transducer.

[0011] In accordance with the present disclosure, a holographic augmented reality ultrasound needle guide system and method has surprisingly been discovered that is cost-effective, minimizes medical waste, provides the practitioner with a full and unlimited range of angle guidance to optimize percutaneous surgical procedures, and can be used with any type of ultrasound transducer.

[0012] In one embodiment, a holographic augmented reality ultrasound needle guide system for guiding percutaneous insertion of a needle into a patient by a user includes an augmented reality display configured to render a virtual ultrasound image of a site (i.e., a portion) of the patient. The augmented reality display is also configured to render a holographic needle guide on the patient based on selection of a reference point within the virtual ultrasound image.

[0013] In another embodiment, a method of using a holographic augmented reality ultrasound needle guide system may include providing an augmented reality display configured to depict a virtual ultrasound image of a region of a patient. The augmented reality display is also configured to depict a holographic needle guide on the patient based on a selection of a reference point within the virtual ultrasound image. The method may include selecting a reference point within the virtual ultrasound image of the region of the patient. Then, the method may include displaying a holographic needle guide on the patient based on the selection of the reference point within the virtual ultrasound image of the region of the patient. Thereafter, the method may include percutaneously inserting a needle along the trajectory of the holographic needle guide.

[0014] In further embodiments, the systems and methods of the present disclosure enable a holographic display of the intended needle trajectory by using spatial computing, augmented reality, and artificial intelligence (AI) to generate a holographic light beam that mimics the intended trajectory of a physical needle guide. Such systems and methods may be used with any augmented reality display and may optionally use electromagnetic or optical tracking. This allows the holographic needle guide to be strategically fitted to any ultrasound probe, adjusted to any desired angle, and sized to accommodate any desired needle or trocar size.

[0015] In certain embodiments, the systems and methods of the present disclosure may include a unique combination of holographic and ultrasound technologies. At least one reference point may be selected on a virtual ultrasound image, allowing a user to actively change the angle of the virtual / holographic needle guide generated by the system relative to the patient's anatomy. The system may include an otherwise conventional ultrasound probe, which may have known coordinate, gyro, and position sensors (e.g., using gyroscopes and accelerometers within the probe). The ultrasound images may include one or more pre-recorded ultrasound images or may include virtual ultrasound images acquired in real time.

[0016] Various embodiments of the present disclosure may include the following aspects: During operation, the needle guide, which would traditionally be a physical bracket, can be "ghosted" or superimposed in the field of view of a practitioner wearing a holographic visualization system, such as, as one non-limiting example, a Microsoft HoloLens® headset. This allows the practitioner to perform needle insertion at any desired angle and without the need for an additional disposable physical needle guide. Ultrasound or EM tracking of the needle may also be employed and conveyed to the practitioner through the holographic visualization system. The system may also generate error bars or associated zones of acceptable placement that can be associated with needle insertion at a particular procedure.

[0017] It should be appreciated that use of the disclosed systems and methods allows for improved needle visualization, reduced procedure time, more reliable clinical outcomes, and the desirable elimination of the need for any physical ultrasound needle guides, brackets, or operating room disinfection. Significant structural avoidance to minimize non-target injury is also provided. Advantageously, the disclosed systems and methods can be used for a wide variety of medical procedures, including, but not limited to, nerve blocks, local anesthesia, vascular access, biopsy, resection, endo-cavity, transvaginal, transrectal to out-of-plane, biplane, curved path, and in-plane needle guidance in a straight path at any variable or fixed angle. The systems and methods are also particularly well-suited for use in mammography and related procedures.

[0018] In yet another embodiment, the systems and methods of the present disclosure address the limitations of mechanical needle guides by providing a holographic needle guide that supports virtually any trajectory or angle, all easily achievable internally. The holographic needle guide can be viewed through a stereoscopic or stereoscopic head-mounted display, such as the Microsoft HoloLens® or other augmented reality device. The holographic needle guide is user-interactive and is not limited to a probe, but also has the ability to guide the proceduralist's needle on any user-defined trajectory to a user-defined destination or target.

[0019] It should be appreciated that the holographic needle guide provides superior guidance to mechanical guides and may replace them in the marketplace. Rather than attaching a physical guide to a probe, the user instead wears a mixed reality headset that powers the application of the present disclosure.

[0020] In certain embodiments, the system and method begin with the user selecting a needle guide target destination on the ultrasound plane. Once set, the holographic target is transformed into real-world space inside the patient. The complete needle guide is instantiated at this real-world location, and the user then begins positioning the holographic guide while simultaneously moving the ultrasound probe to generate any desired view.

[0021] Typically, the ultrasound plane is "swept" up and down the guide, providing visibility to the structures surrounding the needle guide in all directions. This sweeping is not possible with mechanical guides and is one of the primary reasons why mechanical guides have fallen out of favor with experienced practitioners.

[0022] The practitioner needs both hands when performing an ultrasound-guided needle procedure, thus enabling stamping of the holographic target, the needle guide, and the ultrasound probe position, allowing the practitioner to place and then return the tool (e.g., needle or probe) and know where they want to insert the needle based on the known and determined probe position and anatomical target.

[0023] Further areas of applicability will become apparent from the description provided herein. It should be understood that the description and specific examples are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure.

[0024] The drawings described herein are for illustrative purposes only of selected embodiments, not all possible implementations, and are not intended to limit the scope of the present disclosure. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view of a holographic augmented reality ultrasound needle guide system depicting an augmented virtual window and an augmented virtual ultrasound projection showing a virtual ultrasound scan viewable through a headset display during operation, according to one embodiment of the present disclosure. [Figure 2] 2 is a perspective view of the system shown in FIG. 1 depicting a user selecting a reference point on an augmented virtual window according to one embodiment of the present disclosure. [Figure 3] FIG. 3 is a perspective view of the system shown in FIGS. 1 and 2, depicting a holographic needle guide being displayed and a user adjusting the trajectory of the holographic needle guide, according to one embodiment of the present disclosure. [Figure 4] FIG. 4 is a perspective view of the system shown in FIGS. 1-3 depicting a user inserting a needle along a holographic needle guide, according to one embodiment of the present disclosure. [Figure 5]FIG. 5 is an enlarged perspective view of a user of the system taken at marker 5 in FIG. 4, shown from the perspective of a user wearing a headset display and inserting a needle along a holographic needle guide, according to one embodiment of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram of the system shown in FIGS. 1-5, according to another embodiment of the present disclosure. [Figure 7] FIG. 7 is a front view of the system shown in FIGS. 1-6, depicting the system disposed on a movable cart, according to a further embodiment of the present disclosure. [Figure 8] 1 is a schematic diagram of an ultrasound scan depicting an ultrasound scan with a selected reference point and displaying a virtual needle insertion point, according to one embodiment of the present disclosure; [Figure 9A] FIG. 1 is a top view of a needle insertion guide for determining a desired needle insertion point, according to one embodiment of the present disclosure, depicting a physical template with a hole positioned adjacent to an ultrasound probe for use with the needle insertion guide. [Figure 9B] FIG. 10 is a side view of a needle insertion guide for determining a desired needle insertion point according to another embodiment of the present disclosure, depicting a physical template with measurement marks positioned adjacent to an ultrasound probe for use with the needle insertion guide. [Figure 10A] 1A-1C are schematic diagrams illustrating a method for generating and moving a needle insertion guide according to the present disclosure. [Figure 10B] 1A-1C are schematic diagrams illustrating a method for generating and moving a needle insertion guide according to the present disclosure. [Figure 11A] FIG. 11 is a partial perspective view showing a step-by-step flow for needle guide targeting using a heads-up display method. [Figure 11B] FIG. 11 is a partial perspective view showing a step-by-step flow for needle guide targeting using a heads-up display method. [Figure 11C] FIG. 11 is a partial perspective view showing a step-by-step flow for needle guide targeting using a heads-up display method. [Figure 12A] FIG. 10 is a partial perspective view showing a step-by-step flow for needle guide targeting using the flashlight display method. [Figure 12B] FIG. 10 is a partial perspective view showing a step-by-step flow for needle guide targeting using the flashlight display method. [Figure 12C] FIG. 10 is a partial perspective view showing a step-by-step flow for needle guide targeting using the flashlight display method. [Figure 13] 1-12C, according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] The following description of technology is merely exemplary in nature of the subject matter, manufacture, and use of one or more inventions and is not intended to limit the scope, application, or use of any specific inventions claimed in this application or in other applications that may be filed claiming priority to this application or in patents issuing therefrom. With respect to the disclosed methods, the order of steps presented is exemplary in nature; thus, the order of steps may vary in various embodiments, including instances where certain steps may be performed simultaneously. As used herein, "a" and "an" indicate the presence of "at least one" of an item, and where possible, multiple such items may be present. Unless expressly indicated otherwise, all numerical quantities in this description will be understood to be modified by the word "about," and all geometric and spatial descriptors will be understood to be modified by the word "substantially" in describing the broadest scope of the technology. "About," when applied to a numerical value, indicates that the calculation or measurement allows for some slight imprecision in the value (somewhat approaching the exact value, approximately or reasonably close to the value, approximately). If for any reason the imprecision indicated by "about" and / or "substantially" is not otherwise understood in the art in this ordinary sense, then "about" and / or "substantially," as used herein, will at least indicate the variation that can result from ordinary methods of measuring or using such parameters.

[0027] The open-ended term "comprising," as synonymous with non-limiting terms such as including, containing, or having, is used herein to describe and claim embodiments of the present technology, and embodiments may alternatively be described as "consisting of" or "consisting essentially of." The present invention may also be described using more restrictive language such as "of." Thus, for any given example reciting a material, component, or process step, the present technology also specifically includes examples that consist of, or consist essentially of, such material, component, or process step, excluding (to consist of), and excluding (to consist essentially of), additional materials, components, or processes that affect the substantial nature of the example, even if such additional materials, components, or processes are not explicitly recited in this application. For example, a recitation of a composition or process reciting elements A, B, and C specifically contemplates examples consisting of, and consisting essentially of, A, B, and C, excluding element D, which may be recited in the art even if element D is not explicitly described herein as excluded.

[0028] As referred to herein, the disclosure of ranges is inclusive of endpoints unless otherwise specified, and includes all separate values ​​and subranges within the entire range. Thus, for example, a range of "from A to B" or "from about A to about B" includes A and B. The disclosure of values ​​and ranges of values ​​for a specific parameter (such as an amount, weight percentage, etc.) does not exclude other values ​​and ranges of values ​​useful herein. It is contemplated that two or more specific exemplary values ​​of a given parameter may define endpoints of a range of values ​​that may be claimed for the parameter. For example, if parameter X is exemplified herein as having a value A and similarly exemplified as having a value Z, it is contemplated that parameter X may have a range of values ​​from about A to about Z. Similarly, the disclosure of two or more ranges of values ​​for a parameter (whether such ranges are nested, overlapping, or distinct) is contemplated to encompass all possible combinations of value ranges that may be claimed using the endpoints of the disclosed ranges. For example, if a parameter X is exemplified herein as having a value in the range of 1 to 10, or 2 to 9, or 3 to 8, it is also contemplated that the parameter X may have other ranges of values, including 1 to 9, 1 to 8, 1 to 3, 1 to 2, 2 to 10, 2 to 8, 2 to 3, 3 to 10, 3 to 9, etc.

[0029] When an element or layer is referred to as being "on," "engaged to," "connected to," or "coupled to" another element or layer, it is directly on, engaged to, connected to, or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being "directly on," "directly engaged to," "directly connected to," or "directly coupled to" another element or layer, there may be no intervening elements or layers. Other words used to describe relationships between elements should be interpreted similarly (e.g., "between" vs. "directly between," "adjacent" vs. "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0030] Although the terms "first," "second," "third," and the like may be used herein to describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms are used only to distinguish one element, component, region, layer, or section from another region, layer, or section. When used herein, terms such as "first," "second," and other numerical terms do not imply a sequence or order unless clearly indicated by context. Thus, a first element, component, region, layer, or section discussed below may be referred to as a second element, component, region, layer, or section without departing from the teachings of the illustrative embodiments.

[0031] Spatial terms such as "inner," "outer," "beneath," "below," "lower," "above," "upper," and the like may be used herein for ease of description to describe the relationship of one element or feature to another element or feature, as shown in the figures. Spatial terms may be intended to encompass different orientations of the device during use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures were inverted, elements described as "below" or "beneath" the other element or feature would be oriented "above" the other element or feature. Thus, the illustrative term "below" can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or in other orientations) and the spatially related descriptions used herein interpreted accordingly.

[0032] 1-7 illustrate a system 100 for guiding the percutaneous insertion of a needle 102 by a user 104 into a patient 106 for a medical procedure. The system 100 includes an augmented reality display 108. The augmented reality display 108 is configured to render a virtual ultrasound image 110 of a region of the patient 106 in a mode known as a heads-up display or "HUD" mode. The augmented reality display 108 is also configured to render a holographic needle guide 112 on the patient 106 based on selection of a reference point 114 within the virtual ultrasound image 110 in a mode known as a "flashlight" mode. The holographic needle guide 112 may be virtually rendered in the form of, by way of non-limiting example, an elongated shaft, tube, or cylinder, each indicating a preferred angle of trajectory for the actual instrument. Advantageously, system 100 is cost-effective, minimizes medical waste by eliminating the need for actual guides and brackets from the medical procedure, and provides the practitioner with a complete and unlimited range of angle guidance for percutaneous surgical procedures.

[0033] In one example, the system 100 may further include a computer 116 having a processor (not shown) and a memory (not shown). The memory (not shown) may have non-transitory processor-executable instructions that direct the augmented reality display 108 to generate and display or render a holographic needle guide 112 on the patient 106 based on a selection of a reference point 114 in the virtual ultrasound image 110 of a region of the patient 106. In particular, the processor-executable instructions may enable the computer 116 to operate according to a method 200 as shown in FIG. 13 .

[0034] As shown in FIGS. 1-2 , the augmented reality display 108 can be configured to depict a virtual ultrasound image 110 of a region of the patient 106 as part of a virtual window 118 and / or as part of a virtual ultrasound projection 120 on the patient 106. In a more detailed example, the augmented reality display 108 can include a headset display 122 wearable by the user 104 in communication with the computer 116. In an even more detailed example, the computer 116 can be integrated into the headset display 122 wearable by the user 104. Even more particularly, the headset display 122 can be, for example, a Microsoft HoloLens® device having a tracking system (e.g., an inertial measurement unit), an integrated CPU and holographic processing unit, a camera, and a holographic projection lens, as described in U.S. Patent Application Publication No. 2018 / 0303563 to West et al., the entire disclosure of which, including definitions, is hereby incorporated by reference herein. Those skilled in the art can select other suitable displays within the scope of this disclosure.

[0035] The virtual ultrasound projection 120 generated by the computer 116 and rendered on the patient 106 may be further defined as a virtual display of the virtual ultrasound image 110 positioned adjacent to the ultrasound probe 124. During operation, the virtual ultrasound projection 120 may be linked to the ultrasound probe 124 such that the position of the virtual ultrasound projection 120 tracks the position of the ultrasound probe 124. For example, the ultrasound probe 124 may be provided with a tracking means 125 (shown in FIGS. 6, 11A-11C, and 12A-12C), such as an optical tracking cube. Alternatively, the virtual ultrasound projection 120 may be stamped (fixed or recorded) at a virtually locked position (not shown). More specifically, if the virtual ultrasound projection 120 is stamped at a virtually locked position (not shown), the computer 116 does not recognize the movement of the user 104 as an instruction to adjust the position of the virtual ultrasound projection 120.

[0036] In a detailed, non-limiting example, the virtual ultrasound projection 120 may be displayed directly on the ultrasound probe 124 during operation, for example, as shown in Figure 2. Advantageously, the virtual ultrasound projection 120 on the patient 106 may allow the user 104 to continue to view the patient 106 while monitoring the virtual ultrasound image 110.

[0037] 2, the virtual ultrasound image 110 may be selectable by the user 104 to identify the reference point 114. In a more detailed example, as shown in FIG. 6, for example, the virtual ultrasound image 110 may also be selectable by a remote user 126 to identify the reference point 114. In an even more detailed example, the remote user 126 may be located at a different location (not shown) relative to the location of the user 104 to guide the percutaneous insertion of the needle 102 into the patient 106.

[0038] As shown in FIG. 3 , the angle or trajectory 128 of the holographic needle guide 112 is adjustable by the user 104. In a more detailed example, the trajectory 128 of the holographic needle guide 112 may also be adjustable by a remote user 126. In an even more detailed example, the remote user 126 may be located at a different location (not shown) from the user 104 who guides the percutaneous insertion of the needle into the patient 106. Advantageously, the user 104 and / or the remote user 126 may adjust the trajectory 128 of the holographic needle guide 112 to provide a less invasive path for the needle 102. By adjusting the trajectory 128 of the holographic needle guide 112, other non-target anatomical structures 130 may desirably be avoided more precisely and efficiently.

[0039] In a particular case, the computer 116 may be configured within the scope of the present disclosure to automatically or manually define a modality or setting 132, 134 for selecting the trajectory 128 of the holographic needle guide 112. In a more particular case, the setting 132, 134 may be selected from the group consisting of an in-plane modality 132 (shown in FIG. 3 as a substantially vertical direction), an out-of-plane modality 134 (shown in FIG. 3 as a direction set away from the substantially vertical direction by the angle of the trajectory 128), a freehand modality (not shown), and combinations thereof.

[0040] The settings 132, 134 are based on the angle of the trajectory 128 of the holographic needle guide 112 relative to a plane 136 associated with the patient 106. In one non-limiting example, as shown in FIGS. 3, 10A, and 10B, the plane 136 of the patient 106 may be substantially horizontal when the patient 106 lies on an operating table, i.e., parallel to the surface of the table. An in-plane modality 132 may be described as orienting the trajectory 128 of the holographic needle guide 112 to be substantially perpendicular to the plane 136 of the patient 106. An out-of-plane modality 134 may be described as automatically orienting the trajectory 128 of the holographic needle guide 112 to a predetermined or desired angle other than the substantially perpendicular angle of the in-plane modality 132. A freehand modality (not shown) may be used if the computer 116 does not automatically orient the trajectory 128 of the holographic needle guide 112 to the desired angle, instead relying on the user 104 to freely select the desired orientation for the trajectory 128 of the holographic needle guide 112.

[0041] In an even more detailed example, the holographic needle guide 112 may be depicted as a cylinder or rod-shaped structure. The holographic needle guide 112 may be dependent on a selected reference point 114 and extend outward from the patient 106 to or through an external point 115 (shown in FIGS. 10A and 10B ). The external point 115 may be a point on the plane 136, such as a point on the periphery of a circle or oval on the plane 136 that is centered approximately at the location of the ultrasound probe 124. The external point 115 may be selected by any suitable method, including automatic selection based on an algorithm configured to generate an optimal approach angle for the needle 102, or by manual selection by the user 104.

[0042] During operation, the user 104 may select the holographic needle guide 112 by grasping, pinching, tapping, and / or holding the holographic needle guide 112. While grasping, pinching, and / or holding the holographic needle guide 112, the user 104 may adjust the trajectory 128 of the holographic needle guide 112 by moving the user's hand along with the holographic needle guide 112 to a desired position. The movement of the holographic needle guide 112 may be displayed as an arc dependent on a selected reference point 114. As shown in FIGS. 10A-10B, the movement of the holographic needle guide 112 may be correlated with the position of the ultrasound probe 124. A freehand modality (not shown) may be further adjustable in a three-dimensional setting, allowing for arcs to be created in a spherical pattern around the selected reference point 114. Additionally, the holographic needle guide 112 may be stamped in a virtually locked position (not shown). More specifically, if the holographic needle guide 112 is stamped in a virtually locked position (not shown), the computer 116 will not recognize the user's 104 movements as instructions to adjust the position of the holographic needle guide 112. In the most specific case, the computer 116 may have an in-plane modality 132 as a default setting. Advantageously, the user 104 may select the desired settings 132, 134 based on the type of procedure being performed and the anatomy 130 of the patient 106 to more efficiently set the trajectory 128 of the holographic needle guide 112. Those skilled in the art may select other appropriate modalities for setting the trajectory 128 of the holographic needle guide 112 within the scope of this disclosure.

[0043] 1-4 and 6, the system 100 may further include an ultrasound probe 124. In a detailed example, the system 100 may be configured to acquire a virtual ultrasound image 110 of a region of the patient 106 from the ultrasound probe 124. In a more detailed example, the system 100 may be configured to acquire the virtual ultrasound image 110 of the region of the patient 106 in real time from the ultrasound probe 124. In an alternative detailed example, the virtual ultrasound image 110 of the region of the patient 106 may be pre-recorded.

[0044] In a detailed example, the system 100 may also include a robotic arm (not shown). The robotic arm (not shown) may be configured to hold each of the ultrasound probe 124 and the needle 102. In a more detailed example, the remote user 126 may be able to move the robotic arm (not shown) by using the computer 116. In an even more detailed example, the remote user 126 may be located at a different location (not shown) from the user 104 who is moving the robotic arm (not shown) to perform the percutaneous insertion of the needle 102 into the patient 106. Those skilled in the art may select other suitable methods of remotely performing the percutaneous insertion of the needle 102 into the patient 106 within the scope of this disclosure.

[0045] In a detailed example, the system 100 may include a tracking means (shown as 135 in FIGS. 11A-11C ). The tracking means may be configured to provide enhanced visualization of the anatomy 130 of the patient 106 and the needle 102. The tracking means may be placed on the patient 106, on the needle 102, or both. The tracking means (not shown) may be an infrared marker (not shown), an electromagnetic tracker (not shown), an image or model tracker (not shown), and / or an RFID tracker (not shown). As a non-limiting example, an electromagnetic tracker (not shown) may be provided by an Aurora® tracking system, commercially available from Northern Digital Inc. As another non-limiting example, an infrared marker tracker (not shown) may be utilized with a Stylus XR® tracking system, commercially available from Holo-Light GmbH. A non-limiting example of an image or model tracking means (not shown) may include the VisionLib™ tracking system, commercially available from Visometry GmbH. Additionally, a non-limiting example of an RFID tracking means (not shown) may be utilized with autoclavable RFID tags, such as Xerafy® tags, commercially available from Xerafy Singapore Pte Ltd.

[0046] 11A-11C, the tracking means 135 of the system 100 may include at least one optical tracking marker positioned on the patient. The optical tracking marker is configured to track the position of the patient's body. Additionally, the optical tracking marker may be further configured to spatially fix the motion information for viewing by the user through the headset display 108, 122. For example, if the motion projection is an ultrasound plane, the ultrasound plane may be spatially fixed to the patient's body via the optical tracking marker. This may be desirable to allow the practitioner to set aside an untracked instrument or needle 102 while allowing the ultrasound plane to remain fixed to the patient's body.

[0047] Non-limiting examples of optical tracking markers include passive markers and active markers. Passive markers may be comprised of retro-reflective material that reflects incoming infrared light. Active markers may be comprised of infrared light emitting diodes. Nevertheless, it should be understood that one skilled in the art may utilize other types of optical tracking markers within the scope of this disclosure.

[0048] Referring now to FIG. 5 , if the system 100 is provided with tracking, the augmented reality display 108 may also be configured to depict a holographic error bar 137. The holographic error bar 137 may be further configured to alert the user 104 of deviations from a predetermined threshold in the position of the needle 102 compared to the trajectory 128 of the holographic needle guide 112. Non-limiting examples of the alert may include a visual color change, an audible sound, a visual signal, and / or a vibration. In a detailed example, the display of the holographic needle guide 112 on the patient 106 may include a minimum range (not shown) and a maximum range (not shown) depending on the physical characteristics of the needle 102. As a non-limiting example, the holographic needle guide 112 may be adjustable and configured to depict the physical length and diameter of the needle 102 being inserted into the patient 106. Advantageously, by providing the holographic error bar 137 and the physical limits of the needle 102 in the display of the holographic needle guide 112, the user 104 can perform the procedure more accurately, quickly, and confidently. Those skilled in the art may use other methods within the scope of this disclosure to identify and alert the user 104 when the difference in the position of the needle 102 compared to the trajectory 128 of the holographic needle guide 112 deviates from a predetermined threshold.

[0049] As shown in Figures 8, 9A, and 9B, the system 100 may include a needle insertion guide 138 for use in conjunction with the holographic need guide 112, as described herein. The needle insertion guide 138 may be a physical device that provides a visual cue on the patient 106. The needle insertion guide 138 may be configured to indicate a desired or predetermined needle insertion point 140, 148. In a detailed example, as shown in Figures 9A and 9B, the needle insertion guide 138 may be a physical template 142 placed on the patient 106 near the location of the holographic need guide 112. In a more detailed example, the physical template 142 may include a bar 144 with fiducial markings (shown in FIG. 9B) defining adjacent fiducials marking possible insertion points 140, or a bar 145 with multiple holes 146 arranged in a linear row (shown in FIG. 9A) defining possible insertion points 148 therein to provide the user with further visual cues of desired needle insertion points 140, 148 for use in conjunction with the holographic required guide 112 to achieve the desired trajectory (not shown) of the needle 102.

[0050] 8, the needle insertion guide 138 may be disposed on or attached to the ultrasound probe 124. When the needle insertion guide 138 is disposed on the ultrasound probe 124, the needle insertion guide 138 may be configured to provide a visual cue of the intended needle insertion point on the patient 106 and that corresponds with the holographic needle guide 112.

[0051] As shown in FIG. 6 , the various components of the system 100 may be interconnected in a variety of ways. Each of the ultrasound probe 124, the needle 102, and the needle insertion guide 138 may be in communication with the computer 116. The computer 116 may also be in communication with the augmented reality display 108. The computer 116 may also distribute and receive data from the remote user 126 through the remote user's 126 terminal or an external computer communicating with the computer 116 over a wide area network, such as the Internet. Communications may be provided through suitable wired or wireless technology means. An Internet-connected device (not shown) may also be utilized to further provide the aforementioned communications. Those skilled in the art may select other suitable methods for interconnecting and providing communications within the system 100 within the scope of this disclosure.

[0052] 7 , the various components of the system 100 may be provided together on a movable cart 150. The cart 150 may include each of the computer 116, the augmented reality display 108, and the ultrasound probe 124. The cart 150 may house the augmented reality display 108 in a case 152 for added protection from damage. If the computer 116 is provided on the cart 150, the computer 116 may also include an external server 154 and an external wireless router 156 with which the computer 116 communicates. Advantageously, the cart 150 may more easily allow the user 104 to transport the system 100 to the patient 106's location (not shown) for a medical procedure.

[0053] As shown in FIG. 13 , the present technology includes a method 200 of using the holographic augmented reality ultrasound needle guide system 100 described herein. The method 200 may include providing 202 an augmented reality display 108. The augmented reality display 108 may be configured to depict a virtual ultrasound image 110 of a region of the patient 106. The augmented reality display 108 may also be configured to depict a holographic needle guide 112 on the patient 106 based on selection of a reference point 114 on the virtual ultrasound image 110. The method 200 may also include providing 204 the virtual ultrasound image 110 of the region of the patient 106 in real time using an ultrasound probe 124. Desirably, real-time imaging provided by the ultrasound probe 124 may enable more accurate visualization of the anatomical structures 130 of the patient 106. Alternatively, the method 200 may include providing 206 a pre-recorded virtual ultrasound image 110. Advantageously, if the virtual ultrasound image 110 is pre-recorded, the user 104 is not required to hold the ultrasound probe 124 while also adjusting the trajectory 128 of the holographic needle guide 112 .

[0054] The method 200 may include selecting 208 a reference point 114 in the virtual ultrasound image 110 of the area of ​​the patient 106. Thereafter, the method 200 may include displaying 210 a holographic needle guide 112 on the patient 106 based on the selection of the reference point 114 on the virtual ultrasound image 110 of the area of ​​the patient 106.

[0055] 13 , the method 200 may further include a step 212 of adjusting an angle of a trajectory 128 associated with the holographic needle guide 112 after displaying the holographic needle guide 112 on the patient 106. In a detailed example, the computer 116 may include settings 132, 134 for automatically selecting the trajectory 128 of the holographic needle guide 112. The settings 132, 134 may be selected from an in-plane modality 132, an out-of-plane modality 134, and / or a freehand modality (not shown).

[0056] 13 , the method 200 may include a step 214 of stamping the holographic needle guide 112 at a desired position after adjusting the angle of the trajectory 128 associated with the holographic needle guide 112. If the user 104 may need both of their hands when performing an ultrasound-guided needle procedure, the holographic needle guide 112 may be stamped in a virtually locked position (not shown). More specifically, when the holographic needle guide 112 is stamped in a virtually locked position (not shown), the computer 116 does not recognize the user 104's movement as an instruction to adjust the trajectory 128 of the holographic needle guide 112. In an even more detailed example, the computer 116 may enable each of the reference point 114, the holographic needle guide 112, the virtual ultrasound projection 120, and combinations thereof to be stamped in a virtually locked position (not shown). Advantageously, the stamp configuration allows the user 104 to place and then return the needle 102 or ultrasound probe 124 and, based on the holographic needle guide 112 and reference point 114, know where the user 104 wants to insert the needle 102.

[0057] 13 , the method 200 may include a step 216 of displaying the needle insertion guide 138 on the patient 106 to indicate the needle insertion points 140, 148 after displaying the holographic needle guide 112. Next, the method 200 may include a step 218 of tracking the position of the needle 102 compared to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. As shown in FIG. 5 , if the position of the needle 102 is tracked, the method 200 may include a further step 220 of alerting the user 104 via the augmented reality display 108 of deviations from a predetermined threshold in the position of the needle 102 compared to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. Thereafter, the method 200 may include the step 222 of percutaneously inserting a needle along the trajectory 128 of the holographic needle guide 112 .

[0058] Examples The system 100 and method 200 of the present disclosure can be further described as a unique combination of ultrasound and holographic techniques, which can be further illustrated in view of the following non-limiting examples described generally with reference to Figures 10A-10B, 11A-11C, and 12A-12C.

[0059] "Needle Guide Object"

[0060] In a detailed example, a needle guide is defined as a line between two points. A visual target is placed at the distal end of the line. There is also a ghosted shape at the proximal end of the line, resembling a fixture for connecting a real needle to a real guide.

[0061] "Dialogue"

[0062] In certain instances, targets may not be interactable. Fixture shapes may be interactable and may only perform translation (i.e., no rotation or scaling). Fixtures may support both far and near interactions as defined by the Microsoft Mixed Reality Toolkit (MRTK). For "far interactions," the user may select a fixture by pointing a hand ray at the handle and then performing a pinch-and-hold. While pinched, the fixture may be transformed through hand gestures. For "near interactions," the system may support direct manipulation of the fixture via existing patterns in MRTK. The fixture's interaction state may follow MRTK patterns, for example, as described in the MRTK Interactable Feature Overview (found at https: / / microsoft.github.io / MixedRealityToolkit-Unity / Documentation / README_Interactable.html, last accessed November 27, 2020). Interaction states include: i) default (normal, observe), ii) focused (targeted, hover), e.g. the fixture may emit light, and iii) active (pressed) (grabbed vs. not grabbed) - the fixture changes color to blue.

[0063] "Visual specifications of needle guide"

[0064] In a detailed example, a line may be rendered as a cylinder with no end caps. This may also include the original line if there are issues with rendering the cylinder. The cylinder material may be a pale x-ray shader. There may be forward-facing render-only on the cylinder. The target may be a billboard shape. The fixture may be a translucent green x-ray shader, resembling a real-world plastic object. The scale of each of the needle guide configurations may be as follows: i) target, one and a half (1.5) cm diameter; ii) cylinder, one-quarter (0.25) cm diameter; and iii) fixture, approximately three (3) cm in length.

[0065] "Needle-Guided Targeting"

[0066] In yet another example, a nearby cursor on the user's index finger may be the default MRTK cursor. Targeting may be possible in both a heads-up display (HUD) as shown in FIGS. 11A-11C and a flashlight ultrasound plane as shown in FIGS. 12A-12C. Targeting may support gaze, hand ray (far), and direct manipulation (near). When the HUD or flashlight plane is targeted / hovered, the cursor may visually change to a targeting / hover state. Flashlight mode can be toggled on and off. The HUD can be toggled on and off.

[0067] The user can then commit the target location by one of the following: i) a far interaction, where the user extends their arm, points a hand ray, and AirTaps or issues a "target" voice command, or ii) a near interaction, where the user presses their index finger directly onto the HUD or flashlight ultrasound plane.

[0068] Upon committing the target location, the following may occur. First, a visual indication (HUD or flashlight, or both simultaneously) may occur on the ultrasound plane, revealing that the action has been received (e.g., a shock wave). Second, the committed target visual on the HUD or flashlight plane may disappear immediately upon commitment. Third, the target location on the HUD or flashlight plane may be transformed into world space. Fourth, a needle guide may appear in the scene. Fifth, the distal end of the needle guide may be initialized at the target location. Sixth, the same visual indication (e.g., a shock wave) may occur once the target is initialized in 3D space. Seventh, the orientation of the needle guide may be determined by the ultrasound plane and may be "in-plane" when initialized. Eighth, the proximal fixture on the cylindrical shaft of the needle guide may be approximately 4 cm "above" (i.e., outside the patient) of the transducer probe by default, allowing the user to easily grasp it.

[0069] The user can be enabled to "re-target" on the HUD or flashlight plane at any time. Once a target is committed on the HUD, it can be automatically populated in flashlight mode, and vice versa. The user can lock and stamp the target and holographic needle guide in physical space and reference relative to the optical tracking markers on the patient (shown in Figures 11A-11C and 12A-12C) to account for motion between the patient and the ultrasound probe. The user can lock and stamp the position of the ultrasound probe and holographic needle guide in physical space and reference relative to the optical tracking and targeting by referencing the optical markers on the patient.

[0070] Advantageously, the ultrasonic needle guide system 100 and method 200 are cost-effective, minimize medical waste, and provide practitioners with a complete and unlimited range of angle guidance for percutaneous surgical procedures. Importantly, the system 100 and associated method 200 involve holographic augmented reality and can be used with any type of ultrasound transducer.

[0071] The example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. Numerous specific details are set forth, such as examples of specific components, devices, and methods, to provide a thorough understanding of the examples of the present disclosure. It will be apparent to those skilled in the art that specific details need not be utilized, that the example embodiments can be embodied in many different forms, and should not be construed as limiting the scope of the present disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known techniques have not been described in detail. Equivalent changes, modifications, and variations of some examples, materials, compositions, and methods may be made within the scope of the present technology, with substantially similar results.

[0072] Various examples of aspects of the present disclosure are described below as numbered clauses (1, 2, 3, etc.) for convenience, and are provided as examples and not as limitations on the subject technology. [Article 1] 1. A system for guiding percutaneous insertion of a needle into a patient by a user, comprising: The system includes an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient and configured to depict a holographic needle guide on the patient based on a selection of a reference point within the virtual ultrasound image. [Clause 2] The system described in clause 1, wherein the augmented reality display is configured to depict the virtual ultrasound image of the portion of the patient as a member selected from the group consisting of a virtual window, a virtual ultrasound projection on the patient, and combinations thereof. [Article 3] The system described in clause 1, further comprising a computer including a processor and memory, the memory including non-transitory processor-executable instructions that instruct the augmented reality display to display the holographic needle guide on the patient based on selection of the reference point within the virtual ultrasound image of the portion of the patient. [Article 4] 4. The system of clause 3, wherein the augmented reality display comprises a headset display wearable by the user. [Article 5] 5. The system of clause 4, wherein the computer is integrated with the headset display wearable by the user. [Article 6] 10. The system of claim 1, wherein the virtual ultrasound image is selectable to identify the reference point by at least one of the user and a remote user who guides the percutaneous insertion of the needle into the patient at a location different from the user. [Article 7] 7. The system of clause 6, wherein the trajectory of the holographic needle guide is adjustable by at least one of the user and the remote user. [Article 8] 10. The system of claim 1, further comprising an ultrasound probe. [Article 9] 9. The system of clause 8, configured to acquire the virtual ultrasound image of the portion of the patient from the ultrasound probe in real time or as a pre-recorded image. [Article 10] The system described in clause 8, further comprising a needle insertion guide configured to indicate to the user at least one predetermined insertion point on the patient, the needle insertion guide being a physical template configured to be placed on the patient. [Article 11] The system of clause 10, wherein the needle insertion guide is disposed on the ultrasound probe and the needle insertion guide is configured to provide a virtual needle insertion point as the at least one predetermined insertion point on the patient. [Article 12] The system described in clause 1, further comprising a tracking means configured to provide advanced visualization of at least one of the patient's structure and the needle, the tracking means being selected from the group consisting of an infrared marker, an electromagnetic tracker, an optical tracker, and combinations thereof. [Article 13] The system described in clause 1, wherein the augmented reality display is configured to depict a holographic error bar adjacent to the holographic needle guide, and the holographic error bar is configured to alert the user to deviations from a predetermined threshold in the difference in the position of the needle relative to the trajectory of the holographic needle guide. [Article 14] 1. A method for performing a surgical procedure involving percutaneous insertion of a needle into a patient by a user, comprising: providing an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient, the augmented reality display configured to depict a holographic needle guide on the patient based on a selection of a reference point within the virtual ultrasound image; selecting the reference point within the virtual ultrasound image of the portion of the patient; displaying the holographic needle guide on the patient based on the selection of the reference point in the virtual ultrasound image of the portion of the patient; percutaneously inserting the needle along the trajectory of the holographic needle guide; A method comprising: [Article 15] 15. The method of clause 14, further comprising providing the virtual ultrasound image of the portion of the patient using an ultrasound probe, either in real time or as a pre-recorded image. [Article 16] 15. The method of clause 14, further comprising the step of adjusting an angle of a trajectory associated with the holographic needle guide after displaying the holographic needle guide on the patient. [Article 17] 17. The method of claim 16, further comprising the step of stamping the holographic needle guide at a desired location after the step of adjusting the angle of a trajectory associated with the holographic needle guide. [Article 18] 15. The method of clause 14, further comprising the step of tracking the position of the needle compared to the trajectory of the holographic needle guide after inserting the needle along the holographic needle guide. [Article 19] The method of clause 14, further comprising the step of alerting the user via the augmented reality display of deviations of a predetermined threshold in the difference in the position of the needle compared to the trajectory of the holographic needle guide after inserting the needle along the holographic needle guide. [Article 20] 15. The method of clause 14, further comprising the step of providing a physical device for displaying a visual cue on the patient to indicate a needle insertion point after displaying the holographic needle guide.

Claims

1. 1. A system for guiding percutaneous insertion of a needle into a patient by a user, comprising:

1. A system comprising: an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient; and configured to depict a holographic needle guide on the patient based on selection of a reference point within the virtual ultrasound image.

Images