Multi-scale Ultrasound Tracking and Display

Abstract

In one embodiment, the system receives data from a first electromagnetic sensor coupled to a head mounted display (HMD) and detects an electromagnetic field generated by an electromagnetic reference source coupled to an ultrasound probe. The system receives data from a second electromagnetic sensor coupled to a medical instrument and detects the electromagnetic field. The system determines a position of the HMD relative to the ultrasound probe. The system determines a position of the medical instrument relative to the ultrasound probe. The system generates a visualization of a path of the medical instrument oriented relative to an ultrasound image plane. The system provides a graphic for display by the HMD to a user wearing the HMD, the graphic including the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane.

Description

[Technical field]

[0001] The present disclosure relates generally to the display and annotation of information during procedures involving ultrasound within a medical environment. [Background technology]

[0002] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 186,393, filed May 10, 2021, which is incorporated by reference herein in its entirety for all purposes.

[0003] In medical procedures involving ultrasound imaging, one or more operators working in a controlled or sterile environment combine sensed image information with directly observable physical information to complete the medical procedure. The translation, combination, and interpretation of directly and indirectly observable procedural information presents obstacles to the efficient and effective completion of the medical procedure.

[0004] In some cases, the performing physician must maintain asepsis during the procedure. In some cases, with or without asepsis, the physician must maintain direct physical control of ultrasound and other procedure instruments during the medical procedure. Setup, configuration, control, and / or display aspects of the medical procedure information may need to be accessible during some or all stages of the medical procedure. The requirement of sterility or direct control of instruments may impede the ability of the performing personnel to perform critical tasks once the procedure has begun without stopping the procedure and / or without eliciting the assistance of additional personnel to perform tasks under the guidance of the performing personnel using traditional medical information system interfaces, e.g., keyboard, mouse, joystick, or physical buttons. Summary of the Invention

[0005] An embodiment of the present invention provides a system for tracking, organizing, and displaying any number of image data with any number of instrument information. In one embodiment, the method includes receiving data from a first electromagnetic sensor coupled to a head mounted display (HMD) and detecting an electromagnetic field generated by an electromagnetic reference source coupled to an ultrasound probe. The method further includes receiving data from a second electromagnetic sensor coupled to the medical instrument and detecting the electromagnetic field. The method further includes determining a position of the HMD relative to the ultrasound probe by processing the data from the first electromagnetic sensor. The method further includes determining a position of the medical instrument relative to the ultrasound probe by processing the data from the second electromagnetic sensor. The method further includes generating a visualization of a path of the medical instrument oriented relative to an ultrasound image plane based at least in part on the position of the HMD and the position of the medical instrument. The method further includes providing a graphic for display by the HMD to a user wearing the HMD, the graphic including the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane.

[0006] In one embodiment, the method further includes receiving data from a third electromagnetic sensor coupled to the HMD and detecting an electromagnetic field generated by an electromagnetic reference source coupled to the ultrasound probe, and determining an orientation of the HMD based on a differential signal between the first and second magnetic sensors by processing the data from the first and third electromagnetic sensors. In one embodiment, the orientation of the HMD is further determined based on data captured by an inertial measurement unit of the HMD.

[0007] In one embodiment, the method further includes determining an orientation of the medical instrument by aggregating data captured by an inertial measurement unit of the medical instrument and data from a second electromagnetic sensor.

[0008] In one embodiment, the method further includes determining a position of the ultrasound probe by processing image data captured by an image sensor in the HMD, and the graphic is displayed to the user at an offset from the position of the ultrasound probe.

[0009] In one embodiment, the method further includes determining an offset distance between the tip of the medical instrument and a position of a second electromagnetic sensor coupled to the medical instrument, the visualization being generated based on the offset distance.

[0010] In one embodiment, the method further includes determining a first transformation for mapping a first coordinate system of the medical instrument to a second coordinate system of the ultrasound probe, and determining a second transformation for mapping a third coordinate system of the HMD to the second coordinate system of the ultrasound probe, and the visualization is generated using the first transformation and the second transformation.

[0011] In another embodiment, a system includes an electromagnetic reference source coupled to an ultrasound probe and configured to generate an electromagnetic field, a first electromagnetic sensor coupled to a head mounted display (HMD) and configured to detect the electromagnetic field, a second electromagnetic sensor coupled to a medical instrument and configured to detect the electromagnetic field, and a non-transitory computer readable storage medium storing instructions, which when executed by one or more processors cause the one or more processors to determine a position of the HMD relative to the ultrasound probe by processing data from the first electromagnetic sensor, determine a position of the medical instrument relative to the ultrasound probe by processing data from the second electromagnetic sensor, generate a visualization of a path of the medical instrument oriented relative to an ultrasound image plane based at least in part on the position of the HMD and the position of the medical instrument, and provide a graphic displayed by the HMD to a user wearing the HMD, wherein the image comprises the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane.

[0012] In one embodiment, the system further comprises an ultrasound probe, an HMD, and a medical instrument. In one embodiment, the first electromagnetic sensor and the second electromagnetic sensor are wireless. In one embodiment, the medical instrument is a needle, a cannula, a biopsy device, or a cauterization device.

[0013] In another embodiment, a method includes receiving image data for an ultrasound image plane. The method further includes determining a position of a medical instrument relative to the ultrasound image plane. The method further includes determining an orientation of the medical instrument relative to the ultrasound image plane. The method further includes determining a trajectory of the medical instrument based on the position and orientation of the medical instrument. The method further includes displaying, by a head mounted display (HMD), an instrument path indicative of the trajectory of the medical instrument. The method further includes determining, based on an updated position of the medical instrument along the trajectory, that the medical instrument has reached an intersection on the ultrasound image plane. The method further includes displaying, by the HMD, an updated version of the instrument path indicative of the medical instrument having reached the intersection on the ultrasound image plane.

[0014] In various embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform any of the steps of the methods described herein. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 illustrates an exemplary system environment of a processing system for ultrasound imaging and device tracking in accordance with various embodiments. [Diagram 2] FIG. 1 illustrates an exemplary graphical user interface for ultrasound imaging, according to various embodiments. [Diagram 3] FIG. 1 illustrates sensor locations on a head mounted display in various embodiments. [Figure 4] FIG. 2 is a block diagram of the system environment of FIG. 1 according to various embodiments. [Diagram 5] FIG. 2 is a data flow diagram of a processing system according to various embodiments. [Figure 6] 1A-1C illustrate exemplary graphical user interfaces for navigating instruments according to various embodiments. [Figure 7] 11A-11C illustrate additional exemplary graphical user interfaces for navigating an instrument according to various embodiments. [Figure 8] FIG. 1 illustrates a head-up display for navigating an instrument in accordance with various embodiments. [Figure 9] 1 is a flowchart of a process for tracking an instrument using multiple sensors according to various embodiments. [Figure 10] 1 is a flowchart of a process for navigating an instrument according to various embodiments.

[0016] The drawings depict embodiments of the present invention for purposes of example only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be utilized without departing from the principles of the present invention as described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] I. System Environment FIG. 1 illustrates an exemplary system environment of a processing system for ultrasound imaging, according to various embodiments. FIG. 2 illustrates an exemplary graphical user interface for ultrasound imaging, according to various embodiments. In various embodiments, the system includes one or more of a head mounted display (HMD) device 100, an instrument 110 (e.g., a needle, a cannula for inserting a device into a blood vessel, a biopsy device, or an ablation device), an ultrasound probe 120, and an optional display 130. The system may also include one or more of a user input method, any number of configurable ultrasound information sources, any number of configurable instrument tracking information sources, a means of communication between devices, a processor, and a storage device. The means of communication include any number of combinations of wireless and wired communication.

[0018] The ultrasound information source may be from multiple physical devices or from the same physical device from different times, configurations, and positions during a procedure. Image information may also be derived from other imaging modalities such as computed tomography (CT) or magnetic resonance imaging (MRI). In various embodiments, the ultrasound information source is an ultrasound probe 120.

[0019] The tracking information sources include local or global positioning sensors, including electromagnetic (EM), optical, ultrasonic, inertial, magnetometer, accelerometer, or Global Positioning System (GPS) sensors. In various embodiments, the electromagnetic sources may be coupled (e.g., rigidly coupled) to the ultrasound probe 120, and the electromagnetic sensors may be coupled (e.g., rigidly coupled) to the HMD 100 and instrument 110. As a result, the processing system can use data from the electromagnetic sensors to track the position and orientation of the HMD 100 and instrument 110 relative to the electromagnetic source of the ultrasound probe 120.

[0020] The HMD 100 provides a means for a user to view and interact with ultrasound image information or instrument information that may be presented to the user as a heads-up display or an augmented reality display. An example of an HMD is Microsoft's HOLOLENS.

[0021] Display of treatment information is accomplished through the HMD 100 and may be displayed as a projection 200 at the location of the information source in response to the tracking of the processing system. In the example shown in FIG. 2, the ultrasound information is displayed in a projection 200 of the tip of the ultrasound probe in the tracked ultrasound image plane ("probe tip display"). In another embodiment, the ultrasound information may also be displayed in the same orientation (or in a rotated configuration) with a configurable offset from the tool tip to avoid obscuring the physician's working area ("probe lock display"). Furthermore, any additional tracked instruments may be displayed with a tracked orientation and position ("pose") relative to the sensed ultrasound. This additional instrument information includes the current direction and position, the projected direction and position if advanced along the current trajectory, the distance to the instrument's intersection, and the angle of the intersection in response to the tracking information of the system.

[0022] In another embodiment, ultrasound and instrument information may be displayed in response to the tool tip position in a fixed orientation relative to the wearer ("billboarded"). In the example shown in FIG. 2, the HMD 100 presents the billboard display to the user as a heads-up display 210. In one embodiment, the 3D needle tip pose is displayed in 3D directly overlaid on the ultrasound image, intuitively conveying the needle's distance, size, and angle relative to the image plane. The billboard display can also show the projected needle trajectory and intersection with the image plane. In another embodiment, 2D or 3D annotations are overlaid on the image plane, translating the needle's distance to the image plane in the image, which is further described with respect to FIGS. 6-8.

[0023] In some embodiments, ultrasound and instrument pose, distance, and / or intersection information may be fixed to the virtual environment of the HMD 100 via conventional techniques for rendering 3D content to a 2D display, or may additionally be displayed on a physical display device 130 in the medical environment. The tracking information of the HMD 100, instrument 110, and ultrasound probe 120 may be used to automatically reorient or flip the images to display in the correct perspective in billboard, probe tip, and probe lock views.

[0024] Tracking of the device is achieved by a total combination of multiple sources of tracking information of different spatial and temporal resolutions. The device includes an HMD 100, an ultrasound image source (e.g., an ultrasound probe 120), and one or more medical instruments 110. Each tracking source provides any number of tracking information updates in position, orientation, and corresponding velocity and acceleration at different resolutions and different update rates. In one embodiment, different electromagnetic tracking sensors are attached to the HMD 100 and the instrument 110, and an electromagnetic reference source is attached to the ultrasound probe 120. The distance between the ultrasound reference source on the ultrasound probe 120 and the instrument 110 is reduced or minimized to improve accuracy between the ultrasound source and the HMD, for example, due to lower noise in the EM sensor data. Any number of additional electromagnetic sensors or reference sources may provide additional sources of tracking information and improve tracking accuracy.

[0025] 3 illustrates sensor locations on the head mounted display 100 in various embodiments. In some embodiments, one or more electromagnetic sensors or antennas are coupled to the HMD 100 to improve the processing system's determination of the accuracy of the angle and distance between the HMD 100 and an electromagnetic reference source. In the example shown in FIG. 3, the HMD 100 includes a right electromagnetic sensor 300 and a left electromagnetic sensor 310. Alignment and calibration of tracking systems of different spatial and temporal resolutions is achieved by calibration via rigid body transformations or reference data sets of the different sensors.

[0026] In one embodiment, the multiple sensors are fixed to a rigid body with known or measurable distances and orientations between the sensors, allowing for the combination of information from each measurement frame of reference. One example of this is a rigid body transformation from an electromagnetic sensor attached to the HMD 100 to the camera measurement frame of reference. Another example of this is an optical tracking marker (e.g., a QR code of known scale) fixed to the electromagnetic sensor or fiducial. Another example of this is the detection and alignment of an instrument (e.g., a needle) and the corresponding electromagnetic sensor pose in the ultrasound image plane. In some embodiments, the HMD 100 provides time-of-flight cameras, simultaneous localization and mapping, and native spatial tracking from one or more electromagnetic sensors.

[0027] II. Data Processing 4 is a block diagram of the system environment of FIG. 1 in various embodiments. The HMD 100 includes multiple sensors for local-scale alignment, including any number of 3D image sensors (e.g., depth cameras and LiDAR), optical image sensors (2D cameras), inertial sensors, and electromagnetic sensors. The ultrasound probe 120 includes an ultrasound image sensor, an electromagnetic reference source, and an optional inertial sensor. The instrument 110 includes an optional inertial sensor and one or more electromagnetic sensors. In some embodiments, the processing system 400 uses EM sensor data from one or more electromagnetic reference sources different from the electromagnetic reference source coupled to the ultrasound probe 120. For example, the separate electromagnetic reference source may be a fixed source in the system environment.

[0028] The processing system 400 uses the calibration and filtering data to combine and calculate a local scale transformation for each device (i.e., HMD 100, ultrasound probe 120, and instrument 110). The processing system 400 further combines multiple device estimates to update each local device, device-to-device, and sensor reference frame transformation as well as calibration and filtering parameters. By using different filters, the processing system 400 can reduce local scale noise (at the device level) and global scale noise, which allows the processing system 400 to distinguish between measurement errors and actual movement of devices such as the HMD 100 and instrument 110. Two exemplary calibration processes include geometric constraint calibration and hand-eye calibration.

[0029] Geometric constraint calibration uses known configurations such as rotation of the instrument about a pivot point. In various embodiments, the processing system 400 uses geometric constraint calibration to account for an offset distance between the position of the tip of the instrument 110 and the position of an EM sensor coupled to the instrument 110. The offset exists because the EM sensor is not necessarily coupled to the tip that is being tracked for the medical procedure.

[0030] Matching of the surface of the 3D information can be achieved using matching algorithms such as iterative closest point (IPC) or can be registered from 2D to 3D using statistical matching metrics such as mutual information on the rendered 3D data or feature-based approaches such as scale invariant feature transform (SIFT) or speed up robust features (SURF). Position and orientation information can be interpolated using information from different time rates using filtering techniques such as Kalman filtering or particle filtering. Estimated parameters from filtering techniques (e.g., unscented Kalman filtering) can be used to estimate and correct rigid body transformations between sensors of the same modality (e.g., electromagnetic sensor to electromagnetic sensor) or between modalities (e.g., electromagnetic source reference to optical tracker reference).

[0031] A hand-eye calibration (also called "robot-world" or "robot-sensor" calibration) algorithm may also refine the updated estimates to update the solution for the moment arm between each sensor. In various embodiments, processing system 400 uses hand-eye calibration to map a coordinate system to a different coordinate system. Because processing system 400 receives data from multiple sensors and devices with different coordinate systems, processing system 400 performs one or more mappings to aggregate data from multiple sources.

[0032] In hand-eye calibration, the processing system 400 can process each parameter and transformation estimate using a filtering algorithm or an ensemble of filtering algorithms to smooth the response as the transformation is updated. These can include linear quadratic estimators, extended Kalman filters, and unscented Kalman filters, either individually or in combination, with forward prediction using particle filters. These components provide a current best estimate model of the sensed objects within the current estimated model of the environment.

[0033] Extraction of the ultrasound image and the instrument position in the instrument EM pose can be used to update the calibration filter between the image plane and the instrument position. Furthermore, this information can be used to update parameters of the ultrasound image including frequency, depth, and tissue density related to the speed of propagating ultrasound waves.

[0034] In various embodiments, the processing system 400 uses tissue depth to account for any measurement errors due to variations in the speed of sound through different media. Because ultrasound images are based on the travel time of ultrasound propagating through a medium (e.g., tissue), different tissue depths or densities can affect the instrument intersection points predicted by the processing system 400. During calibration, the processing system 400 can use correction factors associated with a particular medium or density to reduce any speed of sound measurement errors.

[0035] 5 is a data flow diagram of the processing system 400 in various embodiments. The processing system 400 includes a global scale estimator 500 that aggregates image alignment data, 3D alignment data, and image-based detection and tracking data from multiple devices, namely, the HMD 100 and the ultrasound probe 120. The ultrasound probe 120 has a local scale estimator 510 that processes data from one or more of an EM reference source, an IMU, an optical alignment target, and an ultrasound image sensor. The HMD 100 has a local scale estimator 520 that processes data from one or more of an EM sensor, an IMU, a 3D image sensor, and other types of image sensors. By using the global scale estimator 500 in combination with the local scale estimators 510 and 520, the processing system 400 can simultaneously solve for different positions and orientations of multiple devices relative to each other. For example, the processing system 400 determines the position and orientation of the HMD 100 relative to the ultrasound probe 120. Additionally, the processing system 400 determines the position and orientation of the one or more instruments 110 relative to the ultrasound probe 120 .

[0036] In some embodiments, the processing system 400 correlates the point data from each (local) tracking system using a matching method such as iterative closest point (IPC), in which the processing system 400 determines the point correspondence between the data using the currently best known time correspondence and geometric transformation between each coordinate system, provides the error, and updates the time correspondence and geometric transformation.

[0037] In another embodiment, the processing system 400 uses data sensed between sensors from different distances and orientations to calculate the transformation from each measurement reference frame. One example of this is the alignment of ultrasound information from different positions and poses. Another example of this is the alignment of 3D volumetric or surface information from the HMD 100 and ultrasound information. The 3D information may be sensed directly using sensors such as time-of-flight optical sensing, pre-treatment CT or MRI, or may be derived using algorithms such as structure from motion (SFM) and simultaneous localization and mapping (SLAM).

[0038] In various embodiments, the processing system 400 uses the predicted transformations to calculate the transformations between each of these devices. These transformations are either interpolated between updates or predicted forward using filtering such as particle filtering and / or Kalman filtering. These transformations are updated using the measured rigid body transformations and the estimated transformations from the sensed information. These transformations may be updated using 3D matching such as ICP, or 3D to 2D techniques such as MI or SIFT / SURF. The 3D data may be calculated from 2D data using SFM or SLAM from ultrasound sensed from the probe or optical images from the HMD. The 2D data may be calculated from 3D to optical or ultrasound images using physically based rendering.

[0039] Each device computes a local scale of transformation between each sensor to provide a pose estimate for each device in a consistent local reference frame of the local scale estimator. The processing system 400 computes a global scale transformation between the local scale data by combining data from multiple local scale estimators and computing the transformation. Additionally, the global scale estimator 500 may compute an updated transformation between the reference frames used in the local scale estimators to update the pose estimate.

[0040] III. User Interface Examples 6 illustrates an exemplary graphical user interface for navigating an instrument, according to various embodiments. In the illustrated use case, a user is navigating a needle instrument 110 to a target intersection, for example, on a patient's vein or other body part.

[0041] Panel 600 illustrates a user interface showing an instrument path indicator 610 superimposed in 3D when the instrument tip is outside a threshold distance from the ultrasound image plane 620. The instrument path indicator 610 is the projected trajectory of the instrument. The processing system 400 updates the instrument path indicator 610 based on, for example, sensor data to reflect any changes in position or orientation in response to user control of the instrument. A marker 630 indicates the projected intersection of the instrument path indicator 610 on the ultrasound image plane 620. The instrument path indicator 610 is filled with a default color (or has no color, e.g., is transparent). In the example shown in FIG. 6, the instrument path indicator 610 is shown as a cylinder and the marker 630 is shown as a circle, but in other embodiments, the instrument path indicator 610 and the marker 630 may be shown using other types of shapes or graphics. The user interface may also include other types of 2D or 3D annotations.

[0042] Panel 640 shows the instrument path indicator 610 when the instrument tip is between the threshold distance from the ultrasound image plane 620, but has not yet intersected the ultrasound image plane 620. The distance between the left and right angle brackets is proportional to the distance between the instrument tip and the ultrasound image plane 620. The portion 650 of the instrument path indicator 610 filled with the "ahead of plane color" is proportional to the distance the instrument has navigated beyond the threshold distance from the ultrasound image plane 620. The remainder of the instrument path indicator 610 is filled with a default color. The tip of each angle bracket is pointed towards the intersection point, indicating that the instrument path indicator 610 is pointed towards the ultrasound image plane 620. In the example shown in FIG. 6, the left and right angle brackets are shown as right angles, but in other embodiments, this pair of indicators may be shown using other angles or types of shapes or graphics.

[0043] Panel 660 shows the user interface when the instrument tip reaches the intersection with the ultrasound image plane 620. The left and right angle brackets intersect at the center of the marker 630. In the example shown in Figure 6, the percentage of the instrument path indicator 610 that is shaded with the "pre-plane color" is half. The remaining portion of the instrument path indicator 610 is filled with the default color.

[0044] Panel 670 shows the user interface when the instrument tip has navigated beyond the ultrasound image plane 620. The apex of each angle bracket points away from the intersections and markers 630, indicating that the instrument path is pointing away from the ultrasound image plane 620. As also shown in panel 660 of this example, the percentage of the instrument path indicator 610 that is shaded with the "fore-plane color" remains half. The portion 680 of the instrument path indicator 610 that is shaded with the darker "behind-plane color" is proportional to the distance of the instrument tip beyond the ultrasound image plane 620. The remainder of the instrument path indicator 610 is filled with the default color.

[0045] 7 illustrates additional exemplary graphical user interfaces for navigating an instrument, according to various embodiments. Panel 700 corresponds to the scenario shown in panel 600 of FIG. 6 when the instrument tip is outside a threshold distance from the ultrasound image plane. Panel 710 corresponds to the scenario shown in panel 660 of FIG. 6 when the instrument tip reaches an intersection with the ultrasound image plane. Panel 720 corresponds to the scenario shown in panel 670 of FIG. 6 when the instrument tip has navigated beyond the ultrasound image plane.

[0046] 8 illustrates a heads-up display for navigating an instrument, in accordance with various embodiments. The processing system 400 updates the heads-up display (presented to the user by the HMD 100) based on the position of the instrument tip. For example, the display of the instrument path indicator, left angle bracket, and right angle bracket are updated according to different scenarios previously described with respect to FIG.

[0047] IV. Process Flow Examples 9 is a flow chart of a process 900 for tracking an instrument using multiple sensors, according to various embodiments. The processing system 400 may use the process 900 to provide visual guidance to a user wearing the HMD 100 and controlling the instrument 110.

[0048] In step 910, the processing system 400 receives data from a first electromagnetic sensor coupled (e.g., rigidly coupled) to the HMD 100 and detects an electromagnetic field generated by an electromagnetic reference source coupled (e.g., rigidly coupled) to the ultrasound probe 120. In step 920, the processing system 400 receives data from a second electromagnetic sensor coupled (e.g., rigidly coupled) to the medical instrument 110 and detects an electromagnetic field. In various embodiments, the first electromagnetic sensor and the second electromagnetic sensor are wireless. That is, the electromagnetic sensors are not hard-wired to the electromagnetic reference source. The system environment can include any number of additional wireless electromagnetic sensors. The system environment can also include mounting accessories for coupling the electromagnetic sensors to other devices, such as the HMD 100, the ultrasound probe 120, and the medical instrument 110. In various embodiments, the processing system 400 aggregates the data from the electromagnetic sensors with data captured by other sensors, such as one or more IMUs, depth sensors, or cameras.

[0049] In step 930, the processing system 400 determines a position of the HMD 100 relative to the ultrasound probe 120 by processing the data from the first electromagnetic sensor. In step 940, the processing system 400 determines a position of the medical instrument 110 relative to the ultrasound probe 120 by processing the data from the second electromagnetic sensor.

[0050] In step 950, the processing system 400 generates a visualization of the path of the medical instrument 110 oriented relative to the ultrasound image plane based at least in part on the position of the HMD 100 and the position of the medical instrument 110.

[0051] In step 960, the processing system 400 provides a graphic to a user wearing the HMD 100 for display by the HMD 100. The graphic includes a visualization and image data captured by the ultrasound probe 120 that is displayed on the ultrasound image plane.

[0052] 10 is a flowchart of a process 1000 for navigating an instrument in various embodiments. The processing system 400 may use the process 1000 to provide visual guidance to a user wearing the HMD 100 and controlling the instrument 110.

[0053] In step 1010, the processing system 400 receives image data of an ultrasound image plane. The image data is captured by the ultrasound probe 120 or another ultrasound image sensor. In step 1020, the processing system 400 determines a position of the medical instrument 110 relative to the ultrasound image plane. In step 1030, the processing system 400 determines an orientation of the medical instrument 110 relative to the ultrasound image plane. In step 1040, the processing system 400 determines a trajectory of the medical instrument 110 based on the position and orientation of the medical instrument 110.

[0054] In step 1050, the HMD 100 displays the instrument path indicating the trajectory of the medical instrument 110. In step 1060, the processing system 400 determines that the medical instrument 110 has reached an intersection point on the ultrasound image plane based on the updated position of the medical instrument 110 along the trajectory. In step 1070, the HMD 100 displays an updated version of the instrument path indicating that the medical instrument 110 has reached an intersection point on the ultrasound image plane.

[0055] V. Alternative Considerations The foregoing description of embodiments of the present invention has been presented for purposes of illustration and is not intended to be exhaustive or to be limited to the precise form disclosed. Those skilled in the relevant art will recognize that many modifications and variations are possible in light of the above disclosure.

[0056] Some portions of this description describe embodiments of the invention in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to effectively convey the substance of their work to others skilled in the art. While these operations have been described functionally, computationally, or logically, they will be understood to be implemented by computer programs or equivalent electrical circuits, or microcode, or the like. Further, and without loss of generality, it is sometimes convenient to refer to arrangements of these operations as modules. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combination thereof.

[0057] Any step, operation, or process described herein may be performed or implemented using one or more hardware or software modules, alone or in combination with other devices. In one embodiment, the software modules are implemented in a computer program product that includes a non-transitory computer-readable medium that includes computer program code that can be executed by a computer processor to perform any or all of the steps, operations, or processes described.

[0058] Embodiments of the invention may also relate to products produced by the computing processes described herein. Such products may include information resulting from the computing processes, where this information is stored on a non-transitory, tangible computer-readable storage medium and may include any embodiment of the computer program products described herein, or other data combinations.

[0059] Finally, the language used in the specification has been selected primarily for ease of reading and explanation, and may not have been selected to delineate or limit the subject matter of the invention. Accordingly, it is intended that the scope of the invention be limited not by this detailed description, but by the claims that will be issued in an application based thereon. Accordingly, the disclosure of embodiments of the invention is intended to illustrate, but not to limit, the scope of the invention, which is set forth in the following claims.

Claims

1. receiving data from a first electromagnetic sensor coupled to a head mounted display (HMD) and detecting an electromagnetic field generated by an electromagnetic reference source coupled to the ultrasound probe; receiving data from a second electromagnetic sensor coupled to the medical instrument to detect the electromagnetic field; determining a position of the HMD relative to the ultrasound probe by processing the data from the first electromagnetic sensor; determining a position of the medical instrument relative to the ultrasound probe by processing the data from the second electromagnetic sensor; generating a visualization of a path of the medical instrument oriented relative to an ultrasound image plane based at least in part on the position of the HMD and the position of the medical instrument; providing a user wearing the HMD with a graphic for display by the HMD, the graphic including the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane; The method includes:

2. receiving data from a third electromagnetic sensor coupled to the HMD to detect the electromagnetic field generated by the electromagnetic reference source coupled to the ultrasound probe; determining an orientation of the HMD based on a differential signal between the first electromagnetic sensor and the second electromagnetic sensor by processing the data from the first electromagnetic sensor and the data from the third electromagnetic sensor; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the orientation of the HMD is further determined based on data captured by an inertial measurement unit of the HMD.

4. 4. The method of claim 1, further comprising determining an orientation of the medical instrument by aggregating data captured by an inertial measurement unit of the medical instrument and the data from the second electromagnetic sensor.

5. 4. The method of claim 1, further comprising the step of determining a position of the ultrasound probe by processing image data captured by an image sensor of the HMD, wherein the graphic is displayed to the user at an offset from the position of the ultrasound probe.

6. 4. The method of claim 1, further comprising the step of determining an offset distance between a tip of the medical instrument and a position of the second electromagnetic sensor coupled to the medical instrument, the visualization being generated based on the offset distance.

7. determining a first transformation for mapping a first coordinate system of the medical instrument to a second coordinate system of the ultrasound probe; determining a second transformation for mapping a third coordinate system of the HMD to the second coordinate system of the ultrasound probe; The method of claim 1 , wherein the visualization is generated using the first transformation and the second transformation.

8. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: receiving data from a first electromagnetic sensor coupled to a head mounted display (HMD) and detecting an electromagnetic field generated by an electromagnetic reference source coupled to the ultrasound probe; receiving data from a second electromagnetic sensor coupled to the medical instrument to detect the electromagnetic field; determining a position of the HMD relative to the ultrasound probe by processing the data from the first electromagnetic sensor; determining a position of the medical instrument relative to the ultrasound probe by processing the data from the second electromagnetic sensor; generating a visualization of a path of the medical instrument oriented relative to an ultrasound image plane based at least in part on the position of the HMD and the position of the medical instrument; providing a user wearing the HMD with a graphic for display by the HMD, the graphic including the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane; A non-transitory computer-readable storage medium that causes

9. an electromagnetic reference source coupled to the ultrasound probe and configured to generate an electromagnetic field; a first electromagnetic sensor coupled to a head mounted display (HMD) and configured to detect the electromagnetic field; a second electromagnetic sensor coupled to the medical instrument and configured to detect the electromagnetic field; A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: determining a position of the HMD relative to the ultrasound probe by processing data from the first electromagnetic sensor; determining a position of the medical instrument relative to the ultrasound probe by processing data from the second electromagnetic sensor; generating a visualization of a path of the medical instrument oriented relative to an ultrasound image plane based at least in part on the position of the HMD and the position of the medical instrument; providing a user wearing the HMD with a graphic for display by the HMD, the graphic including the visualization and image data captured by the ultrasound probe displayed on the ultrasound image plane; The computer-readable storage medium causes A system equipped with.

10. The system of claim 9 , further comprising the ultrasound probe, the HMD, and the medical instrument.

11. The system of claim 9 or claim 10, wherein the first electromagnetic sensor and the second electromagnetic sensor are wireless.

12. The system of claim 9 or claim 10, wherein the medical instrument is a needle, a cannula, a biopsy device, or an ablation device.

13. receiving image data for an ultrasound image plane; determining a position of a medical instrument relative to the ultrasound image plane; determining an orientation of the medical instrument relative to the ultrasound image plane; determining a trajectory of the medical instrument based on the position and the orientation of the medical instrument; displaying, by a head mounted display (HMD), an instrument path indicating the trajectory of the medical instrument; determining, based on an updated position of the medical instrument along the trajectory, that the medical instrument has reached an intersection point on the ultrasound image plane; displaying, by the HMD, an updated version of the instrument path indicating that the medical instrument has reached the intersection point on the ultrasound image plane; The method includes:

14. determining that the medical instrument has moved through the intersection point on the ultrasound image plane; 14. The method of claim 13, further comprising: displaying, by the HMD, another updated version of the instrument path indicating that the medical instrument has moved through the intersection point on the ultrasound image plane.

15. 15. The method of claim 13 or claim 14, wherein the updated version of the instrument path includes a portion of the instrument path that is colored based on a distance that the medical instrument has traveled along the trajectory.

16. The method of claim 13 or claim 14, further comprising, in response to determining that the medical instrument has reached the intersection point on the ultrasound image plane, displaying, by the HMD, an annotation overlaid on the instrument path indicating that the medical instrument has reached the intersection point on the ultrasound image plane.

17. The method of claim 13 or claim 14, further comprising determining the intersection point on the ultrasound image plane based on tissue density.

18. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: receiving image data for an ultrasound image plane; determining a position of a medical instrument relative to the ultrasound image plane; determining an orientation of the medical instrument relative to the ultrasound image plane; determining a trajectory of the medical instrument based on the position and the orientation of the medical instrument; providing an instrument path indicating the trajectory of the medical instrument for display by a head mounted display (HMD); determining that the medical instrument has reached an intersection point on the ultrasound image plane based on an updated position of the medical instrument along the trajectory; providing an updated version of the instrument path for display by the HMD indicating that the medical instrument has reached the intersection point on the ultrasound image plane; and A non-transitory computer-readable storage medium that causes

Images